The EAI Support Survey Tool is a program that tests all the Mail Exchange servers for support of UTF8 and internationalized domain names.

The application is built in Java 8 and uses a MariaDB database to store the tests' results temporarily.

• Java Compiler version 8 or superior.
• Apache Maven version 3.6 or superior.

• Buildah 1.19.6 or superior.

• Java JRE version 8 or superior.
• Docker CE version 20.10.7 or superior.
• MariaDB client version 15.1 or superior.
• SnowSQL version 1.2.16 or superior.
• User must be on the docker group.
• User must be able to run sudo.

• Podman 3.0.1 or superior.
• SnowSQL version 1.2.16 or superior.
• User must be able to run sudo.

• Docker CE version 20.10.7 or superior.
• SnowSQL version 1.2.16 or superior.
• User must be on the docker group.
• User must be able to run sudo.

To compile the source code, the user can use Apache Maven. Apache Maven Assembly Plugin is used to create the distribution package. To create the distribution package is created executing these couple of commands on the command line at the project's root directory.

$ git clone https://github.com/icann/eai-survey-tool.git
$ cd eai-survey-tool
$ mvn package
$ mvn assembly:single

After running these steps, the distribution bundle will be created on the target folder, and will have the name eai-survey-$VERSION.tar.gz. This bundle will contain all the compiled code and dependencies. The only thing missing will be the configuration files: config.properties and logging.properties.

The user can install the application decompressing the tar file at any location. The tar file contains all the needed files and dependencies.

tar -xvf eai-survey-$VERSION.tar.gz

To compile the source code, the user can use the buildah.sh script that utilizes buildah to compile the source code and creates a Podman/Docker image.

$ git clone https://github.com/icann/eai-survey-tool.git
$ cd eai-survey-tool
$ buildah unshare
$ scripts/buildah.sh
$ exit

The user can configure the application using a configuration file named config.properties. The application search this file at the install directory of the application by default.

The following parameters are configurable in the configuration file. Parameters with values in unicode MUST use unicode escape as per the Java Language Specification. For example, for the internationalized email addres, provide an address in escaped form:

\u0645\u062b\u0627\u0644@\u0627\u062e\u062a\u0628\u0627\u0631-\u0627\u0644\u0642\u0628\u0648\u0644-\u0627\u0644\u0639\u0627\u0644\u0645\u064a.\u0634\u0628\u0643\u0629

Instead of the original form:

smtp.tester.email.idn = مثال@اختبار-القبول-العالمي.شبكة

local.addresses.ipv4 = 127.0.0.1
local.addresses.ipv6 = 2600:1f14:df0:400:67ec:ada:c748:6fa2

resolver.address = 8.8.8.8, 2001:4860:4860::8888
resolver.threads = 1250

smtp.tester.threads = 32000
smtp.tester.quarantine.cidr.ipv4 = 20
smtp.tester.quarantine.cidr.ipv6 = 48

db.username = [redacted]
db.password = [redacted]
db.url = jdbc:mariadb://127.0.0.1:3306/eai?useServerPrepStmts=false&rewriteBatchedStatements=true

smtp.tester.domain = testo.tel
smtp.tester.email.ascii = \u0645\u062b\u0627\u0644@xn-----btdabbacdp7b0bzhrf3accct6cwc.xn--ngbc5azd
smtp.tester.email.idn = \u0645\u062b\u0627\u0644@\u0627\u062e\u062a\u0628\u0627\u0631-\u0627\u0644\u0642\u0628\u0648\u0644-\u0627\u0644\u0639\u0627\u0644\u0645\u064a.\u0634\u0628\u0643\u0629

ip.exclude = 1.1.1.1/32, 1.1.1.2/32

The logging is configured using a file named logging.properties located in the root directory of the application.

The application uses the standard Java Logger; here is the link to the complete documentation.https://docs.oracle.com/javase/8/docs/technotes/guides/logging/overview.html

handlers = java.util.logging.FileHandler, java.util.logging.ConsoleHandler
.level = SEVERE

org.icann.eai.survey.EaiSurvey.level = INFO
org.icann.eai.survey.Sentinel.level = INFO

java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter
java.util.logging.FileHandler.pattern = logs/log%g.log
java.util.logging.FileHandler.limit = 10485760
java.util.logging.FileHandler.count = 3

The first step is to configure the scripts. All the scripts use the same configuration file in the script folder with the name scripts.cfg.

The next step is executing the setup.sh script. This script search for the latest zones files, decompress them, parse them, and creates an import database file. This script also creates a Docker container from the latest version of MariaDB image.

Zone files are expected to be found in the location specified by the ZONES_SRC_DIR parameter, following the structure below. In the directory structure, the yyyymmdd directory is replaced with the date corresponding to the date of the zone files to be used in the survey, and tld1, tld2, etc. is replaced with the actual TLDs to be surveyed. If more than one directory is found for multiple dates, the script will consider the directory named after the latest date available only:

• source_directory (as configured for the ZONES_SRC_DIR constant)
  • yyyymmdd
    • tld1
      • tld1.zone.gz
    • tld2
      • tld2.zone.gz

The next step is to execute the main JAR file. This application will search, resolve, and test all the Mail Exchange servers. If a critical error occurs at any point of the execution, the user can re-run this application and resume the test.

java -Xmx4g -jar eai-survey-1.1.0.jar

Users can check the progress of the survey querying the temporary database.

# Check general progress. The timestamp indicates when the step was finished.
select * from progress;

# Count SLD to process
select count(*) from record where status = 'N';

# Count MX to resolve
select count(*) from mx where status_4 = 'N' or status_6 = 'N';

# Count MX to test
select count(*) from ip where status = 'N';

The next step is running the survey using the run.sh script that utilizes Podman or Docker to run the code inside a container.

scripts/run.sh

Users can check the progress of the survey querying the temporary database.

# Check general progress. The timestamp indicates when the step was finished.
select * from progress;

# Count SLD to process
select count(*) from record where status = 'N';

# Count MX to resolve
select count(*) from mx where status_4 = 'N' or status_6 = 'N';

# Count MX to test
select count(*) from ip where status = 'N';

The user also can use the report.sh script to display a summary of the execution. This script can take a while, so please be patient. The user and password to use for accessing the database must be replaced accordingly.

Here is an example of the execution.

$ ./report.sh

+-------------------------+---------------------+
| name                    | timestamp           |
+-------------------------+---------------------+
| Process started         | 2021-06-13 18:47:45 |
| Search MX Servers       | 2021-06-14 18:11:34 |
| Resolve IPv4 MX Servers | 2021-06-14 22:38:52 |
| Resolve IPv6 MX Servers | 2021-06-15 01:26:30 |
| Probe MX Servers        | 2021-06-17 22:53:11 |
| Resolve GeoIP Location  | 2021-07-01 01:48:25 |
+-------------------------+---------------------+

Number of SLD: 207,461,843
+----------+-------------+
| result   | count       |
+----------+-------------+
| 0K       | 100,325,168 |
| No MX    | 93,111,393  |
| NxDomain | 2,037,559   |
| Refused  | 24,007      |
| ServFail | 11,963,716  |
+----------+-------------+

Number of MX: 34,486,103

Number of IP: 2,567,691
- Server responded: 1,730,389
- Server timeout: 837,302

- Test allowed: 1,715,913
- Test denied: 768

- EHLO test passed: 168,941
- ASCII test passed: 155,439
- UTF-8 test passed: 155,071

Once that the survey completes, the user can use the export.sh script to export all database tables into CSV files. The script also compresses the files to optimize upload time to the S3 bucket. And finally, the script imports all the data into the snowflake data warehouse.

The S3 bucket and the SnowSQL client must be configured and available to the user. Here is a copy of the used configurations:

[default]
region = [redacted]

[default]
aws_access_key_id = [redacted]
aws_secret_access_key = [redacted]

[connections]
accountname = [redacted]
username = [redacted]
password = [redacted]
dbname = [redacted]
schema = [redacted]
warehousename = [redacted]
rolename = [redacted]

Depending on the size and the quantity of the zone tested, the EAI survey can take up a lot of hard disk space. If you plan to use Podman or Docker to run it, consider configuring the storage to a big partition.

Here is an example of how to configure both container managers.

[storage]
driver = "overlay"
runroot = "/data/podman"
graphroot = "/data/podman"

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

...
ExecStart=/usr/bin/dockerd --data-root /data/docker -H fd:// --containerd=/run/containerd/containerd.sock
...

The survey code can create a significant number of threads (configurable), and by default Linux and Podman configuration limits the number of threads on the system.

To update the max number of threads, configure the following files:

...
UserTasksMax=150000
...

...
kernel.threads-max=200000
...

[containers]
pids_limit=0
#Note: Maximum number of processes allowed in a container. 0 indicates that no limit is imposed.

Depending on the data, the survey can take a lot of time to process, and if you are using Podman, usually the container exit once you log out of your user session.

You can run the following command to prevent this behavior before closing the session.

# loginctl enable-linger $UID

erDiagram

progress {
    int id PK
    varchar(100) name
    timestamp timestamp
}

record {
    varchar(255) name PK
    varchar(63) zone
    char(1) status
}

record_mx {
    varchar(255) owner PK, FK
    varchar(255) mx PK, FK
}

mx {
    varchar(255) mx PK
    char(1) status_4
    char(1) status_6
}

mx_ip {
    varchar(155) mx PK, FK
    varchar(39) ip PK, FK
}

ip {
    varchar(39) ip PK
    char(1) status
    text(32768) header
    text(4000) ehlo_result
    char(1) ehlo_success
    text(4000) ascii_result
    char(1) ascii_success
    text(4000) idn_result
    char(1) idn_success
    char(1) country
    timestamp timestamp
}

record ||--o{ record_mx : has
mx ||--|{ record_mx: has
mx ||--o{ mx_ip : has
ip ||--|{ mx_ip: has

[1] Status List:

• N - No Searched
• S - Searched (No Error)
• X - Empty Answer
• # - RCode
• E - Unexpected Error

[2] IP Status List:

• N - No Tested
• S - Skipped Special
• X - Excluded
• T - Tested

Copyright 2021 Internet Corporation for Assigned Names and Numbers ("ICANN")

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.