how should I debug smtp delivery latency?

[astrocket]

I am running two different Docker Mailserver instances.

1: Old version of DMS, Using OpenDKIM, sending from a single self-owned domain, with an average latency of around 300ms.
2: Latest version of DMS, Using Rspamd, sending on behalf of multiple customer domains, with an average latency of 8 seconds.

I need help figuring out why the second instance is taking 8 seconds to send emails.

I used tcptrack and found that instance #2 receives the SMTP request but then hangs for 7 seconds without logging anything in the Docker Mailserver logs. After the 7-second delay, it sends the email quickly, with the same logs as instance #1.

I suspect it might be related to the Mail User Agent holding the request for 7 seconds, but I have no clue how to diagnose the cause.

Both instances pass SPF and DKIM checks without any issues.

[polarathene]

In the 2nd container what is ulimit -Sn? Can you observe any spike in CPU usage during that delay?

You should share your actual configuration. You should also compare configurations properly, an old DMS with OpenDKIM of unknown release to the latest v14.0 DMS release with Rspamd instead and multiple mail domains. That's quite a few differences.

Chances are you may have something misconfigured. I've not found any such delay with v14.0 with local testing. Could be that your DNS is associated to the issue? try running the container with --dns 1.1.1.1 if you don't need to reach any other containers on the same network from DMS.

[astrocket]

Thank you for your reply, @polarathene.

There haven't been any CPU spikes. I sent a bulk amount of emails to the second container, and the CPU usage increased smoothly without any spikes. (I even upgraded the instance spec to 4x, but the result was the same).

The ulimit -Sn value for the second container is 1024, which is the same as the first container.

I haven’t changed much aside from the SSL configuration and Rspamd DKIM settings. I’m using a plain Docker Mailserver setup.

Here are the environment variables for the second container (I’ve removed the ones with empty values for better visibility):

mailserver.env

DMS_DEBUG=0
LOG_LEVEL=trace
ONE_DIR=1
ENABLE_UPDATE_CHECK=1
UPDATE_CHECK_INTERVAL=1d
PERMIT_DOCKER=none
ENABLE_IMAP=1
ENABLE_RSPAMD=1
RSPAMD_LEARN=0
RSPAMD_CHECK_AUTHENTICATED=0
RSPAMD_GREYLISTING=0
RSPAMD_HFILTER=1
RSPAMD_HFILTER_HOSTNAME_UNKNOWN_SCORE=6
ENABLE_AMAVIS=0
ENABLE_DNSBL=0
ENABLE_FAIL2BAN=0
FAIL2BAN_BLOCKTYPE=drop
POSTSCREEN_ACTION=enforce
SSL_TYPE=letsencrypt
ENABLE_QUOTAS=1
SPAMASSASSIN_SPAM_TO_INBOX=1
MOVE_SPAM_TO_JUNK=1
MARK_SPAM_AS_READ=0
SA_TAG=2.0
SA_TAG2=6.31
SA_KILL=10.0
SA_SPAM_SUBJECT=***SPAM*****
LOGROTATE_INTERVAL=weekly
POSTFIX_REJECT_UNKNOWN_CLIENT_HOSTNAME=0
POSTFIX_INET_PROTOCOLS=all
DOVECOT_INET_PROTOCOLS=all

Here’s the dkim_signing.conf from the second container:

/mailserver/docker-data/dms/config/rspamd/override.d/dkim_signing.conf

# documentation: https://rspamd.com/doc/modules/dkim_signing.html

enabled = true;
sign_authenticated = true;
sign_local = false;
try_fallback = false;

use_domain = "header";
use_redis = false; # don't change unless Redis also provides the DKIM keys
use_esld = true;
allow_username_mismatch = true;

check_pubkey = true; # you want to use this in the beginning

signing_table = [
 "* keyname"
];

key_table = [
 "keyname %:mail:/tmp/docker-mailserver/rspamd/dkim/rsa-2048-mail.private.txt"
];

Regarding the --dns 1.1.1.1 suggestion, could you provide a bit more detail on how I should go about implementing it? Should I add this DNS flag to the Docker run command, or is there a different approach you recommend? I'd appreciate any further guidance on this.

Thanks again for your help!

[polarathene]

There haven't been any CPU spikes. I sent a bulk amount of emails to the second container, and the CPU usage increased smoothly without any spikes. (I even upgraded the instance spec to 4x, but the result was the same).

The ulimit -Sn value for the second container is 1024, which is the same as the first container.

The spike / load I was expecting would be a single thread at full load. As your command showed 1024 this is a non-concern so it won't be that at fault.

---

I haven’t changed much aside from the SSL configuration and Rspamd DKIM settings. I’m using a plain Docker Mailserver setup.

Here are the environment variables for the second container (I’ve removed the ones with empty values for better visibility):

Yes but the unknown old version of DMS you refer to, have you started with a fresh new DMS, or attempted to copy over anything from the old one? It's important to pay attention to our CHANGELOG.md file that describes breaking changes with each major version release, it is your responsibility to make any adjustments that apply to your setup for a smooth upgrade process.

I can see that you have ENV shared that are no longer valid for example, such as DMS_DEBUG (no longer used for a while now), ONE_DIR also is no longer supported, it is only detected via /var/mail-state mount existing.

It also looks like you've not disabled other services that Rspamd would replace, see this comment for other related ENV you should consider, your container logs should be warning you about this when you start the container.

Also worth considering is trying without Rspamd enabled, and switching back to OpenDKIM temporarily to see if that makes a difference for you. Normally users have issues with OpenDKIM and this is fixed by using Rspamd.

---

Regarding the --dns 1.1.1.1 suggestion, could you provide a bit more detail on how I should go about implementing it?

Yes it's part of the docker run options, or for compose.yaml it would be dns: 1.1.1.1 in your DMS service config.

Sometimes the default DNS can be problematic. For example on my home network the router is from the ISP and fails to handle some DNS queries properly which caused some software to not function properly, when I changed my DNS to 1.1.1.1 (Cloudflare), this fixed the issue for me for that software (not DMS related).

If your first DMS container is running on the same system, it's probably not DNS since you have no issue with that.

---

If you have Postgrey enabled, consider disabling that.

Also look into your DMS logs when mail is being sent with this delay, try to see if there's any information about what is causing the delay, normally this would appear in the logs which you've stated do not seem to differ and have nothing until the delay is done?

Is that also the log about a connection from the mail client being delayed? If so then yes it's probably something with your mail client not DMS, although odd that it's not affecting your other DMS setup.

You can test this with swaks which you could run in the newer DMS container at least:

docker exec -it my-dms-container-name bash

swaks --ehlo mail.dms-new.test --server mail.dms-old.test \
  --port 465 -tlsc \
  --auth PLAIN --auth-user [email protected] --auth-password some-secret \
  --from [email protected] --to [email protected]

That command:

• Sends a HELO/EHLO (SMTP greeting announcing who you are to the other mail server) when connecting to your old DMS container over port 465 with TLS.
• It uses plain-text auth mechanism with the credentials [email protected] user and some-secret password.
• A test mail is submitted with a from address of the DMS account authenticated, and it is sent to the recipient address [email protected] (back to your 2nd DMS instance).

Change the mail.dms-new.test and mail.dms-old.test to the proper hostnames, same with the credentials and from/to addresses. You could change --to address to anything you'd like, as an authenticated submission DMS should accept and relay that to any external address, just like if you sent one from your mail client.

You can also do this without the original 1st DMS container you have. Just send a mail with swaks to DMS within the same container (DMS container 2, v14.0):

# --ehlo might not be needed:
swaks --server mail.dms-new.test \
  --port 465 -tlsc \
  --auth PLAIN --auth-user [email protected] --auth-password some-secret \
  --from [email protected] --to [email protected]

If you don't get a delay this way, then it's very much likely something external of DMS like your mail client and so we won't have much luck troubleshooting why that is unfortunately :(

---

A final change you could consider is trying our :edge image to see if that has any unreleased fixes that help. Although the present changelog doesn't seem like there is anything that'd likely help.

---

You did mention another difference was related to single vs multiple domains, is it possible there was a configuration issue there? Either with DMS or the DNS?

As a reminder DMS should have a hostname configured like mail.example.com, and your mail domains can be anything like [email protected] and [email protected]. Your DNS records for those domains would have MX records point to mail.example.com to deliver mail to DMS, and TXT SPF records that grant permission to mail.example.com for sending mail on behalf of each of those mail domains.

You only need a LetsEncrypt certificate for mail.example.com that DMS will use for TLS connections that connect to it. Nothing required for the mail domains.

Make sure mail is submitted via port 587 or 465 with the DMS account credentials (from setup email add [email protected] for example).

If you have any aliases, they should only be for domains DMS manages (_such as somewhere-else.com, another-domain.com, mail.example.com_), they can point to any internal or external address as the recipient / target of the alias, but do not create an alias for an external domain like setup alias add [email protected] [email protected]as that@gmail.com` is not a mail domain you'd own and it would break things.

DMS does make some assumptions, such as mail.example.com for the hostname with some default mail addresses like postmaster assumed to be [email protected]. Since you mention going from single domain to multi-domain, it might be a subtle difference, but I don't think it'd cause a delay 🤔