STARTTLS and MTA-STS

How STARTTLS works

An SMTP connection on port 25 begins in cleartext. The two mail servers say hello, exchange supported extensions (EHLO response), and one of those extensions is STARTTLS. If both sides support it, the client issues STARTTLS; the server responds 220 ready; both sides perform a TLS handshake on the existing connection. From that point, everything is encrypted.

For client-to-server submission (port 587 or 465), TLS is typically required from the start — submission is not opportunistic. STARTTLS opportunism is specifically a server-to-server property.

The opportunism problem

Opportunistic means "encrypt if you can, otherwise don't bother." An attacker positioned on the network path can exploit this:

1. Sending server connects to recipient on port 25.
2. Server sends EHLO; recipient advertises STARTTLS in cleartext.
3. Attacker, watching the cleartext exchange, strips STARTTLS from the EHLO response.
4. Sending server doesn't see STARTTLS advertised; assumes it's not supported.
5. Mail proceeds in cleartext over the rest of the connection. Attacker reads it.

This is the downgrade attack. STARTTLS by itself can't prevent it because the negotiation is in cleartext.

MTA-STS: making TLS mandatory

MTA-STS lets a recipient domain publish a policy that sending servers must consult:

1. Recipient publishes a TXT record at _mta-sts.example.com with a policy ID.
2. Recipient hosts a policy file at https://mta-sts.example.com/.well-known/mta-sts.txt describing required behavior — which MX hosts are valid, what mode (testing/enforce), how long to cache the policy.
3. Sending servers that implement MTA-STS fetch the policy via HTTPS (validating the certificate), cache it for the lifetime indicated, and apply it.
4. If a connection to one of the listed MX hosts can't establish TLS with a valid certificate matching the host, in enforce mode the delivery fails rather than falling back to cleartext.

The HTTPS fetch is the trust anchor: an attacker can't forge the policy without compromising the recipient's HTTPS certificate.

The policy file format

version: STSv1
mode: enforce
mx: mail.example.com
mx: mail2.example.com
max_age: 86400

Modes:

• none — policy exists but doesn't enforce anything. Use during decommissioning.
• testing — observe what would have happened in enforce mode without affecting delivery. Pair with TLS-RPT to get reports.
• enforce — fail delivery if TLS conditions aren't met. The end state.

TLS-RPT: visibility

SMTP TLS Reporting (RFC 8460) gives recipient domains visibility into delivery TLS issues. The recipient publishes a TXT record at _smtp._tls.example.com with a rua=mailto:tls-reports@example.com directive. Sending servers send daily aggregate reports listing connection failures, downgrades, and policy mismatches. Without these reports, you have no way to know if your MTA-STS deployment is breaking delivery.

DANE: the DNSSEC alternative

DANE (DNS-Based Authentication of Named Entities) is a different approach to making TLS mandatory: publish a TLSA record in DNS that pins the expected certificate or public key. Requires DNSSEC for the DNS records to be trustworthy. Adoption is concentrated among certain ISPs and email-heavy users; MTA-STS has broader adoption because it doesn't require DNSSEC.

Both can coexist on the same domain. Major mail providers implement both; a receiver may use whichever policy a sender supports.

Certificate requirements

For MTA-STS to enforce, your MX hosts must serve valid TLS certificates that:

• Are issued by a publicly trusted CA (Let's Encrypt is fine).
• Cover the MX hostname (e.g., mail.example.com).
• Are not expired.
• Match a name listed in your MTA-STS policy.

Self-signed certificates and certificates issued by private CAs don't pass MTA-STS validation.

Deploying MTA-STS step by step

1. Verify every MX host serves a valid public certificate.
2. Verify TLS connections to those MX hosts work from external sending servers (test with online tools).
3. Set up the HTTPS endpoint hosting the policy file at mta-sts.example.com/.well-known/mta-sts.txt.
4. Publish the policy in testing mode with the policy ID DNS record.
5. Deploy TLS-RPT and watch reports for several weeks.
6. If no issues, switch policy mode to enforce.

What MTA-STS does not protect

• Content visible to the recipient mail server. Mail is decrypted at the recipient; whatever happens after is not MTA-STS's concern.
• Mail content from end users. If you want end-to-end encryption, use S/MIME or PGP. MTA-STS only secures the transport.
• Outbound from your domain. MTA-STS protects incoming mail. Outbound is whatever the destination domain enforces.
• Replay or man-in-the-middle attacks against the policy file fetch. If your HTTPS cert is compromised, MTA-STS is compromised.

The bigger email-security picture

MTA-STS and STARTTLS protect mail in transit between servers. They complement, not replace, the authentication mechanisms (SPF, DKIM, DMARC) that prove who sent a message. A fully-secured mail flow combines: SPF/DKIM/DMARC for authentication, MTA-STS for transport encryption, and TLS-RPT for monitoring.

Frequently Asked Questions

What is STARTTLS?

A protocol command that upgrades a plaintext SMTP connection to encrypted TLS in-place. The two servers start talking in cleartext, then either side can send STARTTLS to initiate a TLS handshake; if successful, all subsequent traffic on that same connection is encrypted. It's the standard way SMTP between mail servers is encrypted.

Why is STARTTLS considered opportunistic?

Because the encryption is offered but not enforced. If STARTTLS is unavailable or the TLS handshake fails, most mail servers fall back to plaintext rather than refusing to deliver. An active attacker can strip the STARTTLS advertisement from the cleartext negotiation and force the fallback — a downgrade attack.

What is MTA-STS?

MTA Strict Transport Security — a policy mechanism that lets a domain owner publish "mail to my domain must use TLS, and the certificate must be valid." The policy is published via DNS plus an HTTPS-served policy file. Sending servers that implement MTA-STS cache the policy and refuse to deliver in cleartext or with an invalid certificate, defeating downgrade attacks.

What is TLS-RPT?

SMTP TLS Reporting — a companion to MTA-STS that lets a sending domain report TLS connectivity issues back to the recipient domain. The recipient publishes a TLS-RPT DNS record with an address to send reports to; senders deliver daily aggregate reports on connection failures, downgrades, and policy violations. Used to monitor and debug TLS deployment.

Should I deploy MTA-STS?

Yes, for any domain that receives email. It's relatively low-effort (a DNS record plus a static HTTPS-served policy file) and meaningfully raises the bar against downgrade attacks. Start with "testing" mode to validate the policy works, then switch to "enforce" once you're confident sending servers can reach your MX hosts with valid TLS.