Iptables – Routing outbound SMTP with IPTables

Yes, the SMTP service in IIS is capable of full end-to-end delivery. In order for it to function this way, the following requirements must be met:

• The server needs to be on a routable IP address (ie not an IP address in one of the reserved private ranges)
• The DNS settings on the server need to be configured properly so it can resolve MX records of the recipient domains properly
• You need to clear out the "smart host" setting. Find this by right-clicking on the SMTP server in the IIS Admin console, select Properties, select the "Delivery" tab, click the "Advanced" button, and you will see the space for entering a smart host.

This is unavoidably tricky. The nature of TLS is such that a plaintext connection to your MTA has to be established before TLS can be negotiated, so iptables (operating as it does at the transport layer) is ill-designed to influence issues at the application layer.

You could write another target module and direct your traffic through that, but unless you're a networking God, this is probably no more feasible for you than it is for me. And I definitely don't know how to do it.

The upshot is that application-layer stuff is much easier to enforce inside the application. You don't say what MTA you're using, but I suspect that most MTAs that are bright enough to do TLS are bright enough to mandate it.

I use sendmail. There's a nice piece on mandating TLS from various providers at http://www.brandonhutchinson.com/Using_TLS_with_Sendmail.html , which directs me to the access database entry

TLS_Clt:communication_partner_MTA                           PERM+VERIFY:112

which requires a particular communication partner, presumably identified by IP address, to both authenticate with a key of at least 112 bits length, and have a properly-signed certificate. The sendmail config page at http://www.sendmail.org/documentation/configurationReadme , in the ANTI-SPAM CONFIGURATION CONTROL section, says that access db entries involving IPv4 addresses can take the form of a single octet, which then apply to all addresses beginning with that octet. So I speculate, and I stress it's just speculation, that sendmail would allow me to have a series of entries

TLS_Clt:1       PERM:112
TLS_Clt:2       PERM:112
TLS_Clt:3       PERM:112
....
TLS_Clt:223       PERM:112

Mandating encryption (though not verifiably-signed certificates; self-signed TLS certs are very common, and I'd be inclined not to bar them) from all IP addresses. I would also not have an entry for TLS_Clt:127, as localhost should probably not be so restricted.

I repeat that I've not tested any of the above, and if your MTA is something other than sendmail, the above won't be specifically helpful; but I wanted to show that my MTA (at least) seems to have hooks for doing what you want. Good luck with your investigations.