What is the appropriate way to set up a DNS record that says "this domain has no mail server"?
I assume I need a special MX record to do this, otherwise it will be assumed that the A record is the answer.
I ask this question because it seems like it would be better to stop the mail at the front line, so it does not become the responsibility of the webserver to reject mail for the domain in question.
Best Answer
Due to the fallback of directly contacting a host via its address records, a single "null MX" record of "MX 0 ." is the apparent preferred way to indicate that the host does not accept e-mail. This is similar to a "null SRV" record ("SRV 0 0 0 .") which specifically marks a service as not available (per the SRV-RR RFC 2782).
This has been standardized by RFC 7505 (as of December 2017 it is a proposed standard).
"MX 0 localhost." (or equivalent label pointing at ::1 and 127.0.0.1) is also acceptable but more appropriate for a host that must send mail to itself (e.g. cron job output) which does not accept external mail. Such hosts may have an operational mail server which is firewalled off from the Internet but other services are accessible.
Having no MX record and blocking the SMTP port does not stop people from wasting one's incoming bandwidth trying to contact a non-existent server. The single MX record methods above do prevent such traffic because address-type records are never tried when at least one MX record is present. This will probably not stop some spammers from trying to contact a host directly via its address records. However, as it does stop legitimate traffic from trying, you'll be able to identify spam sources with 100% certainty.
Using private addresses should not be used because one cannot tell where they'll end up. Using other reserved addresses (e.g. documentation address of 192.0.2.0/24) is also inappropriate except where trying to identify and trap spammers within one's own network when they try to connect.