Standard DNS Record to Indicate IMAP Server for a Domain

[Frankly, you have so much pretend data here that this is probably going to be unnecessarily confusing. I'll try to keep everything straight.]

If I understand your question correctly, you currently have the following DNS records:

example.com.      A   1.2.3.4
example.com.      MX  10 example.com.
www.example.com.  A   4.5.6.7

...and a bunch of unspecified CNAME records. Also, there exists:

some.other.host.  A   2.3.4.5

...hosting your new website.

You want example.com (and presumably www.example.com) to point to your new site without disrupting email.

Try this:

• Register mx.example.com as a new A record pointing to 1.2.3.4. Update the MX record for example.com to point to mx.example.com. At this point, you'll have:

  example.com.      A   1.2.3.4
  mx.example.com    A   1.2.3.4
  example.com.      MX  10 mx.example.com.
  www.example.com.  A   4.5.6.7
  

• Now wait at least twice the TTL for your records to make sure the old MX record has time to expire from DNS caches.

• Next, update the A record for example.com and www.example.com to point to your new site. You'll end up with something along the lines of:

  example.com.      A   2.3.4.5
  example.com.      MX  10 mx.example.com.
  mx.example.com.   A   1.2.3.4
  www.example.com.  A   2.3.4.5
  

At this point, I think you have what you want. Email continues to be handled by your existing mail server, but your web presence has been moved onto your new host.

CNAME records were originally created to allow multiple names that provide the same resource to be aliased to a single "canonical name" for the resource. With the advent of name based virtual hosting, it has instead become commonplace to use them as a generic form of IP address aliasing. Unfortunately, most people who come from a web hosting background expect CNAME records to indicate equivalence in the DNS, which has never been the intent. The apex contains record types which are clearly not used in the identification of a canonical host resource (NS, SOA), which cannot be aliased without breaking the standard at a fundamental level. (particularly in regards to zone cuts)

Unfortunately, the original DNS standard was written before the standards governing bodies realized that explicit verbiage was necessary to define consistent behavior (RFC 2119). It was necessary to create RFC 2181 to clarify several corner cases due to vague wording, and the updated verbiage makes it clearer that a CNAME cannot be used to achieve apex aliasing without breaking the standard.

6.1. Zone authority

The authoritative servers for a zone are enumerated in the NS records for the origin of the zone, which, along with a Start of Authority (SOA) record are the mandatory records in every zone. Such a server is authoritative for all resource records in a zone that are not in another zone. The NS records that indicate a zone cut are the property of the child zone created, as are any other records for the origin of that child zone, or any sub-domains of it. A server for a zone should not return authoritative answers for queries related to names in another zone, which includes the NS, and perhaps A, records at a zone cut, unless it also happens to be a server for the other zone.

This establishes that SOA and NS records are mandatory, but it says nothing about A or other types appearing here. It may seem superfluous that I quote this then, but it will become more relevant in a moment.

RFC 1034 was somewhat vague about the problems that can arise when a CNAME exists alongside other record types. RFC 2181 removes the ambiguity and explicitly states the record types that are allowed to exist alongside them:

10.1. CNAME resource records

The DNS CNAME ("canonical name") record exists to provide the canonical name associated with an alias name. There may be only one such canonical name for any one alias. That name should generally be a name that exists elsewhere in the DNS, though there are some rare applications for aliases with the accompanying canonical name undefined in the DNS. An alias name (label of a CNAME record) may, if DNSSEC is in use, have SIG, NXT, and KEY RRs, but may have no other data. That is, for any label in the DNS (any domain name) exactly one of the following is true:

• one CNAME record exists, optionally accompanied by SIG, NXT, and KEY RRs,
• one or more records exist, none being CNAME records,
• the name exists, but has no associated RRs of any type,
• the name does not exist at all.

"alias name" in this context is referring to the left hand side of the CNAME record. The bulleted list makes it explicitly clear that a SOA, NS, and A records cannot be seen at a node where a CNAME also appears. When we combine this with section 6.1, it is impossible for a CNAME to exist at the apex as it would have to live alongside mandatory SOA and NS records.

(This seems to do the job, but if someone has a shorter path to proof please give a crack at it.)

Update:

It seems that the more recent confusion is coming from Cloudflare's recent decision to allow an illegal CNAME record to be defined at the apex of domains, for which they will synthesize A records. "RFC compliant" as described by the linked article refers to the fact that the records synthesized by Cloudflare will play nicely with DNS. This does not change the fact that it is a completely custom behavior.

In my opinion this is a disservice to the larger DNS community: it is not in fact a CNAME record, and it misleads people into believing that other software is deficient for not allowing it. (as my question demonstrates)