Is it possible to completely avoid a single-point-of-failure in a web back-end

When I use the term "DNS Round Robin" I generally mean in in the sense of the "cheap load balancing technique" as OP describes it.

But that's not the only way DNS can be used for global high availability. Most of the time, it's just hard for people with different (technology) backgrounds to communicate well.

The best load balancing technique (if money is not a problem) is generally considered to be:

1. A Anycast'ed global network of 'intelligent' DNS servers,
2. and a set of globally spread out datacenters,
3. where each DNS node implements Split Horizon DNS,
4. and monitoring of availability and traffic flows are available to the 'intelligent' DNS nodes in some fashion,
5. so that the user DNS request flows to the nearest DNS server via IP Anycast,
6. and this DNS server hands out a low-TTL A Record / set of A Records for the nearest / best datacenter for this end user via 'intelligent' split horizon DNS.

Using anycast for DNS is generally fine, because DNS responses are stateless and almost extremely short. So if the BGP routes change it's highly unlikely to interrupt a DNS query.

Anycast is less suited for the longer and stateful HTTP conversations, thus this system uses split horizon DNS. A HTTP session between a client and server is kept to one datacenter; it generally cannot fail over to another datacenter without breaking the session.

As I indicated with "set of A Records" what I would call 'DNS Round Robin' can be used together with the setup above. It is typically used to spread the traffic load over multiple highly available load balancers in each datacenter (so that you can get better redundancy, use smaller/cheaper load balancers, not overwhelm the Unix network buffers of a single host server, etc).

So, is it true that, with multiple data centers and HTTP traffic, the use of DNS RR is the ONLY way to assure high availability?

No it's not true, not if by 'DNS Round Robin' we simply mean handing out multiple A records for a domain. But it's true that clever use of DNS is a critical component in any global high availability system. The above illustrates one common (often best) way to go.

Edit: The Google paper "Moving Beyond End-to-End Path Information to Optimize CDN Performance" seems to me to be state-of-the-art in global load distribution for best end-user performance.

Edit 2: I read the article "Why DNS Based .. GSLB .. Doesn't Work" that OP linked to, and it is a good overview -- I recommend looking at it. Read it from the top.

In the section "The solution to the browser caching issue" it advocates DNS responses with multiple A Records pointing to multiple datacenters as the only possible solution for instantaneous fail over.

In the section "Watering it down" near the bottom, it expands on the obvious, that sending multiple A Records is uncool if they point to datacenters on multiple continents, because the client will connect at random and thus quite often get a 'slow' DC on another continent. Thus for this to work really well, multiple datacenters on each continent are needed.

This is a different solution than my steps 1 - 6. I can't provide a perfect answer on this, I think a DNS specialist from the likes of Akamai or Google is needed, because much of this boils down to practical know-how on the limitations of deployed DNS caches and browsers today. AFAIK, my steps 1-6 are what Akamai does with their DNS (can anyone confirm this?).

My feeling -- coming from having worked as a PM on mobile browser portals (cell phones) -- is that the diversity and level of total brokeness of the browsers out there is incredible. I personally would not trust a HA solution that requires the end user terminal to 'do the right thing'; thus I believe that global instantaneous fail over without breaking a session isn't feasible today.

I think my steps 1-6 above are the best that are available with commodity technology. This solution does not have instantaneous fail over.

I'd love for one of those DNS specialists from Akamai, Google etc to come around and prove me wrong. :-)

If you want no single points of failure at all, you need to do global server load balancing -- you obviously can't rely on a single datacentre, and even with a redundant BGP configuration, your BGP tables constitute a single point of failure that can be messed up if someone pushes a bad config.

What you do is configure DNS to advertise multiple IP addresses for the A record for your domain name, pointing at copies of your site that are in different datacentres (preferably in different cities), and the browser will pick one (usually at random, but watch out for Windows Vista which implements the stupid bits of RFC3484 and is thus not random), and will store the others. Depending on the browser, it will generally use one of the other addresses if the one it's using becomes unavailable. Your DNS servers have to continually monitor all of the sites and stop advertising any that go down. They also need very short TTLs. There are hardware solutions for this -- e.g. F5's BigIP devices.

You'll also need ways to replicate your database, your files, and your users' session states between the datacentres in realtime.

You'll obviously also need to get network diagrams and supplier lists from all of your ISPs to make sure that all your network routes are fully geographically diverse and the ISPs don't rely on the same upstream supplier. It's probably worth making sure they're not on the same power grid, as well.

See here for more information on global server load balancing (although it's a bit old and out-of-date): http://www.tenereillo.com/GSLBPageOfShame.htm

Your failover won't be quite as fast as BGP failover, but you won't be able to bring your site entirely down with a single bad BGP config. You may mess up the configuration of a single DNS server or datacentre, but that won't bring you down completely (unless you push your DNS updates automatically to all of your DNS servers).