Load balancing for webservers (with state sync)

linux-networkingload balancingnetworking

(I've looked at other questions, I don't think any of them cover the same stuff as this)

I'm looking at a load balancing solution for our farm of webservers. We currently use Cisco CSS11000 series devices, but these have a few limitations:

  • We currently offload SSL, which I'm aware puts load on the LBs, rather than being able to horizontally scale SSL work over multiple webservers
  • The CSS LBs don't support IPv6
  • The CSSs don't work very well for failover – we want to be able to gracefully move user traffic around to let us upgrade webservers without service interruption.

Aims:

  • Let us deactivate half of the webservers in a 'graceful' way (i.e. new user connections go to a specified subgroup of webservers, and it tells us when existing connections are closed – bonus points if it can force them to close gracefully so users don't notice).
  • Let us restart/reconfigure the load balancers without interrupting service
  • Support the usual HA stuff you'd expect (If a single server explodes or crashes, don't break)
  • I'd like to do this without spending $$$ on commercial / hardware load balancers
  • I'd like to do this on Linux if possible to utilise internal experience.
  • The rest of the business likes "Enterprise" stuff because they can blame Someone Else when it goes wrong. So whatever I recommend to put in place needs to be as reliable as a commercial solution.

Ideas:

  • Two pairs of HAProxy. We'd use the HAProxy socket control mechanism (http://code.google.com/p/haproxy-docs/wiki/UnixSocketCommands) to gracefully remove webservers. Each pair would use heartbeat to maintain service, and we'd modify the DNS to point at the other pair to move new user connections over to the other pair. Would need some form of monitoring to tell us when a particular pair had 0 active connections.

  • Two Linux machines doing the balancing with iptables and the -m random module. I'd use heartbeat to keep the HA VIP live on one of the machines, and I'd use conntrackd to synchronise the TCP connection state, so we can failover without loss of service. Would need some scripting to insert/remove iptables rules depending on the state of the backends (Unless anyone knows of a tool?)

Does anyone have any comments on the above? Or any other/better/complimentary ideas?

Thanks!

Best Answer

HAProxy is great (from experience), and I'm not sure if you're aware of more recent features such as syncing the stick table (if you have persistence enabled) using the peers directive. See the 1.5 Manual for more info: http://haproxy.1wt.eu/download/1.5/doc/configuration.txt

HAProxy is also capable of IPv6 -> IPv4 translation; very useful if your internal network is still using IPv4 addressing but your public-facing network isn't.

One thing missing from your list - LVS/IPVS which is part of the Linux kernel. It too is IPv6 capable (though not capable of translation, as not a proxy). Performance is very good, and is often used in conjunction with ldirectord (a Perl daemon) for health-checking and server offline/online functionality. http://horms.net/projects/ldirectord/. You can gracefully reload the ldirectord config file, or use the ipvsadm commands directly to take servers offline, change weights etc.LVS also has sync capabilities: http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.server_state_sync_demon.html

You don't mention whether persistence/stickyness is a requirement, but LVS can stick by source IP, and HAProxy has several options: cookies, source IP, RDP cookies.

Neither perform SSL termination though, this could be handled by Stunnel, Pound, and no doubt several others.

Related Topic