Docker – Rolling update behind HAProxy

consuldockerhaproxyload balancing

I am using an infrastructure platform consisting of docker containers running on a CentOS host.

Scaling multiple containers through docker compose.
Consul for service discovery.
Registrator for service registry
HAProxy as a load balancer
Consul-template for generating haproxy configuration each time the container is added/removed.

Each time we update our build, new containers are spawned and old containers are removed. The problem is containers are removed instantaneously without regards to the existing requests being served by the containers. Ideally, I should have a signal to mark the containers in maintenance or drain mode so that existing connections can be served and then gracefully removed from haproxy first and then physically.

Has anyone faced this problem before or done the similar autoscaling workflow through containers.

Best Answer

From my understanding of what you're asking, you have control over all the steps of this process and want to bring up new versions of a containerized application which sit behind HAProxy without interrupting client connections to the old applications, but preventing new connections.

This seems like a good use of the HAProxy disable server command.

You should be able to disable your deprecated servers, marking them as down for maintenance in HAProxy, which should gracefully allow clients to finish their business, then your consul/HAProxy magic can bring the new servers online.

Related Solutions

How to set up stats for HAProxy

That is not correct. There would be a stats page on http://yoursite.com:1935/haproxy?stats

To overcome this add another listener-

listen stats :1936
    mode http
    stats enable
    stats hide-version
    stats realm Haproxy\ Statistics
    stats uri /
    stats auth Username:Password

Then go to http://yoursite.com:1936/ and login.

Make sure your firewall isn't blocking this request.

Tomcat – Rolling Deploys with Tomcats Behind HAProxy

I sent an e-mail to Willy Tarreau and got back the following options:

You can use hot reconfiguration or socat commands to the stats server to set the weight of your target server to 0. This will prevent new sessions from being balanced to that server, but won't affect existing connections.
You can set the http-check disable-on-404 option in combination with "option httpcheck /myurl" and then have your target servers respond to /myurl such that they send a 200 status if everything is fine, a 404 if the server should stop receiving new requests, and a 500 if the server should receive nothing (i.e. when you're ready to bounce the server). haproxy will re-check on the server at the interval specified in your server line.

Best Answer

Related Solutions

How to set up stats for HAProxy

Tomcat – Rolling Deploys with Tomcats Behind HAProxy

Related Topic