Haproxy – Correct way of queuing + draining requests

haproxyload balancing

Context

I'm trying to implement a deployment script which implements the following idea:

  1. Start queuing new incoming requests, while waiting for current requests to be finished
  2. Wait for all current requests to finish (I think this is called "draining")
  3. Run app-specific deployment script
  4. Process all requests that were queued in step (1) and get haproxy back to normal. No incoming connections should be dropped by haproxy. If the client times out, that is acceptable.

Question

Given this context, I can find a number of ways to implement this in the haproxy docs:

  • set server mybackend/myserver state drain followed by set server mybackend/myserver state ready
  • set maxconn frontend myfrontend 0 followed by set maxconn frontend myfrontend 1000
  • set maxconn backend mybackend/myserver 0 followed by set maxconn backend mybackend/myserver 1000

Which of these is the correct way of implementing what I'm trying to implement?

More context

This is probably related to https://serverfault.com/a/450983/117598 , but the following from haproxy docs is causing me to re-confirm:

Sets the maximum per-process number of concurrent connections to . It is equivalent to the command-line argument "-n". Proxies will stop accepting connections when this limit is reached. [..]

vs another conflicting snippet:

The "maxconn" parameter specifies the maximal number of concurrent
connections that will be sent to this server. If the number of incoming
concurrent requests goes higher than this value, they will be queued, waiting for a connection to be released. [..]

Best Answer

Expand your design considerations beyond the network proxy.

As mentioned in a blog post based on that answer you linked, the usual way of doing things involves data conversions with backward compatability:

Normally when doing a database migration online, the correct way to do this involves multiple steps:

  • perform schema changes in a way that won’t break existing code (e.g., temporarily allowing new non-nullable columns to be NULL).
  • deploy code that works with both the old and new schema simultaneously, populating any new rows according to the new schema.
  • perform a data migration that backfills any old data correctly, and updates all constraints properly.
  • deploy code that only expects to see the new schema.

During the transition, consider if your application should pause processing. Incoming data could be stored in a queue or database, and worked once processing was resumed.

An advantage this has over holding open network connections is the luxury to take your time. Network time outs may be very short.

Related Topic