We have an Autoscaling Group in AWS that we manually scale up and down at the time being. Our ASG is currently attached to one Target Group, which has all of our prod servers in it.
Before using the ASG, we would add and remove servers to the Target Groups manually from the Target Group console, and the nodes would deregister after draining properly. We have a very stateful application so the deregistration delay is really important for us.
Yesterday when scaling down through the ASG interface (specifically by asking for a lower number of desired instances), all the connections dropped instantly, which dropped hundreds of connections instantly instead of honoring the five minute draining policy of the target group.
How do I make my ASG honor my draining policy?
I have tried this:
AWS ASG With Application LB and Connection Draining but it does not set the connections to "draining", but just "terminating: waiting", and my health checks are healthy so I don't think it is actually stopping new connections/draining.
Best Answer
As described in the linked answer you will need an ASG Lifecycle hook to start with.
Whenever the Terminating event occurs fire up a Lambda function and in that Lambda deregister your instance from the Target Group using deregister-targets. That should move it to Draining phase, then wait however long you need to wait and once the instance is drained continue with the termination.
Hope that helps :)