I'm having some problems with RabbitMQ and dead connections. I have a RabbitMQ server and many ec2 instances that connect and comsumes messagens from this server. The problem is that sometimes when I need to terminate an ec2 instance, its conection with RabbitMQ server is not closed.
Because of this we start to see unacked messages that never get back to the ready state. What I currenlty do is use rabbitmqctl to remove this connections and/or use a python script I wrote to kill any connection that is idle for more than X days.
I already tried using the heatrbeat[1] RabbitMQ option, but the problem still occurs. What I would like to do is to automatically close any "dead" (idle for a long time) connections.
Is this something I set in the client side, instead of the server side?
RabbitMQ is 3.3.4 and we use py-amqp[2] on the clients.
[1] https://www.rabbitmq.com/configure.html
[2] https://github.com/celery/py-amqp
Best Answer
We saw similar problems and we are using a PHP client which does not support heartbeat.
Our solution is to shut down all consumer processes before the instances are terminated, so all connections are disconnected cleanly. AFAIK EC2 will attempt a shutdown, and you will have a minute or so before the machine is powered off.
Example :
[1] https://unix.stackexchange.com/questions/48973/execute-a-command-before-shutdown