Increase max number of consumers on RabbitMQ

rabbitmq

I'm running a single instance of RabbitMQ on a fairly big server (32GB RAM, duel SSD, Octa Core Intel, etc) and have a large number of applications publishing and consuming from RabbitMQ without troubles.

This morning while deploying another application, it simply wouldn't start up complaining that it can't connect. I reduced the number of consumers in my application and it connected just fine. Looking at my RabbitMQ dashboard, I can see the consumer count pegged at 900; increasing the number of consumers in my application again and the application is throwing all kinds of exceptions not being able to connect to RabbitMQ.

So looking at the consumer count that's pegged at 900 and the behavior I'm seeing, I'm guessing I've hit some kind of consumer limit.

Server is idling at 1% and looking at Erlang Processes available, I'm seeing 1048576 available, so plenty capacity left on the server.

How do I increase this artificial limit of 900 consumers?

Best Answer

You probably hit the maximum number of file descriptors. I suppose you are using Linux and many distributions set this ulimit to 1024 by default. When RabbitMQ reaches this limit, it can't accept more connections.

The RabbitMQ website has an article on which network-related parameters you can/should tune, in particular how to increase the file descriptors ulimit.

Related Solutions

RabbitMQ message consumers stop consuming messages

I have seen this behavior when using the RabbitMQ STOMP plugin. I haven't found a solution yet.

Are you using the STOMP plugin?

Make RabbitMQ Listen Only to Localhost

Putting the following in /etc/rabbitmq/rabbitmq-env.conf will make RabbitMQ and epmd listen on only localhost:

export RABBITMQ_NODENAME=rabbit@localhost
export RABBITMQ_NODE_IP_ADDRESS=127.0.0.1
export ERL_EPMD_ADDRESS=127.0.0.1

It takes a bit more work to configure Erlang to only use localhost for the higher numbered port (which is used for clustering nodes as far as I can tell). If you don't care about clustering and just want Rabbit to be run fully locally then you can pass Erlang a kernel option for it to only use the loopback interface.

To do so, create a new file in /etc/rabbitmq/ - I'll call it rabbit.config. In this file we'll put the Erlang option that we need to load on run time.

[{kernel,[{inet_dist_use_interface,{127,0,0,1}}]}].

If you're using the management plugin and also want to limit that to localhost, you'll need to configure its ports separately, making the rabbit.config include this:

[ {rabbitmq_management, [ {listener, [{port, 15672}, {ip, "127.0.0.1"}]} ]}, {kernel, [ {inet_dist_use_interface,{127,0,0,1}} ]} ].

(Note RabbitMQ leaves epmd running when it shuts down, so if you want to block off Erlang's clustering port, you will need to restart epmd separately from Rabbit.)

Next we need to have RabbitMQ load this at startup. Open up /etc/rabbitmq/rabbitmq.conf again and put the following at the top:

export RABBITMQ_CONFIG_FILE="/etc/rabbitmq/rabbit"

This loads that config file when the rabbit server is started and will pass the options to Erlang.

You should now have all Erlang/RabbitMQ processes listening only on localhost! This can be checked with netstat -ntlap

EDIT : In older versions of RabbitMQ, the configuration file is : /etc/rabbitmq/rabbitmq.conf. However, this file has been replaced by the rabbit-env.conf file.

Best Answer

Related Solutions

RabbitMQ message consumers stop consuming messages

Make RabbitMQ Listen Only to Localhost

Related Topic