RabbitMQ appearing to crash on startup

rabbitmqubuntu-16.04

How do you diagnose RabbitMQ crash issues on Ubuntu 16?

When I run sudo service rabbitmq-server status it reports:

● rabbitmq-server.service - RabbitMQ Messaging Server
   Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
   Active: failed (Result: timeout) since Wed 2018-03-21 19:44:18 UTC; 19min ago
  Process: 1100 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait (code=killed, signal=TERM)
  Process: 1099 ExecStart=/usr/sbin/rabbitmq-server (code=killed, signal=TERM)
 Main PID: 1099 (code=killed, signal=TERM)

implying it's crashed or failed to start. However, when I run htop, I see dozens of erlang and beam.smp processes, which are launched by Rabbit.

Furthermore, when I go to restart Rabbit with sudo service rabbitmq-server restart it hangs for about five minutes and then finally returns with:

Job for rabbitmq-server.service failed because a timeout was exceeded. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.

When I run journalctl -xe I see a ton of messages like:

Mar 21 20:07:48 server1 postfix/error[3719]: 280524B3A: to=<root@mydomain.com>, orig_to=<root>, relay=none, delay=101268, delays=101268/0/0/0, dsn=4.4.1, status=deferred (delivery temporarily suspende
Mar 21 20:07:48 server1 postfix/qmgr[1784]: 2D046FAC: from=<>, size=3126, nrcpt=1 (queue active)
Mar 21 20:07:48 server1 postfix/qmgr[1784]: 2D8AD474F: from=<root@mydomain.com>, size=751, nrcpt=1 (queue active)
Mar 21 20:07:48 server1 postfix/error[3712]: 2ED9D499A: to=<root@mydomain.com>, orig_to=<root>, relay=none, delay=155868, delays=155868/0/0/0, dsn=4.4.1, status=deferred (delivery temporarily suspende
Mar 21 20:07:48 server1 postfix/qmgr[1784]: 2EBCF3D40: from=<>, size=3128, nrcpt=1 (queue active)
Mar 21 20:07:48 server1 postfix/error[3706]: 2D8AD474F: to=<root@mydomain.com>, orig_to=<root>, relay=none, delay=38268, delays=38268/0/0/0, dsn=4.4.1, status=deferred (delivery temporarily suspended:
Mar 21 20:07:48 server1 postfix/error[3716]: 2D046FAC: to=<root@mydomain.com>, relay=none, delay=76240, delays=76240/0/0/0, dsn=4.4.1, status=deferred (delivery temporarily suspended: connect to porta
Mar 21 20:07:48 server1 postfix/qmgr[1784]: 2C9DE3945: from=<>, size=3134, nrcpt=1 (queue active)
Mar 21 20:07:48 server1 postfix/qmgr[1784]: 2AA2A48B3: from=<root@mydomain.com>, size=751, nrcpt=1 (queue active)
Mar 21 20:07:48 server1 postfix/error[3717]: 2C9DE3945: to=<root@mydomain.com>, relay=none, delay=399644, delays=399644/0/0/0, dsn=4.4.1, status=deferred (delivery temporarily suspended: connect to po
Mar 21 20:07:48 server1 postfix/error[3701]: 2EBCF3D40: to=<root@mydomain.com>, relay=none, delay=181242, delays=181242/0/0/0, dsn=4.4.1, status=deferred (delivery temporarily suspended: connect to po
Mar 21 20:07:48 server1 postfix/error[3712]: 2AA2A48B3: to=<root@mydomain.com>, orig_to=<root>, relay=none, delay=59268, delays=59268/0/0/0, dsn=4.4.1, status=deferred (delivery temporarily suspended:

Am I correct in concluding Rabbit is trying to send a ton of email, is being blocked, and is subsequently crashing? Why is this?

Best Answer

I fixed it with:

sudo killall rabbitmq-server
sudo killall beam.smp
sudo rm -Rf /var/lib/rabbitmq/mnesia/*
sudo service rabbitmq-server start

I also had to re-add my user configurations, but otherwise, that brought it back up.

Related Solutions

Make RabbitMQ Listen Only to Localhost

Putting the following in /etc/rabbitmq/rabbitmq-env.conf will make RabbitMQ and epmd listen on only localhost:

export RABBITMQ_NODENAME=rabbit@localhost
export RABBITMQ_NODE_IP_ADDRESS=127.0.0.1
export ERL_EPMD_ADDRESS=127.0.0.1

It takes a bit more work to configure Erlang to only use localhost for the higher numbered port (which is used for clustering nodes as far as I can tell). If you don't care about clustering and just want Rabbit to be run fully locally then you can pass Erlang a kernel option for it to only use the loopback interface.

To do so, create a new file in /etc/rabbitmq/ - I'll call it rabbit.config. In this file we'll put the Erlang option that we need to load on run time.

[{kernel,[{inet_dist_use_interface,{127,0,0,1}}]}].

If you're using the management plugin and also want to limit that to localhost, you'll need to configure its ports separately, making the rabbit.config include this:

[ {rabbitmq_management, [ {listener, [{port, 15672}, {ip, "127.0.0.1"}]} ]}, {kernel, [ {inet_dist_use_interface,{127,0,0,1}} ]} ].

(Note RabbitMQ leaves epmd running when it shuts down, so if you want to block off Erlang's clustering port, you will need to restart epmd separately from Rabbit.)

Next we need to have RabbitMQ load this at startup. Open up /etc/rabbitmq/rabbitmq.conf again and put the following at the top:

export RABBITMQ_CONFIG_FILE="/etc/rabbitmq/rabbit"

This loads that config file when the rabbit server is started and will pass the options to Erlang.

You should now have all Erlang/RabbitMQ processes listening only on localhost! This can be checked with netstat -ntlap

EDIT : In older versions of RabbitMQ, the configuration file is : /etc/rabbitmq/rabbitmq.conf. However, this file has been replaced by the rabbit-env.conf file.

Unable to start rabbitmq

Your error message is right there in what you pasted: "error,{future_upgrades_found".

Typically this is because you upgraded the version of RabbitMQ on your system and then (a) subsequently downgraded but didn't blow away the database, or (b) tried to run an older version of RabbitMQ against the upgraded database.

Re-Create your database (or upgrade to the appropraite version of RabbitMQ that the DB was created with) and the problem will go away.

Best Answer

Related Solutions

Make RabbitMQ Listen Only to Localhost

Unable to start rabbitmq

Related Topic