RabbitMQ auto-configuration of cluster

ansiblerabbitmq

The RabbitMQ clustering document

https://www.rabbitmq.com/clustering.html

describes a process whereby you can deploy a configuration file for clustering so that a cluster is automatically created when your rabbitmq nodes boot.eg

[{rabbit,
  [{cluster_nodes, {['rabbit@rabbit1', 'rabbit@rabbit2', 'rabbit@rabbit3'], ram}}]}].

You apply this file on each of the 3 nodes at /etc/rabbitmq/rabbitmq.config, and then start rabbitmq on each and a cluster will apparently form.

This doesn't appear to work too well eg

If you start rabbit2, and rabbit3 hasn't already come up, the service will not start on rabbit2, as it is trying to create a cluster with rabbit3.

Equally, if you only apply the config on rabbit1, and have pre-started rabbit2 and rabbit3, rabbit1 will form a cluster with rabbit2 only, as, according to the documentation (I don't understand why):

RabbitMQ will try to cluster to each node provided, and stop after it can cluster with one of them.

The only way this seems to work is if you apply the config file on all 3 nodes, and have them start at exactly the same time. I am trying to deploy this with Ansible, which creates the nodes sequentially, so that doesn't work.

What am I missing here?

Best Answer

I also deploy RabbitMQ clusters with Puppet, and you actually don't have to spin up all nodes at the exactly the same time.

What I usually do, and has so far worked for me is:

  • install RabbitMQ (RPM or DEB)
  • set up hosts file on each node, to conatins entries for all the three. example:

.

192.168.1.11    dev-c1n01-rabbitmq.example.com  dev-c1n01-rabbitmq
192.168.1.12    dev-c1n02-rabbitmq.example.com  dev-c1n02-rabbitmq
192.168.1.13    dev-c1n03-rabbitmq.example.com  dev-c1n03-rabbitmq

nodes we're clustering together (because I don't want to rely on DNS being available) * deploy rabbitmq.config

.

[
  {rabbit, [
    {cluster_nodes, {['rabbit@dev-c1n01-rabbitmq', 'rabbit@dev-c1n02-rabbitmq', 'rabbit@dev-c1n03-rabbitmq'], disc}},
    {cluster_partition_handling, pause_minority},
    {disk_free_limit, 2147483648},
    {heartbeat, 0},
    {tcp_listen_options, [binary, {backlog, 1024}, {nodelay, true}, {keepalive, true} ]},
    {vm_memory_high_watermark, 0.6},
    {default_user, <<"admin">>},
    {default_pass, <<"somedefaultpass">>}
  ]},
  {kernel, [

  ]}
,
  {rabbitmq_management, [
    {listener, [
      {port, 15672}
    ]}
  ]}
].
% EOF
  • deploy erlang.cookie

To create erlang cookie, I usually use http://passwordsgenerator.net/ and set it up to create 20 character string consistent only of uppercase characters. Then, put this string into /var/lib/rabbitmq/.erlang.cookie, like this:

echo -n 'LCQLSHVOPZFHRUXMMAPF' > /var/lib/rabbitmq/.erlang.cookie
  • start nodes (order doesn't matter as long as they have same erlang.cookie and rabbitmq.config)

This should work for you. Tested on 3.2, 3.3 and 3.5 versions.

Related Topic