Cassandra node discovery

cassandra

I just set up a 3 node system with ip addresses "192.168.0.101", "192.168.0.102", "192.168.0.103". I have set the seeds value on the configuration of all three machines as
seeds: "192.168.0.101,192.168.0.102,192.168.0.103"

However, on running nodetool on any of them, only the 103 machine gets discovered.

node101:/opt/cassandra/apache-cassandra-0.8.5/bin# ./nodetool
ring -h 192.168.0.101 Address DC Rack Status
State Load Owns Token

192.168.0.103 datacenter1 rack1 Up Normal 151.96 KB
100.00% 38174485210079977599903748344879358256

Could someone tell me where the problem lies?

Best Answer

Seeds are really only important when bootstrapping new machines into a cluster, and they should only be set to mature members of the cluster.

In a three-node cluster, I usually:

  1. Fully bring up the first node. Thrift and nodetool should both work with it.
  2. Install Cassandra on the second and third nodes.
  3. Stop Cassandra on the second and third nodes and wipe the data (if you're on a Debian-style distro where installing a package starts the service)
  4. Configure the second and third nodes to use the first as a seed.
  5. Configure the listen addresses one all three nodes to be IPs that the other nodes can reach.
  6. Start the second node.
  7. Start the third node.
  8. Watch /var/log/cassandra/system.log on each node for notifications about others joining the ring.

In your case (not seeing discovery), I would double-check the following:

  • That the listen address on each node is ping-able from each other node.
  • That the cluster name is the same on all three nodes.
  • That no node complains about the cluster name on disk not matching the configured one.
  • That nodes mention in system.log that they're hearing about other nodes coming online.

Also, please run the nodetool ring command on each node and see if they all think they're members of single-node rings.

Related Topic