Cassandra snapshot restoring : random missing data

backupcassandradata-lossdatabase-backuprestore

I'm having a hard time restoring snapshot on Apache Cassandra (version 3.0.9). As far as I can say, I'm following the procedure described on datastax blog, along with several other ones (for instance : http://datascale.io/cloning-cassandra-clusters-fast-way/). Yet I may be missing something, and everytime I make a restore, data is missing.

Setup :
6 nodes cluster (1 DC, 3 racks with 2 nodes each) with a replication factor set to 3. Machines are hosted on AWS.

Backup procedure (on each node) :

  1. nodetool snapshot mykeyspace
  2. cqlsh -e 'DESCRIBE KEYSPACE mykeyspace' > /tmp/mykeyspace.cql
  3. nodetool ring | grep "$(ifconfig | awk '/inet /{print $2}' | head -1)" | awk '{print $NF ","}' | xargs > /tmp/tokens

I get the files generated by the nodetool snapshot command and backup them along with tokens and cql on S3.

Restore procedure (for each node unless it's specified) :

(after having created new VMs)

  1. Download snapshots, tokens and keyspace
  2. Stop service cassandra
  3. Delete /var/lib/cassandra/commitlog/* and /var/lib/cassandra/system/
  4. Insert tokens into cassandra.yaml
  5. Start service cassandra
  6. Restore mykeyspace from mykeyspace.cql on one node only
  7. Wait for replication and stop service cassandra
  8. Delete .db files in folder /var/lib/cassandra/data/mykeyspace/
  9. For each table copy snapshots files (.db, .crc32, .txt) into /var/lib/cassandra/data/mykeyspace/$table/
  10. Restart service cassandra
  11. Run nodetool repair mykeyspace -full, one node at a time

Result :

There are always missing rows, approximately the same quantity for each table but never the same ones. I tried to "mix up" a bit the procedure, like restoring keyspace before tokens, running nodetool refresh before repair, but I meet the same issue everytime.

Since I'm not far from having a "good" restore, I think that I'm missing something pretty obvious. Analyzing logs didn't really help, as they don't show any error/failure messages.

Any help would be welcomed 🙂 I can of course give more information if needed.

edit : no one ? I updated the question with cassandra version (3.0.9), which I forgot in the first place. I tried again to restore, but no luck. I don't have any more idea really 🙁

Best Answer

The sed command in that blog post, which is supposed to append -Dcassandra.load_ring_state=false to the $JVM_OPTS, has no effect in its current form.

If you were copying that command directly from the blog post, it's possible that's the issue. You could try this one instead which places it at the bottom of the file:

sudo sed -i '$ a\JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"' /etc/cassandra/cassandra-env.sh

Also you'll need to do a nodetool repair -pr <ks> on each node, one by one, after following this procedure.

Related Topic