First of all, apologies if this has been asked before, I cannot find an answer on Google or StackExchange.
Is it possible to put Apache Cassandra into a split-brain scenario where a network partitioned data service continues to run storing and accessing data without access to the full cluster?
If not, what are the techniques used (links to documentation OK) to avoid a split-brain?
If so, (there seems to be tweets about this happening) what are the recovery options in this case?
Best Answer
So, it really depends on a few things.
First off, Cassandra is designed to be partition tolerant, which means it's meant to continue to work under situations like you're describing. For instance, you might have two datacenters defined, and the network connection drops between them. Whether or not your queries return successfully depends on the consistency level that you query your cluster for. If you choose LOCAL_QUORUM, you will get a result back even if the 2 datacenters can't communicate. This is an intended feature of the database.
Each node in the cluster maintains the full topology of the cluster, so when the network connection is resolved, hinted handoff will kick in and the datacenters will resolve their issues. It may be necessary to run nodetool repair to ensure all data is consistent at this point. (it can't hurt)