Cassandra: Not enough replica error in single node cluster

Question

Over the weekend, we started seeing errors in Cassandra. Essentially, complaining that it couldn't get enough nodes together for SERIAL consistency.

This appeared to be a problem with AWS vpn across regions. So, to simplify, I dropped one the other node (There were only two nodes at the time). I did this by removing the seed from the last remaining node's cassandra.yaml:

seed_provider:
...
- seeds: "single node ip"

I also ran nodetool removenode on the old node (which was showing a dead state).

So the topology is dead simple. Application is a Java app which connects to the Cassandra node via Java API.

Below is the error I am seeing now:

Caused by: com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive)
        at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:45)
        at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:34)
        at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:182)
        at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:66)
        ... 21 more

Which is the exact same error as before, except node we have QUORUM where we had SERIAL before.

---

I just tried setting the replication factor to 1:

ALTER KEYSPACE my_keyspace WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };

and restarted cassandra. No change.

Also just ran nodetool cleanup keyspace_name based on answer below. Also no change.

---

Just to be clear, the topology is this:

Java App -> Single Cassandra node

Still seeing: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive)

Solution

You mentioned you dropped a node. Given you are using QUORUM and it requires 2 replicas, we can assume you have a replication factor of 2 or 3. How many nodes do you have right now and what is your replication factor? Based on your question I think you are indicating you have 1 left, but I wasn't sure. If you only have 1 node and your RF is 2 or 3, you will never be able to meet quorum consistency.

You could alter your replication factor to 1 to resolve this, i.e.:

ALTER KEYSPACE keyspace_name WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };

If you do this, you should also run nodetool cleanup keyspace_name on each node to get previous replicated data.

Another alternative is to add enough nodes to help you meet a QUORUM consistency level with your replication factor.