Cassandra nodetool connection timed out

Question

Im trying to use nodetool to check the status of my cluster, but its unable to connect.

My cassandra.yaml is configured with listen_address and rpc_address set as the server IP (e.g. 10.10.10.266).

Im able to connect through cqlsh and cassandra-cli using the same IP, but when I connect to nodetool it doesnt work.

/bin$ nodetool -h 10.10.10.266 ring
Failed to connect to '10.10.10.266:7199': Connection has timed out

I dont think I have a firewall enabled on the server (Ubuntu). Im running this directly on the server in question, so I wouldnt have thought it would be a firewall issue anyway.

Solution 1

You probably need to uncomment the following parameter in cassandra-env.sh:

-Djava.rmi.server.hostname=<public name>

Replace with the address of the interface you want the jmx interface to listen on.

Solution 2

nodetool connects through JMX interface. By default it's listening on port 7199 (other tools use RPC interface listening on port 9160 by default). Check JMX settings in cassandra-env.sh file. Most likely JMX server is listening on wrong interface (or probably loopback interface).

Default JMX configuration section (cassandra ver. 1.1.5) contains link to troubleshooting guide:

# jmx: metrics and administration interface
# 
# add this if you're having trouble connecting:
# JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<public name>"
# 
# see 
# https://blogs.oracle.com/jmxetc/entry/troubleshooting_connection_problems_in_jconsole
# for more on configuring JMX through firewalls, etc. (Short version:
# get it working with no firewall first.)
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT" 
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false" 
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false" 
JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS"

It also worths to list all network interfaces using ifconfig and try telnet'ing port 7199 on all interfaces.

Solution 3

I was facing the same timeout issue. However I found that my cluster was not getting started properly because of token issue and I was getting "Host ID collision between active endpoint". Once i deleted data directory and restarted cluster then nodetool started working fine.

Solution 4

I also saw this same issue but it turned out to be some weirdness in my hosts file that was preventing JMX from binding to the interfaces.

Specifically, the host file had an entry for the external IP address with the hostname. Our servers had two interfaces, one external and one for an internal network. Removing that hosts entry did the trick.

View more solutions