Elasticsearch won't start anymore

Question

I restarted my elasticsearch instance 5 days ago and I haven't manage to start it since then.

I get no output in the log file /var/log/elasticsearch/ nor does the elasticsearch binary print any information when running at using elasticsearch -f.

I once manage to get this output.

[2012-11-15 22:51:18,427][INFO ][node                     ] [Piper] {0.19.11}[29584]: initializing ...
[2012-11-15 22:51:18,433][INFO ][plugins                  ] [Piper] loaded [], sites []

Running curl http://localhost:9200 resulted in curl: (7) couldn't connect to host.

I've tried increasing the memory from 3gb to 10gb, but that didn't make any diffrence.

Running /etc/init.d/elasticsearch start takes 30 seconds. ps aux | grep elasticsearch results in this output.

/usr/local/share/elasticsearch/bin/service/exec/elasticsearch-linux-x86-64 /usr/local/share/elasticsearch/bin/service/elasticsearch.conf wrapper.syslog.ident=elasticsearch wrapper.pidfile=/usr/local/share/elasticsearch/bin/service/./elasticsearch.pid wrapper.name=elasticsearch wrapper.displayname=ElasticSearch wrapper.daemonize=TRUE wrapper.statusfile=/usr/local/share/elasticsearch/bin/service/./elasticsearch.status wrapper.java.statusfile=/usr/local/share/elasticsearch/bin/service/./elasticsearch.java.status wrapper.script.version=3.5.14

/usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java -Delasticsearch-service -Des.path.home=/usr/local/share/elasticsearch -Xss256k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Xms1024m -Xmx1024m -Djava.library.path=/usr/local/share/elasticsearch/bin/service/lib -classpath /usr/local/share/elasticsearch/bin/service/lib/wrapper.jar:/usr/local/share/elasticsearch/lib/elasticsearch-0.19.11.jar:/usr/local/share/elasticsearch/lib/elasticsearch-0.19.11.jar:/usr/local/share/elasticsearch/lib/jna-3.3.0.jar:/usr/local/share/elasticsearch/lib/log4j-1.2.17.jar:/usr/local/share/elasticsearch/lib/lucene-analyzers-3.6.1.jar:/usr/local/share/elasticsearch/lib/lucene-core-3.6.1.jar:/usr/local/share/elasticsearch/lib/lucene-highlighter-3.6.1.jar:/usr/local/share/elasticsearch/lib/lucene-memory-3.6.1.jar:/usr/local/share/elasticsearch/lib/lucene-queries-3.6.1.jar:/usr/local/share/elasticsearch/lib/snappy-java-1.0.4.1.jar:/usr/local/share/elasticsearch/lib/sigar/sigar-1.6.4.jar -Dwrapper.key=k7r81VpK3_Bb3N_5 -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=23888 -Dwrapper.version=3.5.14 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.tanukisoftware.wrapper.WrapperSimpleApp org.elasticsearch.bootstrap.ElasticSearchF

My current system:

• ElasticSearch Version: 0.19.11, JVM: 23.2-b09
• Ubuntu 12.04 LTS
• OpenJDK 64-Bit Server VM - 1.7.0_09

I've tried re-installing elasticsearch, removing old directories, reinstalling java and restarting the server.

Why can't I get it to start?

Solution

The solution was to increase the amount of cores on the VPS from one to two.

Comments

[Driftpeasant]

Have you tried clearing the log entries? We had a similar issue (albeit on Windows with Elastisearch) where we had to clear the log directory before it would reinstall/restart.

[Linus Oleander]

Yes. I've done that.

[kms]

Could you run an strace -fF -p <pid_of_elasticsearch>? this may alude to a wait( syscall, if this is wait(11 for example, a lsof -p <pid_of_elasticsearch> should be able to tell you what the process is waiting on.

[David L]

Why does that cause a problem?

[Linus Oleander]

I've no idea why that worked. Sorry.