Elasticsearch service often goes down or gets killed

Question

My webserver is running ubuntu 14.10 with elasticsearch 1.5.0 and java 1.7u55

For some reason, the elasticsearch service often goes down, resulting in my website not being available to my users anymore (using FOSElasticaBundle with symfony).

I am using systemctl to restart it automatically, but I would prefer a good fix once and for all. I feel the logs I have are not descriptive enough. Being pretty new to managing a server, I need some help.

Can someone help me figure out the reason for this failure ? What are the right files I can output here to better understand the issue ?

Thanks !

My systemctl status gives :

elasticsearch.service - ElasticSearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
   Active: active (running) since Mon 2015-04-20 12:04:24 CEST; 1h 56min ago          <------------- Here it means restarted 1h56 ago. Why did it fail in the first place ?
 Main PID: 9120 (java)
   CGroup: /system.slice/elasticsearch.service
           └─9120 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingO...

In my journalctl, I have :

Apr 18 18:56:19 xx.ovh.net sshd[29397]: error: open /dev/tty failed - could not set controlling tty: Permission denied
Apr 20 13:52:45 xx.ovh.net sshd[9764]: error: open /dev/tty failed - could not set controlling tty: Permission denied

EDIT: it often restarts when I launch the first search request:

elasticsearch.service - ElasticSearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
   Active: activating (auto-restart) (Result: signal) since Tue 2015-04-21 12:27:43 CEST; 10s ago
  Process: 15618 ExecStart=/usr/share/elasticsearch/bin/elasticsearch (code=killed, signal=ABRT)
 Main PID: 15618 (code=killed, signal=ABRT)

Solution 1

Well, due to low rep I cannot comment, but one elasticsearch instance needs at least 1.5 gB to run comfortably, maybe a lot more depending on what you're asking of it. By default elastic takes 1 gB of memory and then lucene (the search library that ES leverages) will take up some other memory to perform searches. Below are a couple links from elastic's site about managing memory and stopping your process from being killed by OOM killer because of its memory usage.

https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html#setup-configuration-memory

https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html

Solution 2

When it happens next check free - m and monitor the Elasticsearch logs at /var/log/elasticsearch/your cluster name.log.

Comments

[EEAA]

Is your system going OOM? Check the output of dmesg.

[Sébastien]

well, it just got killed and there is no log edited on this day in this folder

[NerdyBiker]

How much ram has your machine for? Can you check free -m just before it restarts again?

[Sébastien]

it was actually a memory issue. i've upgraded my server and it does not get killed anymore