What does "GC (Allocation Failure)" mean in my ElasticSearch 5.6 logs?

Question

I see these constantly in my Elasticsearch 5.6.3 logs. Is this a signal that I should scale up and add more RAM? Or is this just normal operations for ElasticSearch?

    [GC (Allocation Failure) [ParNew
 Desired survivor size 11141120 bytes, new threshold 15 (max 15)
 - age 1: 1761848 bytes, 1761848 total
 - age 2: 126464 bytes, 1888312 total
 - age 3: 165056 bytes, 2053368 total
 - age 4: 50584 bytes, 2103952 total
 - age 5: 105120 bytes, 2209072 total
 - age 6: 99072 bytes, 2308144 total
 - age 7: 2024 bytes, 2310168 total
 - age 8: 95632 bytes, 2405800 total
 - age 9: 24960 bytes, 2430760 total
 - age 10: 62552 bytes, 2493312 total
 - age 11: 95816 bytes, 2589128 total
 - age 12: 54248 bytes, 2643376 total
 - age 13: 50704 bytes, 2694080 total
 - age 14: 856 bytes, 2694936 total
 - age 15: 4736 bytes, 2699672 total
 : 134451K->3691K(152320K), 0.0031179 secs] 375121K->244360K(500480K), 0.0032213 secs] [Times: user=0.02 sys=0.01, real=0.00 secs] 
 [GC (Allocation Failure) [ParNew
 Desired survivor size 11141120 bytes, new threshold 15 (max 15)
 - age 1: 731712 bytes, 731712 total
 - age 2: 447632 bytes, 1179344 total
 - age 3: 120736 bytes, 1300080 total
 - age 4: 165056 bytes, 1465136 total
 - age 5: 50584 bytes, 1515720 total
 - age 6: 105120 bytes, 1620840 total
 - age 7: 98552 bytes, 1719392 total
 - age 8: 1872 bytes, 1721264 total
 - age 9: 94736 bytes, 1816000 total
 - age 10: 24664 bytes, 1840664 total
 - age 11: 62256 bytes, 1902920 total
 - age 12: 95816 bytes, 1998736 total
 - age 13: 53696 bytes, 2052432 total
 - age 14: 4112 bytes, 2056544 total
 - age 15: 856 bytes, 2057400 total
 : 134251K->3308K(152320K), 0.0042321 secs] 374920K->243982K(500480K), 0.0043257 secs] [Times: user=0.02 sys=0.01, real=0.00 secs] 

Solution

GC (Allocation Failure) is a JVM message (not an Elasticsearch-specific one) that can be a sign that there's memory pressure, but it's not catastropic to the JVM (that would cause a OutOfMemoryError log line). It can also be completely innocuous.

GC (Allocation Failure) means that the Java garbage collector tried to run, ran out of space in the heap, then tried to allocate more memory. It's not a bad sign, necessarily. If you're receiving OutOfMemoryError errors and the JVM is crashing, then you know you're in trouble.

Side note / disclaimer / full disclosure / whatever the heck you want: I work for a DBaaS company that hosts Elasticsearch clusters.