CentOS, kernel: out of memory

Question

I have manually compiled gammu-1.42.0 on raspbian via: ./confugure make make install and copied the gammu-smsd-init-script to /etc/init.d/ as gammu-smsd. When I start gammu with systemctl start gammu-smsd it does not seem to start the daemon. systemctl status gammu-smsd shows: Active: active (exited) instead of Active: active (running). Any help would be greatly appreciated.

Solution

It's been a while since I had to read an OOM-killer log, but as I recall, this

Jun 28 05:06:23 crushftp kernel: Killed process 1491, UID 0, (java) total-vm:9620220kB, anon-rss:3245824kB, file-rss:128kB

means that java was using 9GB of VM when the OOM-killer shot it in the head. Given that you have 4GB of core, and 4GB of swap, that seems like a reasonable thing to do. You then write

When I understand it correct, the value must be ok and if the process needs more RAM it should be able to get it.

which I don't understand.

Firstly, setting that value to 0 doesn't turn off overcommitment. As Red Hat write, setting this to 0 requires that the

kernel performs heuristic memory overcommit handling by estimating the amount of memory available and failing requests that are blatantly invalid. Unfortunately, since memory is allocated using a heuristic rather than a precise algorithm, this setting can sometimes allow available memory on the system to be overloaded.

Setting it to 2 does what you seem to want:

The kernel denies requests for memory equal to or larger than the sum of total available swap and the percentage of physical RAM specified in overcommit_ratio. This setting is best if you want a lesser risk of memory overcommitment.

But even turning off overcommit doesn't guarantee that a process can always get more RAM: only infinite VM guarantees that. As long as core+swap is finite, it can be used up - and if you have a process that's comsumed all the free VM at the moment the kernel needs a bit more, then the OOM-killer will wake up, and, well, that looks like what happened.

My recommendations are:

1. Don't run java as root. Ideally, don't run it at all, but if you must, not as root; that gives it a weighting in the OOM-killers eyes which may result in something important getting killed instead.

2. Find the memory leak in whatever's using java.

3. If you really believe you don't have a memory leak, then you don't have enough core; pony up for a bigger server. Give it more swap, as well.

4. Monitor your java's VM footprint better; shoot it in the head if it gets all swollen.

Comments

[dawud]

You have yet to show the heap assigned to this java process when it starts up. It looks like it's been started with an appropriate -Xms value, but a wrong -Xmx. You should be getting an OOM from the Java process, never from the operating system.

[user3772108]

Hello and thanks for your answer. This are the start values of the Java program. I am no Java expert so I don't know if something is wrong. java -Ddir=/var/opt/CrushFTP7_PC/ -Xmx1024M -jar plugins/lib/CrushFTPJarProxy.jar -d

[user3772108]

Thanks for your great answer! :) I meant that the /proc/sys/vm/overcommit_memory settings are 0 which should be the default value as I had read. I read it twice but the part with the "infinite VM garantees"... I don't understand it really. The problem is that the program must be running as root. Maybe I can ask the developers for a workaround but the start script is checking for that. The developer just answered me that the Java program should only use something about 512MB RAM. So it must be a memory leak when it is suddenly using 9GB for no reason on a Sunday.

[MadHatter]

The bit about infinite VM was in response to your assertion that if the process needs more RAM it should be able to get it - only infinite resources can guarantee that all requests can be satisfied. A decent monitoring system (see point 4) should give you more confidence that your java's VM footprint is normally ~0.5GB; without that, it's hard to know what you should say to your developer. And I really don't understand why java needs to run as root; anyone telling you that may be selling snake oil.

[Michael Hampton]

An FTP server needs to run as root in order to: (1) bind to ports 21 and 20; (2) switch to different users. As for Java, I haven't the slightest idea why anyone would write an FTP server in Java. It's just about the worst language for such a thing.

[MadHatter]

I agree. I confess that I hadn't realised that the Java process in question was the FTP server - it's such a stupid idea that I'd assumed the FTP server was running on the server along with this java thing, and the java thing was treading on its feet. user3772108, I don't think you can expect any easy fix for this, save careful monitoring, and a controlled response when the VM footprint starts to grow.

[user3772108]

It seems like Java is really not the best FTP solution. Normally I know (s)FTP's written in PHP with some Javascript. But the server was already there when I started in this company. Anyway thanks for your help. :)