Linux reboot out of memory

Question

I have a server with Intel(R) Atom(TM) CPU D525 and 1 GB memory. I noticed the server would shut down and restart automatically about every 7 days.

I checked the memory usage and found that when the memory usage reached 90%, the kernel reboots. When I checked the kernel log in /var/log/messages file, I didn't find anything about the kernel shutting down, just a message about the kernel start. I checked the file /proc/sys/vm/min_free_kbytes, the value is "3765".

---

I guess when the available memory is very low, but doesn't get to the number the system starts reclaiming memory. Then the kernel can't do anything, so it then reboots.

Can you give me some insight?

Solution 1

On some demand-paged virtual memory systems, the operating system refuses to allocate anonymous pages (i.e. pages containing data without a filesystem source such as runtime data, program stack etc.) unless there is sufficient swap space to swap out the pages in order to free up physical memory. This strict accounting has the advantage that each process is guaranteed access to as much virtual memory they allocate, but is also means that the amount of virtual memory available is essentially limited by the size of the swap space.

In practice, programs tend to allocate more memory than they use. For instance, the Java Virtual Machine allocates a lot of virtual memory on startup, but does not use it immediately. Memory accounting in the Linux kernel attempts to compensate for this by tracking the amount of memory actually in use by processes, and overcommits the amount of virtual memory. In other words the amount of virtual memory allocated by the kernel can exceed the amount of physical memory and swap space combined on the system. While this leads to better utilization of physical memory and swap space, the downside is that when the amount of of memory in use exceeds the amount of physical memory and swap space available, the kernel must somehow free memory resources in order to meet the memory allocation commitment.

The kernel mechanism that is used to reclaim memory to fill the overcommitment is called the out-of-memory-killer (OOM-killer). Typically the mechanism will start killing off memory-hogging "rogue" processes to free up memory for other processes. However, if the vm.panic_on_oom sysctl setting is non-zero, the kernel will panic instead when the system runs out of memory.

The possible values for the vm.panic_on_oom setting are as follows:

• 0 (default) When an out-of-memory situation arises, the OOM-killer will kill a rogue process.

• 1 The kernel normally panics, but if process that has reached its memory allocation limit set with mbind(MPOL_BIND) or cpuset, the process is killed instead.

• 2 The kernel always panics in an out-of-memory situation.

The heuristic used by the OOM-killer can be modified through the vm.oom_kill_allocating_task sysctl setting. The possible values are as follows:

• 0 (default) The OOM-killer will scan through the task list and select a task rogue task utilizing a lot of memory to kill.

• 1 (non-zero) The OOM-killer will kill the task that triggered the out-of-memory condition.

The kernel memory accounting algorithm can be tuned with the vm.overcommit_memory sysctl settings. The possible values are as follows:

• 0 (default) Heuristic overcommit with weak checks.

• 1 Always overcommit, no checks.

• 2 Strict accounting, in this mode the virtual address space limit is determined by the value of vm.overcommit_ratio settings according to the following formula:

  virtual memory = (swap + physical memory * (overcommit_ratio / 100))
  

When strict memory accounting is in use, the kernel will no longer allocate anonymous pages unless it has enough free physical memory or swap space to store the pages. This means it is essential that the system is configured with enough swap space.

The sysctl settings can be checked or modified at runtime with the sysctl command. To make changes permanent the settings can be written to /etc/sysctl.conf. The above settings are also available via the /proc/sys/vm interface. The corresponding files are:

• /proc/sys/vm/panic_on_oom

• /proc/sys/vm/oom_kill_allocating_task

• /proc/sys/vm/overcommit_memory

• /proc/sys/vm/overcommit_ratio

Solution 2

• I think that you should find out the process consumed lot of memory.Maybe the program has a memory leak bug.You should fix it.
• If there is no enough memory,Linux will shutdown some process consumed lot of memory,or reboot itself.Because kernel must use some memory.
• The different behavior is determined by config.But I think that you should fix the memory leak bug.

Comments

[poige]

Very poorly worded question.

[jofox]

means the number the system starts reclaiming memory with "...,but don't get to the value kernel recover,..."

[innocent-world]

$ man swapon - swapon, swapoff - enable/disable devices and files for paging and swapping

[psusi]

It is perfectly normal for buffers/cache to consume nearly all of memory. Your problem is unrelated to memory.

[jofox]

"The different behavior is determined by config", what config do you mean?

[Edward Shen]

I do not remember.

[jofox]

I'm sure that there is not the memory leak bug. Why does Linux reboot itself but not reclaim memory when there is no enouth memory?

[jofox]

Thank you very much! But the sever just shut down and started, I find nothing about oom in /var/log/messages file. I find the buffer size is growing and growing, never go down!

[jofox]

I have tested. The physical memory is ok.