Suspend/resume single process to/from disk

Question

On occasion, we would like to suspend memory-intensive processes on our Ubuntu and OS X servers to temporarily free up some RAM for other jobs. If all we were worried about was CPU usage, a simple Ctrl-Z would work. However, we need to be able to free up the RAM (by writing it to disk) and then restart the process (disk -> RAM) or in other words "hibernate" a single process. Any clues on how to do this? (Preferably from CLI.) Thanks.

Solution 1

There is no general facility for hibernating a single process, only the whole system.

However, if you don't care that the process image won't survive a reboot, there is a built-in facility for saving the process image to disk: swapping. Make sure you have enough swap space, and if there is memory pressure and the process is not active (for example because it is suspended), its memory will be swapped out.

If you know that the process is likely to be inactive for a long time and that you're going to need a fast response time at some point, you can force a lot of RAM to be freed now by allocating a lot of RAM in a short-duration process, e.g.

perl -e '$tmp = "a" x 999999999' # allocate about 1GB

You don't get any control on what gets swapped out, so other processes may get swapped out. On Linux, you can swap a process back in by forcibly accessing the pages it maps. The script in this answer will do that: load to memory all the pages mapped by a process (note that this includes open files; you can traverse regions selectively based on the map information to avoid swapping in data that you know you won't need, see this answer for more information).

Solution 2

The term you're looking for is "Application checkpointing".

The tools I know that can do that are CryoPID and CryoPID2.

Both tools are for Linux only.

I don't know a similar tool for BSD or OS X.

Solution 3

Make sure you have plenty of swap space. Make sure your system is set to prefer to swap inactive pages (vm.swappiness=100). Then it should be sufficient to suspend the process. The kernel will prefer to swap the inactive pages out.

Comments

[Admin]

I suppose that when a process is in a suspended state, its used memory is a candidate for been swapped to disk, if needed, and this should be done automatically from the kernel.

[Admin]

@enzotib - I have no idea if this occurs and, in theory, this sounds good. However, I just did a rough test and in practice it doesn't work. Any new processes spawned while 'RAM eaters' are suspended are extremely sluggish and I don't detect any disk activity that would be expected if RAM were being swapped to disk.

[Admin]

@frozenwithjoy I can confirm that this does occur. The only reasons I can think of that a suspended process wouldn't gradually be swapped out is if it's requested unswappable memory (pretty rare, and (mostly) reserved for root), or if the memory it's mapping is shared with another, active process.

[Admin]

A term that might be help here is "checkpointing" (yes, a ugly verb-form). This is generally a difficult problem because the program may be using some resources that could change state while the program is asleep. Managing that problem used to be one of the distinction between Big Iron and little fiddly computers.

[Gilles 'SO- stop being evil']

purge wouldn't help here. It removes the disk cache, but that won't cause a process's memory to be swapped out, and it won't make it faster for another process to obtain RAM.

[Deer Hunter]

Another app to use is DMTCP: dmtcp.sourceforge.net

[flarn2006]

Are you sure there's no mechanism provided by the kernel for manually swapping out specific processes, even as the superuser? I don't know of any but it seems like a reasonable thing to expect.

[Gilles 'SO- stop being evil']

@flarn2006 That would be difficult to implement and that level of control is rarely useful, so no, it's not something you can expect.

[flarn2006]

Ah okay, well hopefully suspending a process will continue working as well to prevent thrashing as it did the first time I tried.