How can I dump the full system memory?

Question

After starting VirtualBox, the computer became sluggish and then hung completely due to OOM. Usually, OOM should be starting killing processes in order to free up some space, but this did not happen (this was the second time that I experienced this).

I had some unsaved important work in a text editor, so I was hoping to find it back in the system RAM after killing all processes in the current console using SysRq + K. The machine in question is a laptop with 8 GiB RAM running Linux x86_64 3.7.5 with a SSD as target disk.

My first attempt was dd if=/dev/mem of=memory, but this failed after reading 1MiB of data. Next, I tried dd if=/dev/fmem of=memory bs=1M, but this stopped after reading 3010461696 bytes (exactly 2871 MiB). After looking at /proc/mtrr (shown below), I decided to try adding skip=4096. This ultimately slowed down, reading at a speed of only 3 MiB/sec, so I interrupted it (yielding a file of 5.8 GiB). (at least the last 100 MiB of the file contains FFs)

reg01: base=0x000000000 (    0MB), size= 2048MB, count=1: write-back
reg02: base=0x080000000 ( 2048MB), size= 1024MB, count=1: write-back
reg03: base=0x100000000 ( 4096MB), size= 4096MB, count=1: write-back
reg04: base=0x200000000 ( 8192MB), size= 1024MB, count=1: write-back
reg05: base=0x23c000000 ( 9152MB), size=   64MB, count=1: uncachable
reg06: base=0x0b4000000 ( 2880MB), size=   64MB, count=1: uncachable
reg07: base=0x0b8000000 ( 2944MB), size=  128MB, count=1: uncachable

I could not find the data I had open for some hours in the text editor, so I believe I have skipped some memory while doing a dump. So, given my goal (recovery of data from userspace programs), what is the most efficient method to dump the system memory to a file? What are some points that must be considered while doing such a dump?

Solution 1

Check this project: foriana

Foriana is (FOrensic Ram Image ANAlyzer)

input: dump of (physical) RAM output: various information

Version 1.0 can list processes and modules from memory dump of i386/x86_64/arm linux/bsd kernels, and provide option for reading linear memory from dumps.

There is a kernel module fmem:

Fmem is kernel driver, that creates /dev/fmem device. /dev/fmem behave in same way that /dev/mem (direct access to physical memory), but does not have limits that /dev/mem have. It is possible to dump whole physical memory through /dev/fmem.

I have use it, compile pretty easy.

Solution 2

You might want to use ddrescue or similar program, that can skip inaccessible data. dd conv=noerror might be helpful too. Also check this question on superuser.

More importantly though, if you got yourself into an OOM situation the sluggishness was very likely caused by kernel swapping out pages from anything else than the requesting application. Hence if you want your data, check the swap instead of /dev/mem - chances are it will be there. Similarly, if the OOM killer doesn't kick in and you kill processes by hand, once e.g. your editor gets killed first the memory hungry process might still happen to get some time to grab those pages.

As mentioned by Gilles in comment, the data can easily be in some special structure, so you won't be able to find them that easily even if you manage to reconstruct killed process's address space mappings and have enough luck to find all of the needed pages still intact.

Comments

[Lekensteyn]

@ott-- CONFIG_DEVKMEM is disabled, looking in the source code it seem to allow unrestricted access, but I am still not convinced that this is the best way to do it (IO mem access?)

[Gilles 'SO- stop being evil']

Maybe you did get the memory of the editor but you didn't recognize it. Reconstructing data structures from a memory dump can be hard. The first thing you'll need to do is reconstruct the memory mapping from the kernel data structures (there are probably existing forensic tools for that) so as to get the virtual memory of the process you're interested in (which is likely to be spread around many disjoint 4kB pages in physical memory). Then the text may not be in one consecutive blob and may use UCS4 or other representations and may store lines or other blocks in separate chunks.

[peterph]

@Gilles +1, once the process is killed I would expect kernel to free the task descriptors -> forget all about its address space mapping. As for the data representation, it can easily be a tree (with enough luck allocated by JVM :)).

[Lekensteyn]

@Gilles I could recognize other files that were opened in the text editor (Kate), so it must be possible to get the data (partially) back. As for the virtual memory, that would mean that data could be shuffled, but does that also imply that data is mangled before writing to RAM? As peterph noted, address mappings get lost when a task is killed. But the memory is still there right, nothing clears it?

[Lekensteyn]

@psusi I wrote a C program that can search the dump for a literal string in a few seconds, it is not like some private key got lost or something. But if there is a way to get the data back, why not try it? If it is possible to spend ten minutes on retrieving 80% of the data, that would already help a lot.

[Lekensteyn]

I have seen that SU answer before, that is how I found fmem. ddrescue is not going to help me since 256 pages (1 MiB) is a hard-coded limit. I would expect to get into an OOM condition, but the OOM killer did not kick in (pastebin.com/DvYTCcRK). One week ago, I had the same issue (still Linux 3.7.5, I have not rebooted, only suspended to ram). There is no swap file/partition since I have a SSD. (swappiness = 60 (default)).

[Tobu]

The asker did try /dev/fmem.