Defragging RAM / OOM failure

Question

This question is fairly lengthy, so I'll ask the questions at the top and then go through my method of coming to the questions:

1. Did (Busybox based) rm not execute because there wasn't enough contiguous RAM?
2. If so, is there a lightweight method of defragging the DMA - without resorting to a system restart?
3. If not, what caused it? How can I prevent it from happening in the future?

After our test system had been running fairly intensively over the last few days - I telnet'd into the system and checked the test results. When I came to delete some data, the system returned the command line (as if the command had executed correctly). When I came to check the directory for another set of results, I saw the file still existed (using ls).

After this, I noticed more and more of my shell commands wern't performing as expected.

I'll start off with an output from dmesg after rm failed to execute correctly:

Allocation of length 61440 from process 6821 (rm) failed

DMA per-cpu:

CPU 0: hi: 0, btch: 1 usd: 0

Active_anon:0 active_file:1 inactive_anon:0 inactive_file:0 unevictable:6 dirty:0 writeback:0 unstable:0 free:821 slab:353 mapped:0 pagetables:0 bounce:0

DMA free:3284kB min:360kB low:448kB high:540kB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:0kB unevictable:24kB present:8128kB pages_scanned:0 all_unreclaimable? no

lowmem_reserve[]: 0 0 0

DMA: 31*4kB 47*8kB 42*16kB 64*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3284kB

14 total pagecache pages

Unable to allocate RAM for process data, errno 12

Initially, I thought I was unable to run the program in the largest portion of contiguous memory. Meaning the DMA was too fragmented and I would have to find a way of getting the system to defragment the memory.

Then I did a quick math/sanity check and realised that the program should have been able to run in the sole 64kB contiguous memory slot. Rm was requesting 61440 bytes (60kB).

I did a good old "manual defrag" and rebooted the system. When I rebooted the sytem I output /proc/buddyinfo:

Node 0, zone DMA 2 8 3 12 0 1 0 1 0 1 0

Which I suspect map to:

• 2 x 4 kB
• 8 x 8 kB
• 3 x 16 kB
• 12 x 32 kB
• 1 x 128 kB
• 1 x 512 kB

But if one sums the above list of values, it doesn't match up with the output of /proc/meminfo:

MemTotal:           6580 kB
MemFree:            3164 kB
Buffers:               0 kB
Cached:              728 kB
SwapCached:            0 kB
Active:              176 kB
Inactive:            524 kB
Active(anon):          0 kB
Inactive(anon):        0 kB
Active(file):        176 kB
Inactive(file):      524 kB`
Unevictable:           0 kB
Mlocked:               0 kB
MmapCopy:            844 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:             0 kB
Mapped:                0 kB
Slab:               1268 kB
SReclaimable:        196 kB
SUnreclaim:         1072 kB
PageTables:            0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:        3288 kB
Committed_AS:          0 kB
VmallocTotal:          0 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB

To recap, my questions are:

1. Did rm not execute because there wasn't enough contiguous RAM?
2. If so, is there a lightweight method of defragging the DMA - without resorting to a system restart?
3. If not, what caused it? How can I prevent it from happening in the future?

I am using Lantronix's XPort Pro (8MB, Linux OS) running uClinux version 2.6.30. The shell in use is hush.

Solution 1

It's taken a bit of time, but I thought I'd hold off answering until I had answers to all 3 of my sub-questions.

Before I start though, I will mention that the correct term when it comes to "de-fragmenting" working memory is referred to "compacting" working memory.

1. Did rm not execute because there wasn't enough contiguous RAM?

I was correct in my conclusion - rm didn't execute because there was insufficient contiguous RAM. The system had been acquiring RAM and fragmenting it, thus making it unreclaimable.

2. If so, is there a lightweight method of defragging the DMA - without resorting to a system restart?

Turns out there is no way of compacting the memory, short of restarting the embedded system. In the case of a MMU-less system, prevention is the name of the game.

Part of me ponders if its possible to hack the linux kernel to emulate the MMU in software. I figure if it were possible, someone would have done it already. I can't imagine its an entirely new concept ;)

3. How can I prevent it from happening in the future?

For this project, I was using cron to manually initiate the program each time it was required. A much better way to do this is to call the program on start up, and then force the program to sleep until it is required. This way, the memory doesn't need to be allocated on each use. Thus reducing fragmentation.

On the first iteration of the project, we relied on my shell script calls to perform critical functions (such as rm). We didn't see the need in re-inventing the wheel if we didn't need to.

However, I would recommend avoiding the shell where possible for an MMU-less system -

(Question, what happens if you execute ls -la /path/to/directory/ | grep file-i-seek?)

(Answer: it starts a new subprocess)

If you need to implement some of the core shell script functionality in your C program, I recommend checking out the source code used in BusyBox. Chances are you'll be using C in your embedded system..

Solution 2

On your question 2 (defragmenting memory), quoting from https://www.kernel.org/doc/Documentation/sysctl/vm.txt :

compact_memory

Available only when CONFIG_COMPACTION is set. When 1 is written to the file, all zones are compacted such that free memory is available in contiguous blocks where possible. This can be important for example in the allocation of huge pages although processes will also directly compact memory as required.

this implies that the following command (executed with root privileges and if the kernel option mentioned above was enabled)

echo 1 > /proc/sys/vm/compact_memory

should tell the kernel to attempt to defragment the memory as much as possible. Beware that e.g. on some RHEL6 versions, this can crash the kernel...

Comments

[Alex Selby]

Minor point: you left out 1 x 2048 kB from your list of memory chunks. If you include that, then the sum is 3192 kB which is very close to the 3164 kB listed in /proc/meminfo.

[OldTinfoil]

facepalm - I guess that ties up the last little bit of mystery.

[tcoolspy]

Thanks for taking the time to come back and share your findings.

[derobert]

[I realize this is old] Emulating an MMU is hard... Without an MMU, every program directly uses the physical addresses as they appear on the memory bus. You could emulate one, but you'd have to intercept every memory access (just like an actual MMU does). Performance would be terrible. Alternatively, you could use indirect pointers (like Mac OS Classic did, calling them "handles"), but then you have a completely difficult API, and a very difficult one in the face of pre-emption (Mac OS Classic used cooperative multitasking).

[OldTinfoil]

Thanks for coming back and spending the time to write that response. Didn't know MacOS classic did that.

[OldTinfoil]

Thanks for spending the time to come back and comment on an old question!

[rogerdpack]

Did you figure out why it couldn't use the free 64KB block?