End kernel panic not syncing out of memory and no killable processes

Question

I have in VMWARE a virtual machince, Lubuntu17-32bit, it can run on 500mb RAM easily (even 300mb is enough). I compiled a kernel "linux-4.12.3.tar" and install it inside the VM i.e. Lubuntu. When I restarted with the new kernel I am getting:

End kernel panic not syncing out of memory and no killable processes

I had to increase the RAM for the VM till 1500mb to avoid that error and when the system lauches and reaches the desktop, I checked the memory usage it is same as previous around 300mb.

Note: the hard disk assigned to that virtual machine is sata 40GB.

So why is all of this RAM needed!! and most important how can I reduce that?

Update:

Please notice guys: I am neither saying I faced problems in compiling, problems in installing, nor problems in working on the new kernel. The only problem is on booting from the new kernel it needs 1500mb only for booting and when booting is completed and reaches the login screen and desktop the OS (Lubuntu) all together needs less than 300mb. So why that 1500mb and how to reduce it?

Answer:

The /boot/initrd.img file was the problem, it has a huge size. To reduce its size:

cd /lib/modules/<version>
sudo find . -name *.ko -exec strip --strip-unneeded {} +
sudo update-initramfs -c -k <version>

Reboot

Thanks for the guys who pointed this out.

Solution

I guess kernel panic occured due to one of the following reasons or may be more

1 . Compiling the kernel is basically installing the kernel so instead of replacing the old kernel it instead got stuck in deadlock where neither of the kernels were able to use the memory thus the panic.

2. The new kernel entered into some kind of loop, think it this way "it tried to call itself" or simply it became a type of fork bomb thus the out of memory error and panic.

3. The partition allocated for the new kernel was not sufficent. Try to make a different partition,mount it,sys link it so that others programs can execute it. Now re compile it using gcc and chains(linker)

4. Instead of replacing the kernel it killed itself.

5. If you extract a kernel tarball which is initially 300 mb of size, after extracting i.e "tar -xvf kernel.zip" it becomes a giant of 900 mb of size. After installation of packages like gcc,g++,gnu-make,etc and compiling it, the compiled kernel may become a mamoth of 2 GB of space. So If it the partition doesnt have that much space out of memory may occur.

6. You didnt provide a swap area. As building a kernel is resource intensive it needs both a good amount of space,a swap are and ram.

7. During installation it creates sub folder like /mount, /proc, /boot ,/dev etc . So that each part gets respective location which some how failed due to out of memory.

8. Usually a kernel is compiled in chroot like enviroment, but if the chroot is not properly set it may fail to get a resource like proc,mnt,dev thus the panic.

   If you still need some help then checkout these two places #1 kernel team for ubuntu , #2 kernel debugging team for ubuntu

New Update:

As your update seems to implying compiling the kernel worked.

It could only mean that you compiled the whole kernel.

Usually a kernel has everthing in it which may be important to one but a waste of space and time for other.

So either you could use the new kernels from ubuntu kernel . In my case its v4.14-rc1.Download the kernel which match your arch. I have x86_64 so I would choose 64 bits one. I would download these

• linux-headers-4.14.0-041400rc1_4.14.0-041400rc1.201709162031_all.deb
• linux-headers-4.14.0-041400rc1-generic_4.14.0-041400rc1.201709162031_amd64.deb
  • linux-image-4.14.0-041400rc1-generic_4.14.0-041400rc1.201709162031_amd64.deb

and would then issue "sudo dpkg -i *.deb" to compile all.

Now it seems that the kernel you compiled are of more than 1GB thus its using more resources.

The ubuntu kernel has many parts stripped out so its low i size and uses heavy compression.

Your kernel might have everything in it and can be used by IOT,servers,desktop, and may be compatible with arm 64,amd64, etc as it may have all the modules present to run these.

So to reduce it you need to remove all the parts useless in your case like drivers for hardware you never have heard of. Stripping server parts where ever possible.

Great work by the way as kernel compiling is a headache as its uses intensive resource ,time and patience

Update #2:

As you mentioned that your initrd.img file was huge. Could you atleast post a screenshot of your /boot/ or in terminal use "ls -l " to list the size of everything.

Kernel happens to be the heart of any Linux distro. Its very complex so without knowing anything one can just guess whats the issue.

I would still suggest you to strips the unwanted modules from your kernel and then compile it.

Comments

[Panther]

support for a custom kernel is limited at best

[Boris Hamanov]

You've enabled too many options in your custom kernel.

[Doug Smythies]

Yes, and/or you left too much debug stuff enabled.

[Mosab Shaheen]

Thanks guys for the reply. Please see the update. @bodhi.zazen bro the kernel as I said is working properly only booting is the problem @heynnema I enabled what was enabled before i.e. cp /boot/config-uname -r* .config which means copy the current configuration of the kernel to the new configuration @DougSmythies I didn't get you but as I said I used the current config for the new one so I think there should be no difference in behaviour

[Panther]

How did you build the kernel? Did you strip it ?

[Panther]

You can look at unix.stackexchange.com/questions/257812/…

[Mosab Shaheen]

I used these steps: kernelnewbies.org/KernelBuild $$$$$$$$ cp /boot/config-uname -r* .config $$$$$$$$$$ make -j2 $$$$$$$$ sudo make modules_install install $$$$$$$$$ sudo update-grub2

[Mosab Shaheen]

@bodhi.zazen the issue my friend is that the old kernel doesn't cause that issues. so if the kernel is loaded entirely into RAM, why didn't I face this issue in the old kernel.

[Panther]

I can not tell from what you posted. Either you new kernel is too large or your initramfs is too big those are the 2 things loaded.

[Panther]

Did you strip the kernel ? unix.stackexchange.com/questions/270390/…

[Mosab Shaheen]

@bodhi.zazen It may be the cause, so how can I check the size of the new kernel? and where is the initramfs file located?

[Panther]

they are in /boot

[Mosab Shaheen]

Let us continue this discussion in chat.

[Mosab Shaheen]

Thanks for the reply. Please see the update. The issue is on booting only other things like installation, compilation, ... are all fine. So how can I reduce the RAM for booting?

[Mosab Shaheen]

Thanks, but I got it working by reducing the initrd.img size. please see the answer above. By the way why initrd.img becomes huge after installing the new kernel?