Android: How to get kernel logs after kernel panic?

Question

I am using an Android Custom ROM on my device, also with a custom boot.img (custom kernel + cmdline + ramdisk). I now want to be able to view the kernel logs immediately after a kernel panic, but unfortunately I can not use a serial console.

The good news: There seem to be some sources/modules in the Linux kernel of Android that are written exactly for this purpose. For example, the following lines are activated in my .config file for the kernel:

CONFIG_ANDROID_RAM_CONSOLE=y
CONFIG_ANDROID_RAM_CONSOLE_ENABLE_VERBOSE=y
CONFIG_APANIC=y
CONFIG_APANIC_PLABEL="oem_log"

My problem is: After I forced a kernel panic in order to test this, i.e. by loading a simple panic kernel module with insmod panic.ko, it seems that no log was written to the MTD named oem_log (which exists on my device). Secondly, the RAM also does not contain logs after reboot because it seems to be cleared - or the logs are not written either.

So how can I get the kernel logs after a panic? Also it would be helpful if there is a way I could test the APANIC on the running system. Maybe by using the kernel debug system? As of now I am pretty new to this.

Thanks in advance for any help!

Solution 1

As for me,

cat /proc/last_kmsg 

after reboot (caused by the kernel panic during insmod) does list messages relevant to the crash, like

[  424.909515] Kernel panic - not syncing: Fatal exception
[  424.909606] Backtrace: 
[  424.909790] [<c005a5ec>] (dump_backtrace+0x0/0x10c) from [<c05f38dc>] (dump_stack+0x18/0x1c)
[  424.909973]  r6:c5cccf00 r5:00000000 r4:c08505a0 r3:00000000

So you can at least try. I am working with Linux 3.0.31-g4f6d371 on Galaxy Nexus.

Solution 2

It seems at Android-7.0 or above, the last_kmesg log is moved to: /sys/fs/pstore/console-ramoops, so try:

cat /sys/fs/pstore/console-ramoops

it works well for me on nexus-5x

Solution 3

How about /data/dontpanic folder? After kernel panic happens, you can connect USB cable with your Android device and check the files in that folder through ADB.

I found this folder contains some apanic files after a kernel panic happens. For example, if a kernel panic just happened and you go check the folder, you might find these two files:

apanic_console

apanic_threads

You can find out in apanic_threads which thread / process is running when the kernel panic happens. In apanic_console you might find out more information such as the stack trace and values of some critical registers: PC, LR, etc.
They will help you get your debugging started.

Solution 4

Android creates a RAM console and attempts to save the last kernel message buffer for you in RAM (assuming the power doesn't go out). You can access this file through the proc interface and on my system it is world readable:

cat /proc/last_kmsg

For more information see the kernel code @ drivers/staging/android/ram_console.c

Solution 5

I faced a similar issue of collecting shutdown logs in Android. I had posted this question long back and it has 2 approaches. I use the 2nd one as the 1st one doesn't work for me. Here is the question

Where does Android store shutdown logs?

Hope this helps.

View more solutions

Comments

[mreichelt]

Unfortunately, all the files described there are only available if the APANIC module works correctly - which is what I am trying to achieve...

[Pavan Manjunath]

@mreichelt No luck with connecting to adb and trying to pull kmsg?

[mreichelt]

kmsg is the current log. I want to get the kernel logs after a kernel panic occured, i.e. when the system crashed and the device is rebooted. That is what APANIC is for.

[Pavan Manjunath]

If you want to figure out what caused a kernel panic, then I think the kernel logs leading to the panic should be of interest to you. Irrespective of whether before or after, AFAIR kernel logs are always in kmsg

[mreichelt]

I am marking this as the correct answer - but it only seems to work if this functionality was implemented correctly for the particular device. Which, for me (at the time I asked the question and for my particular device) was not the case.

[Suici Doga]

Does this file exist is the device hasn't panicked

[ogurets]

Doesn't exist, kernel version 3.0.8+.

[ogurets]

I have this folder, but no files inside (after the panic). Executed 'find / -name "panic"' and found something interesting: "/sys/module/kernel/parameters/panic" and "/proc/sys/kernel/panic", both containing "2" on my device.

[nyanpasu64]

For me, it was cat /sys/fs/pstore/console-ramoops-0.

[zhiqiu]

Yes, it may add id as suffix.

[imbr]

Excellent tip. Thx!

[user1874594]

https://android.stackexchange.com/questions/239500/analyzing‌​-boot-loop-root-caus‌​e-from-console-ramoo‌​ps-0-logcat this question carrying a bounty might be of intrest if you are that 'colonel' of kernel