Messages from syslogd, what do they mean and what do I do?

Question

Recently, these messages started popping up straight into my prompt when I'm connected to the OpenSUSE system in question via PuTTY:

Message from syslogd@host at Aug  5 11:04:03 ...
 kernel:[ 6177.851012] EIP: [<75c0234e>] 0x75c0234e SS:ESP 0068:f324dde1

Message from syslogd@host at Aug  5 11:15:01 ...
 kernel:[ 6836.654020] Process sh (pid: 6245, ti=f2bee000 task=f32fd2b0 task.ti=f2bee000)

Message from syslogd@host at Aug  5 11:15:01 ...
 kernel:[ 6836.654020] Stack:

Message from syslogd@host at Aug  5 11:15:01 ...
 kernel:[ 6836.654020] Call Trace:

Message from syslogd@host at Aug  5 11:15:01 ...
 kernel:[ 6836.654020] Inexact backtrace:

Message from syslogd@host at Aug  5 11:15:01 ...
 kernel:[ 6836.654020]

Message from syslogd@host at Aug  5 11:15:01 ...
 kernel:[ 6836.654020] Code:  Bad EIP value.

Message from syslogd@host at Aug  5 11:15:01 ...
 kernel:[ 6836.654020] EIP: [<75c0234e>] 0x75c0234e SS:ESP 0068:f2befead

I know some very basic stuff about Linux but this catches me off guard. What does this mean? How do I troubleshoot?

edit Turns out the system is actually unreachable now, while it does reply to ping I cannot connect via SSH to it. Is there anything I can do physically on the machine?

Solution 1

These messages look like the kernel was having a kernel panic. It was basically crashing. There's little you can do if the system is unstable/crashing.

Typically these types of messages will show up on the console. There you can use the command dmesg -n 1 to disable them from being printed, since they can get a little annoying when they're benign.

excerpt from dmesg man page

-n level
       Set  the  level  at  which  logging  of messages is done to the 
       console.  For example, -n 1 prevents all messages, except panic
       messages, from appearing on the console.  All levels of messages
       are still written to /proc/kmsg,  so syslogd(8)  can  still be used 
       to control exactly where kernel messages appear.  When the -n option 
       is used, dmesg will not print or clear the kernel ring buffer.

Bad EIP Value

This error often always means you're having some type of failure of your hardware, often your RAM. I would enlist something like memtest86+ to verify that the RAM is functioning correctly.

Solution 2

I'm seeing it on a Raspberry Pi Zero only when running a particular fork of cgminer (vt_hoang). It usually runs about 2 days and crashes the system. The same cgminer also crashes a different Pi. I hadn't noticed messages to stderr before, I'd just set logging to be persistent across reboots and was looking at journalctl output. I need to try again with valgrind and gdb.

But the error message is just that, making them shut up isn't going to fix the underlying problem. A different kernel might act differently, but it can also be some software you're running. It isn't necessarily a hardware problem.

Comments

[pzkpfw]

Thanks for pointing out how I can ignore these messages, but since they are making the system crash and become unreachable via SSH I would need to try a solution. Should I try booting an older kernel in GRUB and see if the problem goes away? Should I try updating OpenSUSE via zypper?

[pzkpfw]

If it helps, I saw on the physical screen attached to the server that I am getting an error saying "Bad EIP value". I can't get a new prompt by pressing CTRL+ALT+F1 or whatever the combination is, I just get the same error. Rebooting the system fixes the issue temporarily, I'm running an older kernel now. Any ideas on how to proceed with troubleshooting?

[slm]

@bigbadonk420 - see updates, sounds like RAM is failing.