Monitoring dmesg output

Question

I find that when something goes wrong at a low level on one of my linux servers, I can see messages about it in the kernel ring buffer. These can be viewed from the command line using the dmesg command.

I am wondering if there is an easy way to get my servers to email me whenever something is added to the kernel ring buffer?

Right now I have a script that runs every hour, makes a copy of the output of dmesg and runs a diff against the file from the previous hour. Unfortunately this doesn't work so well, because as lines are appended to the end of the dmesg output, other lines are truncated from the beginning. Also, if I have alot of the same message, it just stops notifying me altogether.

• Is there a better way of doing this?

• Does anyone else think it is important to see these messages when they happen?

added

• is the information reported by the dmesg command the same as that in one of the logfiles? (If so, then the solution is easier than I thought)

Solution 1

The following script will send any new entries to /var/log/kern.log to the root user.
Putting it in /etc/cron.hourly will send an email every hour, but only if there are new kernel messages.

#!/bin/bash

MAILTO=root
LOG=/var/log/kern.log

OFFSET_FILE=$0.offset
if [ ! -f $OFFSET_FILE ]; then echo 0 > $OFFSET_FILE; fi
OFFSET=`cat $OFFSET_FILE`
FILESIZE=`cat $LOG|wc -c`

# Check if log has been rotated
if [ "$OFFSET" -gt "$FILESIZE" ]; then
  OFFSET=0
  echo 0 > $OFFSET_FILE
fi
if [ "$FILESIZE" -gt "$OFFSET" ]; then
  tail -c+$OFFSET $LOG|sed "s/^/  /"|mail $MAILTO -s "new kernel alerts"
  echo $FILESIZE > $OFFSET_FILE
fi

Solution 2

This is a quick and dirty solution. You may get lots of mail. I'd advise adding some grep and/or grep -v commands. Of course you can use this technique for other log files, too. Add this command in your /etc/rc.d/rc.local or your system's equivalent (after having tested it from the command line).

sudo tail -F /var/log/messages | while read line ; do echo "$line"|mail -s Subject recipient; done &

edit: changed to capital F to make tail follow the file by name to be able to handle log rotation.

Solution 3

There are a number of tools designed to gather this information and report it regularly.

I find the Lire tool (from the LogReport system) to be a good reporting tool, but you may also be interested in Logcheck and Logwatch. All are free software and can be installed directly from most major GNU/Linux package repositories.

Comments

[Dennis Williamson]

I think whether and which log files match dmesg is system dependent. You can configure kern.* messages to go to a file or files of your choice.

[David Pashley]

I think the asker wanted something that happened a little more often, but I don't really think there is anything. I use logcheck and have voted +1

[Kjetil Joergensen]

Instead of tailing /var/log/messages (which may get log-entries from other syslog facilities) you could configure syslogd to log kern.* to another file and tail that. In addition, be mindful of logrotate which probably will break this when it does it's thing.

[Dennis Williamson]

I edited my answer to handle the case of log rotation. Your suggestion regarding kern.* messages is a good one. I'd keep them going to /var/log/messages, too, though (just to be clear).

[phirschybar]

On my system (Debian), the output of "dmesg" is different from /var/log/messages.

[Dennis Williamson]

@Brent - This Fedora 11 system doesn't have kern.log, but my Ubuntu desktop does.

[Dennis Williamson]

$OFFSET and $FILESIZE are in terms of characters (wc -c), but then the tail command is done in terms of lines (-n). They need to match. I would make it wc -l, but tail -c makes it where the outputs don't overlap by one line so that might be preferred.

[phirschybar]

Good catch Dennis, I will add -c to the script above. Thank you.