How to read the shut down messages afterwards?

Question

I'm using Lubuntu 11.10. Every time I shut down I can read FAIL in red letters, but I can not read more.

So, how to read the log messages and try to solve the problem?

Solution 1

There seems to be no way to log this data to a file. For the boot process, there is the bootlogd package which creates the file /var/log/boot, but nothing for the shutdown/reboot process. As far as I can see there is no way to log with rsyslog either, and even if there was, there are messages printed after rsyslog is stopped. Part of my shutdown/reboot process is to remount the rootfs readonly and umount everything else, after this logging to a file that will still be there at the next boot is virtually impossible.

The easiest way I can see to view the messages is to edit the /etc/init.d/halt and/or /etc/init.d/reboot scripts to pause just before the actual halt/reboot. For the halt script, run the command sudoedit /etc/init.d/halt (or use a GUI editor) and look for the line that does the actual halt. For me this is the line:

halt -d -f $netdown $poweroff $hddown

Otherwise it should be at the end of the do_stop function and the only line that calls the halt command. Once you find the line, just insert a new line above with the following:

read -p "Press enter to halt" reply

Save the file and exit. Now when you shutdown, the system will pause until you press enter (or CTRL-C, CTRL-D, etc). You can the read the messages printed on the screen. If there is more than a single screenful of text, you can see terminal scrollback by pressing Shift+PgUp. If this is still not enough, there are ways to increase the size of the scrollback buffer (perhaps a different question though).

To do the same when the system reboots, you have to edit the /etc/init.d/reboot file. The command used here is of course reboot as opposed to halt and should again be at the end of the do_stop function. For me the line is:

reboot -d -f -i

Again just insert the following on a new line above:

read -p "Press enter to reboot" reply

Note also that these files are listed as conffiles for the initscripts package. These edits won't be clobbered by default when the packages is upgraded, although they will cause a conflict.

---

A more complete solution would be to use the following script:

#! /bin/sh
### BEGIN INIT INFO
# Provides:          pause_hook
# Required-Start:
# Required-Stop:     halt reboot
# Default-Start:
# Default-Stop:      0 6
# X-Stop-After:      umountroot
# X-Interactive:     true
# Short-Description: Pause before halt or reboot
# Description:
### END INIT INFO

do_stop () {
    [ -r /etc/pause_hook.conf ] && . /etc/pause_hook.conf

    [ "$PAUSE_HOOK_ENABLED" = true ] && read -p "Press enter to continue" reply
}

case "$1" in
    start)
        # No-op
        ;;
    restart|reload|force-reload)
        echo "Error: argument '$1' not supported" >&2
        exit 3
        ;;
    stop)
        do_stop
        ;;
    *)
        echo "Usage: $0 start|stop" >&2
        exit 3
        ;;
esac

This should be placed in /etc/init.d/pause_hook and can be enabled to run at shutdown/reboot with the following command:

sudo update-rc.d pause_hook defaults

To then enable the actual hook, create the files /etc/pause_hook.conf containing the line:

PAUSE_HOOK_ENABLED=true

The shutdown/reboot process should now pause just before the halt or reboot script is called, giving time to view the messages. It can also be easily disabled/re-enabled by commenting/uncommmenting the enable line in /etc/pause_hook.conf. There will also be no dpkg conffile conflicts during upgrades this way.

Solution 2

Easiest solution might be to try taking a video instead of a photograph. You can step through it frame-by-frame later.

Solution 3

Seeing the log in real time

I've found that during shutdown there is usually an Ubuntu logo and flashing lights, which is shown instead of a log of the shutdown process. If there are errors, then they're kind of shown, but messily. However, whilst shutting down, if I press the Windows key and r (Metar), then I get to see the success and failure of the system services, as they occur. So then I know what exactly is broken. No idea whether this keyboard shortcut is specific to my Kubuntu setup or what; I didn't add it.. One of those I found by accident, somehow...

Viewing the logs after rebooting

When the system is rebooted, the error messages should be saved into a logfile. Which logfile depends on which service is broken/misconfigured. The relevant log will almost definitely be in /var/log/ (or a subdirectory thereof). ls, less, grep and find have been the only programs I've needed for finding error messages in the logs...

Once you've found the error and the service that caused it, then you shouldn't need to reboot to test the new configuration; just restart the service.. Hopefully you can test the fixed configuration with a command like:

sudo service <service name> restart

Solution 4

After looking into how bootlogd actually manages to do the logging, it turns out that it can be persuaded to log the shutdown/reboot process as well as the startup one. bootlogd is started early in the boot process. It then does some magic with tty and pts devices to log all output to the terminal it is connected to. It is then stopped later in the boot process, presumably before it would start logging the output from a user working on the tty.

If bootlogd is fired up again during the shutdown/reboot process and stopped before whichever file system containing the log is unmounted, there will be a record of most of the shutdown process available to read at the next boot.

bootlogd manages to be both started and stopped during the startup by having two init scripts. One is a normal init script which is starts/stops the process as normal. The other is a 'reversed' init script in that when called with start, it calls the first script with stop. This 'tricks' the sysvinit process, it appears that it is starting two separate services when it is really starting and stopping the same service. This is needed to make sure everything happens in the right order.

To make bootlogd run during shutdown, what is required is:

1. Install bootlogd if you haven't already.
2. Copy the two init scripts
3. Make the 'reversed' script change stop to start instead of start to stop.
4. Alter the LSB headers so that both scripts run at the right times during shutdown/reboot
5. Install them with update-rc.d.

Here are my changes as a copy/paste bash script (if you want to make the changes manually, the LSB headers I have used are in the patch):

cd /etc/init.d
cp bootlogd shutdown-bootlogd
cp stop-bootlogd shutdown-start-bootlogd

echo -e 'diff -ur ./shutdown-bootlogd /etc/init.d/shutdown-bootlogd
--- ./shutdown-bootlogd\t2014-02-20 13:59:23.426109512 +0000
+++ /etc/init.d/shutdown-bootlogd\t2014-02-20 11:10:56.238656828 +0000
@@ -1,14 +1,13 @@
 #! /bin/sh
-### BEGIN INIT INFO
-# Provides:          bootlogd
-# Required-Start:    mountdevsubfs
-# X-Start-Before:    hostname keymap keyboard-setup procps pcmcia hwclock hwclockfirst hdparm hibernate-cleanup lvm2
-# Required-Stop:
-# Default-Start:     S
-# Default-Stop:
-# Short-Description: Start or stop bootlogd.
-# Description:       Starts or stops the bootlogd log program
-#                    which logs boot messages.
+### BEGIN INIT INFO 
+# Provides:          shutdown-bootlogd 
+# Required-Start:
+# Required-Stop:     umountroot halt reboot 
+# Default-Start:      
+# Default-Stop:      0 6 
+# X-Stop-After:      umountfs 
+# Short-Description: Stop bootlogd at shutdown. 
+# Description: 
 ### END INIT INFO

 PATH=/sbin:/bin  # No remote fs at start
diff -ur ./shutdown-start-bootlogd /etc/init.d/shutdown-start-bootlogd
--- ./shutdown-start-bootlogd\t2014-02-20 13:59:23.430107513 +0000
+++ /etc/init.d/shutdown-start-bootlogd\t2014-02-20 11:10:56.238656828 +0000
@@ -1,24 +1,24 @@
 #! /bin/sh
 ### BEGIN INIT INFO
-# Provides:          stop-bootlogd
-# Required-Start:    $local_fs $all
-# Required-Stop:
-# Default-Start:     2 3 4 5
-# Default-Stop:
-# Short-Description: Stop bootlogd
-# Description:       See the init.d/bootlogd script
+# Provides:          shutdown-start-bootlogd
+# Required-Start:
+# Required-Stop:     $local_fs $all
+# Default-Start:
+# Default-Stop:      0 6
+# Short-Description: Start or stop bootlogd at shutdown.
+# Description:
 ### END INIT INFO

-NAME=stop-bootlogd
+NAME=shutdown-start-bootlogd
 DAEMON=/sbin/bootlogd

 [ -x "$DAEMON" ] || exit 0

 case "$1" in
-  start)
-\t/etc/init.d/bootlogd stop
+  stop)
+\t/etc/init.d/bootlogd start
 \t;;
-  stop|restart|force-reload)
+  start|restart|force-reload)
 \t# No-op
 \t;;
   status)
' | patch

update-rc.d shutdown-bootlogd defaults
update-rc.d shutdown-start-bootlogd defaults

All messages that appear before bootlogd is stopped will be stored in /var/log/boot. bootlogd removes the escape characters from the text stream. The following (bash) command will display the log in colour, as it appears during shutdown:

sed $'s/\^\[/\E/g;s/\[1G\[/\[27G\[/' /var/log/boot | less -r

See this question for more details on this - https://stackoverflow.com/questions/10757823/display-file-with-escaped-color-codes-boot-messages-from-bootlog-daemon/19011140

The location of the log can be changed by further editing the scripts. Unfortunately every appearance of the file must be changed (also, replacing /ver/log/boot isn't enough as the script does a cd to /var/log at one point).

The above will also only work if /var/log is on the rootfs. If not the dependencies need to be reworked so that umountfs is done after bootlogd is stopped. Or else log to a file on the rootfs.

Solution 5

You need to edit all the /etc/rc6.d/* files and redirect their output to some file, which you can read later.

This redirection, will also tell you, which program failed and why.

Additionally, you have to modify the start-stop-daemon by removing --quiet parameter and adding -v parameter in all those files.

If you are doing so, then make sure, you will rollback all the changes all you made. Backing up the existing files before making changes will be good idea.

View more solutions

Comments

[Admin]

Take a look at this answer on askubuntu. It may help you with your problem.

[Admin]

everything recored at /var/log/syslog and /var/log/messages, So you need to read them and throubleshoot. You can use io redirection with error redirection when you use shutdown command

[Sigur]

I can not believe that there is no file with those messages recorded!!! I'll try to record by video.

[cheshirecatalyst]

Some of those messages are printed after filesystems are unmounted! So they can't all end up in a file.

[Sigur]

There is no /etc/syslog.conf. There are only sysctl.conf sysctl.d/ systemd/.

[VinoPravin]

A pretty nice solution, especially the init script.

[Wahid Masud]

try /etc/rsyslog.conf

[Sigur]

So I guess that this is a totally safe proccess. No damage to the hardware?!

[Sigur]

OK. This will take a lot of work from me. I'll try. Thanks.

[SHW]

Let me know the result then

[Graeme]

Editing sripts in /etc/init.d (everything in /etc/rc?.d links to here) won't do anything. The output is not direct, it is through the functions in /lib/lsb/init-functions and other files which it in turn sources. You could edit these, some upgrades would clobber the changes though.

[Graeme]

@SHW Nope, look at the scripts. Most of the output comes from other parts of the script. I don't think start-stop-daemon produces any output, at least not normally. The scripts generate ok/fail output based on its return output using log_end_msg. See /lib/lsb/init-functions.d/20-left-info-blocks.