"system is going down for reboot now"

Question

I am connected to a beagleboard via tty and I suddenly see this message :

Broadcast message from root@arm
        (unknown) at 11:40 ...

The system is going down for reboot NOW!
[  143.036193] Restarting system.

The system is Ubuntu 12.10. Is there a place where I can find why the system decided to go down ?

UPDATE: More info

• This actually happens all the time, the board reboots every 1 to 5 minutes.
• I made sure no program or daemon runs at startup.
• I can't find Restarting anywhere in any file in /var/log
• In syslog before a reboot there is consistently a Feb 10 09:20:00 arm rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="317" x-info="http://www.rsyslog.com"] exiting on signal 15.

Solution 1

The way I'd track this down is to replace the utilities halt and shutdown with a script.

First create a script such as the following at /bin/fakehalt:

#!/bin/bash
exec >>/tmp/fakehalt.log 2>&1
date
echo "CMD=$0 PID=$$"
ps -ef --forest
echo '========'

Then install it with:

chmod a+x /bin/fakehalt
mv /sbin/halt /sbin/halt.orig
ln -s /bin/fakehalt /sbin/halt
mv /sbin/shutdown /sbin/shutdown.orig
ln -s /bin/fakehalt /sbin/shutdown

This will create a log file at /tmp/fakehalt.log each time it is called. It'll log the name it was called as (halt or shutdown) it's own PID, and then a tree diagram of all processes at the time.

This should give you all the necessary info to track it down. Just look through the ps tree and find what called the script.

---

/sbin/reboot should be a symlink to /sbin/halt. If it isn't, then replace it as well.

---

If this still doesn't capture it, replace /sbin/init as well (as it can also be used to reboot the system). But this is dangerous as if the system reboots, it won't come up properly.

Solution 2

Run you kernel with something like init=/bin/bash. This should make that no processes are running than the this shell.

Verify that there is on rebooting issue anymore.

Next find out what init system you use normally and have a look at all services started by it.

Try to disable services until the rebooting issue disappears.

Solution 3

Well it's one of these possibilities.

• Someone is running shutdown
• cron is running shutdown (or has started something which is running shutdown)
• power management triggers a shutdown (no power, hibernate, sleep or idle system, the power hardware button is pressed)
• reboot requested after package install

Comments

[Thomas]

@JMCF125 I have updated my question with some more info

[Ievgen Chuchukalo]

It turns out that rsyslogd is simply the daemon that gives those messages... Perhaps you got a rare Linux malware... 1 to 5 minutes is little time, try to insert a Live CD and find out more. If you get nothing, I'd advise you to reinstall, possibly another distro.

[Thomas]

I was at the beginning thinking to a physical problem on the board like electrical voltage going a little too low, but given would the system know about it ?

[Ievgen Chuchukalo]

If the power was the problem, with the Live CD the same would happen.

[Thomas]

Unfortunately cron and fcron are empty, and I am the only one connected to the board, no one is running shutdown

[Ievgen Chuchukalo]

The OP says: «I made sure no program or daemon runs at startup.». And I doubt someone has hacked into his computer to run shut it down once in a while. Maybe it's something else.

[X Tian]

Search for shut (case independent not restarting) in var/log/*

[Thomas]

no result for "shut"

[Thomas]

in what file do you write init=/bin/bash ?

[Thomas]

I like your solution, I have tried it and it helped me find the problem (a job in fcron left by someone else that did shutdown on purpose when a check was false)

[michas]

This depends on your boot loader. (grub, syslinux, etc.) Usually your boot loader should give you some interactive way to choose your kernel and add additional parameters. It also has some configuration file (usually in /boot) to permanently add parameters.

[X Tian]

@Thomas You mentioned earlier that fcron was empty, just wondered why the task didn't show, can you see it now ?

[Thomas]

@XTian human error, I checked for cron but not fcron

[Ievgen Chuchukalo]

@Thomas, BTW, why was there a job in fcron to shut down?

[Thomas]

@JMCF125 if something that should be plugged in does not show up in /dev/... we reboot for instance

[Ievgen Chuchukalo]

@Thomas, was that the case? What wasn't plugged in? Can you show the job?

[Thomas]

@JMCF125 Yes it was the case (beagleboard usb disconnect issues) and led me to read very long google groups threads. It was my mistake with the fcron, this was a false fire. Somehow the script in fcron says "reboot" and that's why there is a broadast message in kern.log. Thanks for following up

[Ievgen Chuchukalo]

@Thomas, you're welcome. This was an interesting problem. But I still don't see why you didn't mark this answer as accepted, if it is the one you used.