How to know why server keeps restarting?

Question

It seems that my server keeps restarting. I want to know why.

How can I know when the last time server was rebooted and why?

root     pts/0        139.193.156.125  Thu Aug  8 21:10   still logged in
reboot   system boot  2.6.32-358.11.1. Thu Aug  8 20:38 - 21:11  (00:33)
reboot   system boot  2.6.32-358.11.1. Thu Aug  8 20:15 - 21:11  (00:56)
reboot   system boot  2.6.32-358.11.1. Thu Aug  8 19:16 - 21:11  (01:55)
reboot   system boot  2.6.32-358.11.1. Thu Aug  8 18:56 - 21:11  (02:14)
reboot   system boot  2.6.32-358.11.1. Thu Aug  8 18:24 - 21:11  (02:47)
root     pts/1        139.193.156.125  Thu Aug  8 18:16 - crash  (00:07)
root     pts/0        195.254.135.181  Thu Aug  8 18:10 - crash  (00:13)
reboot   system boot  2.6.32-358.11.1. Thu Aug  8 17:52 - 21:11  (03:19)
root     pts/0        195.254.135.181  Thu Aug  8 17:38 - crash  (00:13)
reboot   system boot  2.6.32-358.11.1. Thu Aug  8 17:08 - 21:11  (04:02)
reboot   system boot  2.6.32-358.11.1. Thu Aug  8 16:58 - 21:11  (04:12)
reboot   system boot  2.6.32-358.11.1. Thu Aug  8 16:45 - 21:11  (04:26)
reboot   system boot  2.6.32-358.11.1. Thu Aug  8 16:35 - 21:11  (04:36)
reboot   system boot  2.6.32-358.11.1. Thu Aug  8 16:27 - 21:11  (04:44)
reboot   system boot  2.6.32-358.11.1. Thu Aug  8 15:59 - 21:11  (05:12)
reboot   system boot  2.6.32-358.11.1. Thu Aug  8 06:15 - 21:11  (14:56)
root     pts/1        208.74.121.102   Wed Aug  7 06:03 - 06:04  (00:00)
root     pts/1        208.74.121.102   Tue Aug  6 15:34 - 17:40  (02:05)
root     pts/0        139.193.156.125  Tue Aug  6 11:28 - 04:40 (1+17:11)

In Linux is there ANY WAY to know why the system rebooted? Specifically did high load cause it? If not that then What?

Solution 1

/var/log/messages

That is the main log file you should check for messages related to this. Additionally either /var/log/syslog (Ubuntu) or /var/log/secure (CentOS)

To find out when your server was last rebooted just type uptime to see how long it has been up.

Solution 2

You can use the following 2 commands (who & last) to find out the last time the system was rebooted and also messages about previous shutdown or runlevel changes.

Last time system booted?

For this you can use the who command. Specifically with the -b switch.

$ who -b
         system boot  2013-08-01 17:56

This says the last time the system was booted was 2013-08-01.

Past reboots

If you're interested in seeing a more extensive list of previous reboots you can use the last command.

$ last reboot | less
reboot   system boot  2.6.35.14-106.fc Thu Aug  1 17:56 - 02:03 (7+08:06)   
reboot   system boot  2.6.35.14-106.fc Thu Aug  1 09:41 - 17:55  (08:14)    
reboot   system boot  2.6.35.14-106.fc Thu Jul 25 15:24 - 17:55 (7+02:31)   
reboot   system boot  2.6.35.14-106.fc Thu Jul 18 18:05 - 15:23 (6+21:17)   
...

Past system shutdowns & runlevel changes?

You can use the last command for this too. You'll need to use the -x switch.

$ last -x | less
saml     pts/7        :pts/6:S.0       Sat Aug  3 21:30 - 21:30  (00:00)    
saml     pts/6        :0.0             Sat Aug  3 21:29 - 21:30  (00:01)    
saml     pts/4        :0.0             Fri Aug  2 21:49 - 22:16 (2+00:26)   
saml     pts/2        :0.0             Fri Aug  2 13:30 - 22:16 (2+08:45)   
saml     pts/1        :0.0             Fri Aug  2 13:05   still logged in   
saml     pts/0        :0.0             Fri Aug  2 12:37   still logged in   
saml     pts/0        :0.0             Fri Aug  2 12:35 - 12:37  (00:02)    
saml     pts/0        :0.0             Thu Aug  1 17:58 - 12:35  (18:36)    
saml     tty1         :0               Thu Aug  1 17:56   still logged in   
runlevel (to lvl 5)   2.6.35.14-106.fc Thu Aug  1 17:56 - 02:04 (7+08:08)   
reboot   system boot  2.6.35.14-106.fc Thu Aug  1 17:56 - 02:04 (7+08:08)   
shutdown system down  2.6.35.14-106.fc Thu Aug  1 17:55 - 17:56  (00:00)    
runlevel (to lvl 6)   2.6.35.14-106.fc Thu Aug  1 17:55 - 17:55  (00:00)    
saml     tty2                          Thu Aug  1 17:54 - down   (00:01)    
root     tty2                          Thu Aug  1 17:53 - 17:54  (00:00)    
...

References

• who man page
• last man page

Solution 3

To know when the last time server was rebooted you can use tuptime:

$ tuptime -t
No.        Startup Date                                          Uptime       Shutdown Date   End                                       Downtime

1     10:33:14 01/05/18     45 days, 0 hours, 39 minutes and 55 seconds   11:13:09 02/19/18   BAD   27 days, 21 hours, 51 minutes and 36 seconds
2     09:04:45 03/19/18      37 days, 7 hours, 6 minutes and 58 seconds   17:11:43 04/25/18   BAD   67 days, 18 hours, 36 minutes and 15 seconds
3     11:47:58 07/02/18                        6 minutes and 57 seconds   11:54:55 07/02/18   OK                                      27 seconds
4     11:55:22 07/02/18                         1 minute and 28 seconds   11:56:50 07/02/18   BAD                       2 minutes and 43 seconds
5     11:59:33 07/02/18                         1 minute and 38 seconds   12:01:11 07/02/18   OK                                      26 seconds
6     12:01:37 07/02/18                        6 minutes and 52 seconds   12:08:29 07/02/18   OK                                      26 seconds
7     12:08:55 07/02/18                                      14 seconds   12:09:09 07/02/18   BAD                       8 minutes and 40 seconds
8     12:17:49 07/02/18                        5 minutes and 41 seconds   12:23:30 07/02/18   OK                                      27 seconds
9     12:23:57 07/02/18   111 days, 19 hours, 56 minutes and 41 seconds

And to know why, the command journalctl and the files under the directory /var/log/ are those that you need to look at.

Comments

[Aquarius Power]

did you discover what caused it?

[dr_]

I have never seen a Linux server crashing so hard that it reboots itself. Your servers must have something really wrong. Otherwise, it's a power failure issue.

[user2935706]

I got tons of system boot. There is no clue of why the system is booting.

[telcoM]

On server-grade systems, there can also be a hardware error log, viewable either by ipmitool or by a vendor-specific utility. Such a log could indicate if the system is shutting down because of overheating, loss of power, failure of too many cooling fans at the same time, failure of error-correcting RAM module, or some other hardware fault.

[Chaminda Bandara]

This command is not working on Red Hat 7.7