Is this a network problem or a server crash? How do you identify what the real problem is when you can't ping?

Question

Today I was unable to ping my server remotely (CentOS 5.4, on a Dell). Then I tried traceroute and it showed destination host unreachable. This is very confusing because it should not do this.

When someone on-site physically restarted it, I was then able to log in again. I realize there is a problem but still cannot identify the root cause.

Here are the details:

1. The ISP uses the server's MAC address to automatically assign it a static IP via DHCP.

2. So the server was in auto DHCP mode

   • I left it as DHCP because the ISP will provide the same IP based on the MAC address.

3. For a few months it was working. What happened today is something I've never seen before.

4. After physically rebooting the server, it came back online, and I was able to log in again.

But still I am not sure if that was a DHCP and Static IP issue or an ISP cable issue or did the ISP itself have a problem? How do you, in a case like this, quickly identify -- or determine from the logs -- what caused this?

Aug  4 23:13:54 DellServer dhclient: bound to xx.xx.xx.1xx -- renewal in 3363 seconds.
Aug  5 00:09:57 DellServer dhclient: DHCPREQUEST on eth0 to xx.xx.137.3 port 67
Aug  5 00:09:57 DellServer dhclient: DHCPACK from xx.xx.137.3
Aug  5 00:09:57 DellServer dhclient: bound to xx.xx.xx.1xx -- renewal in 1823 seconds.
Aug  5 00:40:20 DellServer dhclient: DHCPREQUEST on eth0 to xx.xx.137.3 port 67
Aug  5 00:40:20 DellServer dhclient: DHCPACK from xx.xx.137.3
Aug  5 00:40:20 DellServer dhclient: bound to xx.xx.xx.1xx -- renewal in 3070 seconds.
Aug  5 00:59:42 DellServer auditd[2510]: Audit daemon rotating log files
Aug  5 01:31:30 DellServer dhclient: DHCPREQUEST on eth0 to xx.xx.137.3 port 67
Aug  5 01:31:30 DellServer dhclient: DHCPACK from xx.xx.137.3
Aug  5 01:31:30 DellServer dhclient: bound to xx.xx.xx.1xx -- renewal in 1658 seconds.
Aug  5 01:59:08 DellServer dhclient: DHCPREQUEST on eth0 to xx.xx.137.3 port 67
Aug  5 01:59:08 DellServer dhclient: DHCPACK from xx.xx.137.3
Aug  5 01:59:08 DellServer dhclient: bound to xx.xx.xx.1xx -- renewal in 2907 seconds.
Aug  5 02:47:35 DellServer dhclient: DHCPREQUEST on eth0 to xx.xx.137.3 port 67
Aug  5 02:47:35 DellServer dhclient: DHCPACK from xx.xx.137.3
Aug  5 02:47:35 DellServer dhclient: bound to xx.xx.xx.1xx -- renewal in 2060 seconds.
Aug  5 03:09:36 DellServer kernel: bnx2: eth0 NIC Copper Link is Down
Aug  5 03:09:38 DellServer kernel: bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex
Aug  5 03:21:55 DellServer dhclient: DHCPREQUEST on eth0 to xx.xx.137.3 port 67
Aug  5 03:22:29 DellServer last message repeated 3 times
Aug  5 03:23:34 DellServer last message repeated 5 times
Aug  5 03:24:36 DellServer last message repeated 5 times
Aug  5 03:25:39 DellServer last message repeated 5 times
Aug  5 03:26:53 DellServer last message repeated 6 times
Aug  5 03:27:59 DellServer last message repeated 5 times
Aug  5 03:29:04 DellServer last message repeated 5 times
Aug  5 03:30:10 DellServer last message repeated 5 times
Aug  5 03:31:13 DellServer last message repeated 5 times
Aug  5 03:32:21 DellServer last message repeated 4 times
Aug  5 03:33:27 DellServer last message repeated 4 times
Aug  5 03:34:38 DellServer last message repeated 5 times
Aug  5 03:35:39 DellServer last message repeated 4 times
Aug  5 03:36:49 DellServer last message repeated 4 times
Aug  5 03:37:50 DellServer last message repeated 4 times
Aug  5 03:38:56 DellServer last message repeated 4 times
Aug  5 03:39:58 DellServer last message repeated 4 times
Aug  5 03:41:13 DellServer last message repeated 4 times
Aug  5 03:42:20 DellServer last message repeated 5 times
Aug  5 03:43:40 DellServer last message repeated 5 times
Aug  5 03:44:50 DellServer last message repeated 4 times
Aug  5 03:45:51 DellServer last message repeated 4 times
Aug  5 03:47:02 DellServer last message repeated 4 times
Aug  5 03:48:07 DellServer last message repeated 6 times
Aug  5 03:49:19 DellServer last message repeated 6 times
Aug  5 03:50:08 DellServer last message repeated 4 times
Aug  5 03:50:23 DellServer dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67
Aug  5 03:50:58 DellServer last message repeated 3 times
Aug  5 03:52:13 DellServer last message repeated 4 times
Aug  5 03:53:18 DellServer last message repeated 5 times
Aug  5 03:54:27 DellServer last message repeated 6 times
Aug  5 03:55:41 DellServer last message repeated 6 times
Aug  5 03:56:48 DellServer last message repeated 4 times
Aug  5 03:57:54 DellServer last message repeated 5 times
Aug  5 03:59:03 DellServer last message repeated 5 times
Aug  5 03:59:09 DellServer NET[32604]: /sbin/dhclient-script : updated /etc/resolv.conf
Aug  5 03:59:09 DellServer avahi-daemon[3284]: Withdrawing address record for xx.xx.xx.1xx on eth0.
Aug  5 03:59:09 DellServer avahi-daemon[3284]: Leaving mDNS multicast group on interface eth0.IPv4 with address xx.xx.xx.1xx.
Aug  5 03:59:09 DellServer avahi-daemon[3284]: iface.c: interface_mdns_mcast_join() called but no local address available.
Aug  5 03:59:09 DellServer avahi-daemon[3284]: Interface eth0.IPv4 no longer relevant for mDNS.
Aug  5 03:59:09 DellServer kernel: bnx2: eth0: using MSIX
Aug  5 03:59:09 DellServer kernel: bnx2i: iSCSI not supported, dev=eth0
Aug  5 03:59:11 DellServer kernel: bnx2i: iSCSI not supported, dev=eth0
Aug  5 03:59:11 DellServer dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 6
Aug  5 03:59:11 DellServer dhclient: send_packet: Network is down
Aug  5 03:59:11 DellServer kernel: bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex
Aug  5 03:59:17 DellServer dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 14
Aug  5 03:59:31 DellServer dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 10
Aug  5 03:59:41 DellServer dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 12
Aug  5 03:59:53 DellServer dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 11
Aug  5 04:00:04 DellServer dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8
Aug  5 04:00:12 DellServer dhclient: No DHCPOFFERS received.
Aug  5 13:51:03 DellServer shutdown[1828]: shutting down for system halt
Aug  5 13:51:03 DellServer init: Switching to runlevel: 0
Aug  5 13:51:04 DellServer smartd[3426]: smartd received signal 15: Terminated 
Aug  5 13:51:04 DellServer smartd[3426]: smartd is exiting (exit status 0) 
Aug  5 13:51:04 DellServer avahi-daemon[3284]: Got SIGTERM, quitting.
Aug  5 13:51:07 DellServer xinetd[3026]: Exiting...
Aug  5 13:55:06 DellServer syslogd 1.4.1: restart.
Aug  5 13:55:06 DellServer kernel: klogd 1.4.1, log source = /proc/kmsg started.

Solution

Here's what your log seems to show (to me anyway):

The link went down, and then came back up.

Your DHCP client waited for a long time to hear from the DHCP server to get it's old address back, but it didn't hear from the DHCP server so it declared the network was down (even though the link was up).

It then made some network adjustments and tried again (probably requesting any IP address, from any DHCP server), but again never managed to discover a DHCP server, and so at that point it probably gave up completely, waiting for someone to fix it and reset the network connection (which you did by rebooting; resetting the NIC from within the OS may have been sufficient).

Since we have no idea what hardware or configuration(s) exists between your server's network port and the ISP, it's almost impossible to say what failed, especially when based on just (this) one incident report.

Suggestions:

• Your server's network configuration should be hard-set and not dependant on DHCP.
• You should have auto-restart (and fail-over) facilities in place if you want to maintain high up-time without your intervention.

Comments

[user1686]

Have you checked /var/log yet?

[Zoredache]

The best method is almost always to look at see what is on the console of the server instead of just blindly rebooting it. You could have had a kernel panic, or some other things break that would have been pretty obvious if you had the opportunity to see the console.

[YumYumYum]

@grawity: Please check above the log. Does this mean mid night my server network interface is crashed or damaged?

[Tamara Wijsman]

I don't see a point in closing this question, it's perfectly valid.

[YumYumYum]

I have it now static (hard-set). But is it not CentOS bug, because he should be retrying after several hours later, because it was assigned to DHCP, and itself no where mentioned that he will shutdown his interface until someone physically go to on-site?

[YumYumYum]

I have also other problems in Fedora 15. When it use lan/wireless and randomly get changed. It does not smoothly gets up requires manually $service NetworkManager restart. Isn't this CentOS bug? Because what happend with me it was very hard to trace, and i am sure ISP wont do that because the server was in ISP rack.