NTP service stopped in centos7.1 linux machine

Question

In our 3 linux boxes centos 7.1, we saw below messages in the logs and then ntpd service got stopped since the offset is more than 1000s.

systemd: Time has been changed

ntpd[2626]: 0.0.0.0 0617 07 panic_stop -26789 s; set clock manually within 1000 s.

All these 3 linux boxes are under ESX machine. Also note many other linux boxes are under same ESX machine apart from these linux boxes.

We have external NTP servers configured in those boxes and no issues with those ntp servers.

Given the above scenario, What are the possibilites where system clock gets changed apart from manual intervention since this issue caused NTP service to be stopped automatically.

Solution

Speculative answer: on-board clock might drift if power is cut. Maybe on-board battery is nearly dead. If machine is powered down for a while and power is cut off then the time set upon reboot might be outside ntpd's max allowed adjustment.

If you are on VMs then only the service on the VM server should need controlling.

I have a CentOS 7.1 machine currently (not a VM) . . . During this month it had power down of 47min + 57min + 1day7min + 2min. There was some electrical work done in machine room. Look at 'last -x shutdown reboot':

[root@boxymcboxface ~]# last -x shutdown reboot 
reboot   system boot  3.10.0-229.el7.x Sun Jan 15 16:41 - 16:43 (8+00:02)   
shutdown system down  3.10.0-229.el7.x Sun Jan 15 16:38 - 16:41  (00:02)    
reboot   system boot  3.10.0-229.el7.x Sun Jan 15 16:16 - 16:38  (00:22)    
shutdown system down  3.10.0-229.el7.x Sat Jan 14 09:09 - 16:16 (1+07:07)   
reboot   system boot  3.10.0-229.el7.x Fri Jan 13 12:18 - 09:09  (20:50)    ** first ntpd panic_stop seen @ Jan 13 12:38:39 **
shutdown system down  3.10.0-229.el7.x Fri Jan 13 11:21 - 12:18  (00:57)    ** down for 57 mins **
reboot   system boot  3.10.0-229.el7.x Tue Nov 22 11:49 - 11:21 (51+23:31)  
shutdown system down  3.10.0-229.el7.x Tue Nov 22 11:02 - 11:49  (00:47)    

The first panic_stop message:

ntpd[733]: 0.0.0.0 c617 07 panic_stop -1027 s; set clock manually within 1000 s.

It would be interesting to see what clock is set to after each reboot. But only the latest message can be seen. 'dmesg |grep clock':

[    0.810823] rtc_cmos 00:08: setting system clock to 2017-01-15 16:40:57 UTC (1484498457)

So it looks like over the space of 57mins when probably maybe the power was out for ~30mins~ the clock drifted out (too fast) by 17mins.

Comments

[Michael Hampton]

Why are you running ntpd? The system came with chrony already installed and enabled. Don't run two different ntp daemons.

[xrkr]

Michael, I completely agree that chronyd is already installed and much faster compared to ntpd. We have stopped chronyd and disabled completely inorder to avoid conflict. However the question here why systemd observed the time change and what made system clock to be changed?

[Andreas Rogge]

Did this happen during normal operation or did it happen a short time after you rebooted the VMs? I've seen RTC configuration problems with VMs where the VM would always come up with the RTC set to localtime whereas the OS running in the VM wants the RTC set to UTC.

[xrkr]

Jones, Read the article and article doesn't talk anything about centos 7+ versions. Moreover, we have this vmtools time synchronization to false as well.

[Geraint Jones]

Okay - so it is recommended that you run NTP with no special boot arguments. in your ntpd config add maxpoll 6 and minpoll 4 so that the clock is updated very frequently (16s with these options)

[xrkr]

Okay How do we even conclude it is because of hypervisor that drift happened. and system clock got changed?

[Geraint Jones]

Mostly experience/gut feeling

[xrkr]

Experience/gut feeling doesn't work out unless we have proof :)