Why is ntpd not updating the time on my server?

Question

I have ntpd running on my server. It's all the default settings, except I commented out its ability to be a server to other machines:

# restrict -4 default kod notrap nomodify nopeer noquery                                                                    
# restrict -6 default kod notrap nomodify nopeer noquery   
restrict default ignore

If I run ntpdate -q ntp.ubuntu.com, I'm told that my machine's clock is off by 7 seconds.

What's going on? How can I diagnose what's happening, is there a log I can turn on?

more info #1

# ntpq -np
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 91.189.94.4     193.79.237.14    2 u   30   64    7  108.518   -0.136   0.361

more info #2

Here's what this looked like when I asked the question:

# ntpdate -q ntp.ubuntu.com
server 91.189.94.4, stratum 2, offset 7.191308, delay 0.13310
10 Jan 20:38:09 ntpdate[31055]: step time server 91.189.94.4 offset 7.191308 sec

And here's what it looks like now, after restarting ntpd a couple times (I'm assuming that's what fixed it):

# ntpdate -q ntp.ubuntu.com
server 91.189.94.4, stratum 2, offset 0.000112, delay 0.13164
10 Jan 20:47:03 ntpdate[31419]: adjust time server 91.189.94.4 offset 0.000112 sec

more info #3

I uninstalled ntp and installed openntpd and ran /usr/sbin/ntpd -d, and I'm seeing output like this:

reply from 64.73.32.134: offset 6.715003 delay 0.041152, next query 30s
reply from 208.53.158.34: offset 6.700224 delay 0.036263, next query 31s
adjusting local clock by 6.734120s
reply from 72.18.205.156: offset 6.708575 delay 0.035885, next query 30s
reply from 64.73.32.134: offset 6.701463 delay 0.044199, next query 33s

Which to me pretty clearly indicates that I'm not able to set the time on my server (although, with regular ntp, it does seem to update sometimes...).

more info #4

My VPS provider says:

The latest kernels should not lock your system to our dom0's clock, to be on the safe side you can set xen.independent_wallclock = 1 in your sysctl.conf.

Which I suppose still does not address the issue of the VPS needing a CPU available in order to do correct timing calculations.

Solution 1

Alright folks, in the time since asking this question, I've reinstalled ntp with the default vendor (Ubuntu 10.0.4) config and let it run for a few days. As of this writing, ntpdate -q ntp.ubuntu.com shows that my time is accurate to within 0.000216 seconds. So, the problems I was having must have been with my customized config (where I was trying to make it impossible for external hosts to query my server, which I'm already doing with my firewall so I'm not too worried about). Here is the Ubuntu 10.0.4 ntp.conf in its entirety, with comments removed:

driftfile /var/lib/ntp/ntp.drift

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable

server ntp.ubuntu.com

restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery

restrict 127.0.0.1
restrict ::1

I welcome feedback on how this config might be improved.

I also made a ticket with my VPS provider asking them for a detailed recommendation on the best thing to do. I pointed them to this thread, and some other documentation indicating that maybe the CPU allocation would cause a timing problem. Here is what they said:

The latest kernels should not lock your system to our dom0's clock, to be on the safe side you can set xen.independent_wallclock = 1 in your sysctl.conf. This will make sure the server instance isn't following the clock on the host server.

and:

I think you may be mis-understanding the exact degree to which this issue affects NTP clients in a virtualized environment. In my experience on a virtualized system on a Xen host (such as our setup at Rackspace Cloud) the inaccuracy inherited by not having a dedicated system clock to process the interrupts amounts to fractions of a second, even on highly loaded systems. This slight inaccuracy is easily managed by NTP even if it is only set to update the servers time once per day (or even less frequent than that).

Solution 2

You can enable logging in ntpd by adding this to ntp.conf:

logfile /var/log/ntpd.log

Source: ntp manual

If you turn off ntpd, can you update the clock by command line? If you run the ntpdate command and receive an error like so:

# ntpdate ntp.ubuntu.com
10 Jan 23:47:57 ntpdate[26284]: Can't adjust the time of day: Operation not permitted

This means that you are probably on a VPS, and in that case you can not modify the system clock - this can only be done on the host machine.

Solution 3

One of your comments says you are running on a vhost. In this case, you are probably not going to have much success because your vhost's sense of time is going to be dependent both on the real host it is running on and by how busy overall the vhost is.

Depending on the virtualization used, the vhost may not get a steady share of interrupts in a given time period. This will make the clock run faster, or slower, than is really happening. Since ntp is trying to measure changes on the assumption that your clock is a fixed-rate faster or slower than the rest of the world, this speeding up and slowing down will give ntp fits and it will probably eventually just give up, with the result that ntp -np shows time servers that ntp has deemed unsuitable.

Your best bet if this is the case is probably a brute-force rdate -s $server every so often (like every six hours) to yank the clock around by its nose so that it doesn't drift excessively out of sync. But fine-grained accuracy is probably out of reach.

Solution 4

• If ntpd wouldn't be able to connect with the remote server, you wouldn't see an offset for that server.
• If ntpq would be blocked by ntpd, you would see a clear error message from ntpq.
• If some other service would set also the time (like vmware tools), you would see a jumping offset for the server (run ntpq -p every 70 seconds).

The reach 7 in ntpq output indicated, that you let ntpd only run for around 4 minutes. 7 is 111 binary, which means the server was reached already 3 times. ntp reaches out every 64 seconds (poll value) and waited already 30 seconds (when value) since the last contact.

The offset -0.136 indicated, that the system is already synchronized. Only ntpd hasn't marked the server as source, yet. Just give it more time and a little star will appear.

So, actually your ntpd was syncing. But ntpd doesn't usually sync in one big jump (like ntpdate), but tries to adjust the time slowly and ensures over several cycles, that the time is stable.

PS: I am aware, that the question is very old. But the issue is timeless. And all the other answers are just misleading IMHO. ntpd is even recommended by VMWare to keep the time in sync.

Solution 5

Things I've found in the past, when I used ntpd instead of openntpd:

1. You need to permit access to localhost for ntpd to start properly and actually do stuff

   restrict 127.0.0.1
   restrict ::1
   

2. Although you can use hostnames for server rules, opening back up holes to talk with those servers means using restrict which requires IP addresses, so I ended up having to use IPs for everything anyway.

3. You don't mention using restrict to open access back up to your servers. That's an issue. Try blocks such as the following:

   # ntp.xs4all.nl
   server          194.109.22.18
   restrict        194.109.22.18
   

4. You need multiple peers or servers for ntpd, since it tries to use majority-rules voting to deal with a bad actor. So a minimum of 4, to still be able to have a majority when you lose one, preferably 5.

5. To lock down the default access, I could use:

   restrict default notrust nomodify
   

   so as to still be able to query, but I ended up using restrict default ignore as you do when ntpd 4.2 changed the meaning of notrust. sigh

6. If you're not providing time-service to others, then you probably don't need the full power of regular ntpd and you should consider openntpd instead. Written by the OpenBSD crew, it's a far more minimal implementation, using privilege separation and a much simpler config file. It allegedly won't provide the highly precise time that ntpd will, but it's easily good enough for a regular server or workstation.

View more solutions

Comments

[David Mackintosh]

Is that your entire config file? If you run ntpq -np, what is the output?

[Aaron Copley]

Where's the rest of the config? There's no upstream server for your host to get time from.

[John Bachir]

no, it's not my entire config file, it's just the part i changed from the vendor config file. editing my question to show the output if ntp -np

[Aaron Copley]

Got it. It looks like ntpd was working normally. NTPd will "slew" your clock back into sync gradually. A sudden change in time can cause big problems for certain running processes so NTP works by speeding up or slowing down the length of a second to gradually make adjustments.

[John Bachir]

My server has been running for many months, so I don't think slewing is the issue -- unless every time I reboot it gets reset to the underlying hardware's clock or some such thing.

[Aaron Copley]

Yes, the kernel will start with the hardware clock on boot since on boot that is it's only reference. If it's been running for many months, as you say, then that isn't it. You can tell NTP to sync to your hardware clock. I'm not sure about Ubuntu but on Red Hat based systems that's in /etc/sysconfig/ntpd. You can look there or refer to your hardware's documentation.

[Aaron Copley]

I also don't think you understand that ntpdate is a standalone application. It doesn't have anything to do with ntpd and shouldn't be used to troubleshoot it. The reason that ntpq was suggested with the -p options to show peering. If ntpd sees your peer(s), then it should be bringing the system back into sync. Looks like everything is good now, though. I just hoped to provide some extra insight. Hope this helps in the future!

[John Bachir]

ntpdate is happy to do its thing -- But I wonder if when my server reboots the clock gets reset to the hardware's clock or some such thing?

[Dave Drager]

After you use ntpdate to set the clock, use 'hwclock --systohc' to sync "running" time to your hardware clock. It is supposed to sync on a reboot but if your machine crashes (or otherwise had a problem doing a proper shutdown) it could not sync it.

[John Bachir]

Well it's a vhost, so I don't have access to the hardware clock (at least I hope not!)

[Keith Stokes]

There's an emulated hardware clock just as there is emulated BIOS.

[John Bachir]

My hosting provider (rackspace cloud) has told me that NTP works fine in their environment.

[Dave Drager]

I was the admin on several VPS platforms, none of them (openvz, Xen) have access to set the clock on the system. They all had to be done at the host level. Submit a ticket with your host to indicate the time is off, they should be running ntp and have the time synchronized for you.

[John Bachir]

This is great info. I'm checking out openntpd. Question: do you concur or disagree with the other folks who claim that setting the clock on a vhost is impossible?

[John Bachir]

And perhaps you can answer this question: serverfault.com/questions/223511/…

[John Bachir]

Is it possible that sudo ntpdate ntp.ubuntu.com succeeds, but isn't "really" working? Or does the failure of that command correspond 1-1 with my inability to set the clock in a VPS?

[John Bachir]

I don't understand what you are saying with your various restrict rules… do those rules affect which servers I can query for time too? I thought it only affected which nodes can ask me for time.

[John Bachir]

Here's an idea: wanna change your answer into a complete minimal ntp.conf file with comments? :-)

[Phil P]

Setting the clock on a vhost is ill-advised, unless you're guaranteed to always be scheduled with at least one CPU, since otherwise the time perceived by the vhost does not match the time of the outside world. The dom0 should be maintaining time. The other question's answer is good. NTP is UDP, so you need to permit packets from the servers which you query for time. My ntpd is stale as I moved to OpenNTPD a few years ago.

[John Bachir]

Phil P: Thanks for all your info and ideas. See my submitted and accepted answer for my resolution to this, and let me know what you think of that config.

[John Bachir]

See my submitted/accepted answer for what my VPS provider said about the clock, and access to setting the time.

[Cerin]

You might also need to use sudo...

[rackandboneman]

automatic rdate can set the clock backwards, which can have unexpected consequences aplenty.

[Naveed Abbas]

Great first answer, Robert. Welcome to the site.