Why is my EC2 server's time off by ~10 seconds every day?

Question

I have what I believe is a system file, /etc/cron.daily/ntpupdate which runs
ntpdate ntp.ubuntu.com daily to sync with the network time. Every day it generates output very similar to this:

/etc/cron.daily/ntpupdate:
16 Jan 06:30:42 ntpdate[21446]:
step time server 91.189.94.4 offset -12.646804 sec

I'm not positive what the 91.189.94.4 means but I'm pretty sure -12.646804 sec means that my server is off by around 12 seconds. But I don't know why it is off by around the same amount every day. This is an Amazon EC2 instance running Ubuntu.

I can only guess that either it is losing / gaining 12 seconds per day, or something else is syncing the time with another clock that is off by 12 seconds and then I am re-syncing it.

What should I do to try and track this down further? I don't see any other cron jobs in the /etc/cron.* directories or in the users' cron jobs...

UPDATE

Just thought I'd share that I started running this hourly to see if there would be a big jump at a certain hour. This is what the hourly output is:

16 Jan 15:17:04 ntpdate[8346]:
adjust time server 91.189.94.4 offset -0.464418 sec

So apparently every hour the clock is off by around half a second, so that makes sense that each day (24 hours) the clock would be off by around 12 seconds. Guess the clock is just running fast! Thanks!

Solution

There are a number of factors that might make a software clock run slow or fast. Clocks on virtual servers are especially prone to a whole class of these problems. 12 seconds a day is pretty bad until you come across virtual boxes with clocks that run at 180–200% speed! Clocks on laptops that suspend can suffer from time-keeping issues too.

You should consider dropping ntupdate in favour of ntpd. The package name is ntp on Debian (and presumably Ubuntu too). The NTP daemon keeps your time in sync a lot more proactively than a cron job, synchronising with one or more other NTP servers and keeping your clock much more accurate. It's another implementation of the same protocol ntpdate uses, except ntpd monitors the time continuously.

If you don't want the (very small) overhead of ntpd, you might consider running ntpdate once an hour. Assuming you're 0.5s off every hour, that should be sufficient.

Comments

[phemmer]

The virtual machine issues can also be largely resolved by running a tickless kernel (CONFIG_NO_HZ). Not sure if this is possible with ubuntu kernel, or if you'd have to build your own.

[Simon Gates]

I just checked the kernel config on an Ubuntu 11 installation (3.0.0-14-generic), and it definitely has CONFIG_NO_HZ enabled.

[Gilles 'SO- stop being evil']

This is not true. On the contrary, NTP attempts to reduce the difference as much as it can. There is a threshold in the other direction: NTP won't update the clock if the difference is too large (it suspects a misconfiguration, e.g. wrong timezone).