ntpdate and ntpd failing to sync clock on Linux

Question

I have a strange problem with one of my servers. ntpd and ntpdate fail to work, but debugging shows no errors at all. At first I thought maybe a local or network firewall was blocking UDP port 123, but that is not the case- this server can talk UDP port 123 (the ntp protocol) to the Internet and get answers.

Let me demonstrate the issue.

date -s "30 DEC 2012 02:30:00" - works, so I can successfully set the clock without error.

ntpq -pn pool.ntp.org - works, I do get detailed time data from the timeserver, and proves UDP packets are functioning.

ntpdate -d pool.ntp.org - debug mode works, shows a ton of debug data and shows the current time offset: 30 Dec 02:38:56 ntpdate[19267]: step time server 208.97.140.69 offset 228.234554 sec

All looks normal, until: ntpdate pool.ntp.org - after a 4.7 second pause, it returns: 30 Dec 02:41:29 ntpdate[19274]: no server suitable for synchronization found

Similar problem running ntpd, it does not update the clock.

After ntpd is started, ntpq -pn results in all refid's forever stuck on .INIT. which means they cannot sync.

/var/lib/ntp/drift is the driftfile setting in ntp.conf, which is chmod 644 and owned by ntp:ntp, same as all my other systems.

I tried a dozen other ntp time servers, disabled iptables firewall, and confirmed the datacenter is not filtering udp traffic. Any ideas what is stopping ntpd and ntpdate from syncing my clock?

This is CentOS 6.3 x64 on a dedicated server with Intel CPU.

Solution 1

ntpdate (and ntpd) will refuse to (easily) set the time if the offset is too high. Both applications will try to slowly adjust your time, so as to not to confuse your system or any applications that may not handle big time jumps very well.

Try ntpdate -b instead. It will set the time no matter how unreasonable it may seem.

You may also need to add the -u flag, which will prevent ntpdate from using privileged (<1024) ports. Note that -u is implied by -d! And it sounds like -d is working fine.

If adding -u makes a difference between working and non-working, then you have a firewall on the way that is causing these issues.

And unfortunately it doesn't seem to be possible to make ntpd use an unrestricted port.

Solution 2

Can you provide the following outputs in pastebin.

cat /etc/ntp.conf
cat /etc/sysconfig/ntpd
ntpq -pn
ntpdc -c sysstat
ntpdc -c kerninfo
ntpdc -c loopinfo
ntpdate -d <time-server-IP>
ntptrace

Are you syncing from stratum 1 servers or anything else.

No server suitable for synchronization means what it tells, that the communication between client and server cannot be established.

If we can't find clues from this set of data, tcpdump might be required to see where the packet is being lost.

tcpdump -s0 -i ethX -p udp -w /tmp/ntp.pcap

Stop and start ntpd daemon and wait for the reach to go to 377 and then stop the tcpdump. That should give further clues.

Comments

[ravi yarlagadda]

Can you clarify what you mean by "dedicated server" - is it physical hardware, or a virtual machine?

[Crash Override]

Dedicated server = physical hardware. NOT a virtual machine.

[Crash Override]

Still fails. ntpdate -b pool.ntp.org results: 30 Dec 03:00:10 ntpdate[1341]: no server suitable for synchronization found The ntpdate debug flag which is -d will show debug data but not actually sync, and that works: ntpdate -d pool.ntp.org results: 30 Dec 03:00:55 ntpdate[1343]: step time server 128.10.254.6 offset 228.030338 sec

[chutz]

@CrashOverride, I updated my answer to try to explain why -d may be working while otherwise it is not.

[Crash Override]

Humm, maybe there is a firewall blocking source UDP port 123 from my server. Checking that. Again, thanks for your answer.

[chutz]

I updated with a link to explain that it is not possible to change the ntpd source port. Sorry, I haven't looked further than finding that link.

[Crash Override]

Thanks. Looks like ntpd only uses UDP source port 123, so if something blocks UDP source port 123, it won't work. My case is strange, because the datacenter swears they are not blocking any UDP traffic. My iptables is OFF, and I tried explicitly accepting outbound UDP 123 and still fails. If I do nmap -sU -p 123 --source-port 123 130.88.200.4 is shows filtered which usually means no response. If I do nmap -sU -p 123 --source-port 124 130.88.200.4 it shows open meaning it got a response. Port 124 here, but trying any source port other than 123 works. Source port 123 fails.

[Crash Override]

pastebin.com/Rw0nZTA8

[skeggse]

Worth noting that ntpdate has been depricated, and when moving to ntpd -q, use -g instead of -b.

[killthrush]

One thing I noticed is that you should probably sudo or use root with ntpdate -b -u. I was getting permission errors when I tried it, which were oddly omitted from stdout when using the -d flag.