How to deal with the amazon ec2's clock drift?

Question

We have a couple of ec2 instances on amazon, and we used amazon AIM Linux to set up. The instance type is m1.medium.

Everything is going well until this week.

We found from log file that there would be a huge clock drift happened suddenly, and sometimes there has a 5-minutes discrepancy。

I would like to ask if anyone else encounter this problem? How to deal with it?

Solution 1

Time drift can occur due to over utilization of your instances resources. I would encourage you to check you cloudwatch CPU graphs to try identify if your CPU was being over utilized.

If you are running amazon linux AMI, NTP is installed by default.

To verify if NTP is running run

sudo service ntpd status

If it is not installed you can run the following to install it:

sudo yum install ntp

If NTPD is running and your clock time is off I would try to restart it and see if it synchs with the NTP servers your have setup. You can also add more NTP servers by editing the /etc/ntp.conf file. Just search for NTP servers to find ones to add.

Solution 2

In 2017, Amazon launched the Amazon Time Sync Service. From that announcement post:

You can access the service via the link local 169.254.169.123 IP address. This means you don’t need to configure external internet access and the service can be securely accessed from within your private subnets.

They go on to recommend Chrony instead of ntpd.

Solution 3

Yes, best not to trust the system clock. Just install and configure NTP.

Solution 4

Yes even I faced this issue. Worst part is that I use SES to send emails; and emails sending started failing with the error "Time mismatch".. So, because of AWS one issue; another issue happened.

Restarting the server fixed the problem with immediate effect. Also, create a customer support ticket and let them know. Its a bug on their end. I think this happen with some of their instances. Because of this issue; my business was impacted very adversely hence I even got some dollars credit.

View more solutions

Comments

[user2028644]

How to stop the system? Can you give some advice? Thanks!

[chrskly]

I've not used Amazon's own Linux AMIs, but, as far as I'm aware, they're basically RHEL. Here's a guide to setting up NTP on RHEL6: access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Li‌​nux/…

[The Internet]

Did you mean sudo service ntp status ?

[Ciaran]

No when you install ntp it runs as a daemon (ntpd). Example: sudo service ntpd status ntpd (pid 1312) is running... sudo service ntp status ntp: unrecognized service

[warvariuc]

In Ubuntu the service is ntp not ntpd. So sudo service ntp status for Ubuntu is the right command.