/var/log/syslog 'systemd[1]: Time has been changed' message every 5 seconds

Question

I have a problem. There are messages like

Mar  1 15:52:34 ubuntu systemd[1]: Time has been changed
Mar  1 15:52:34 ubuntu systemd[1]: snapd.refresh.timer: Adding 1h 16min 25.978183s random time.
Mar  1 15:52:34 ubuntu systemd[1]: apt-daily.timer: Adding 8h 21min 914.167ms random time.
Mar  1 15:52:34 ubuntu systemd[6041]: Time has been changed

every 5 seconds in /var/log/syslog. Looks like something is changing time constantly. It is really annoying and i understand that something wrong and would like to fix it. Also RTC time isn't correct.

Some info about my system:

• Ubuntu 16.04, 4.4.0-64-generic, x86_64, running on VDS

timedatectl:

      Local time: Ср 2017-03-01 16:05:03 MSK
  Universal time: Ср 2017-03-01 13:05:03 UTC
        RTC time: Пт 2017-03-03 11:33:10
       Time zone: Europe/Moscow (MSK, +0300)
 Network time on: yes
NTP synchronized: yes
 RTC in local TZ: no

iptables:

Chain INPUT (policy DROP)
Chain OUTPUT (policy ACCEPT)

The INPUT chain has my rules inside.

systemctl list-units | grep running

acpid.path                  loaded active   running   ACPI Events Check
init.scope                  loaded active   running   System and Service Manager
acpid.service               loaded active   running   ACPI event daemon
asterisk.service            loaded active   running   Asterisk PBX
atd.service                 loaded active   running   Deferred execution scheduler
cron.service                loaded active   running   Regular background program processing daemon
dbus.service                loaded active   running   D-Bus System Message Bus
[email protected]          loaded active   running   Getty on tty1
hv-kvp-daemon.service       loaded active   running   Hyper-V KVP Protocol Daemon
hv-vss-daemon.service       loaded active   running   Hyper-V VSS Protocol Daemon                                                                
iscsid.service              loaded active   running   iSCSI initiator daemon (iscsid)                                                            
lvm2-lvmetad.service        loaded active   running   LVM2 metadata daemon                                                                       
lxcfs.service               loaded active   running   FUSE filesystem for LXC                                                                    
mdadm.service               loaded active   running   LSB: MD monitoring daemon                                                                  
mysql.service               loaded active   running   MySQL Community Server                                                                     
[email protected]      loaded active   running   OpenVPN connection to server                                                               
rsyslog.service             loaded active   running   System Logging Service                                                                     
snapd.service               loaded active   running   Snappy daemon                                                                              
ssh.service                 loaded active   running   OpenBSD Secure Shell server                                                                
systemd-journald.service    loaded active   running   Journal Service                                                                            
systemd-timesyncd.service   loaded active   running   Network Time Synchronization                                                               
systemd-udevd.service       loaded active   running   udev Kernel Device Manager                                                                 
[email protected]           loaded active   running   User Manager for UID 1000                                                                  
vpsguard.service            loaded active   running   VPSGUARD                                                                                   
acpid.socket                loaded active   running   ACPID Listen Socket                                                                        
dbus.socket                 loaded active   running   D-Bus System Message Bus Socket                                                            
lvm2-lvmetad.socket         loaded active   running   LVM2 metadata daemon socket                                                                
snapd.socket                loaded active   running   Socket activation for snappy daemon

I have tried:

• sudo systemctl stop systemd-timesyncd.service
• installing/uninstalling NTP
• sudo hwclock -w

I don't have sclockadj installed. Also I asked my VDS provider if they block any NTP requests. No, they don't.

Update 1
I set up another one VDS#2 completely the same. systemd[1]: Time has been changed was getting this message couple minutes, after that message CRON[3684]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) showed up and the problem has gone. However I have tried to upgrade VDS#2 and right after reboot have this time has been changed again.

Found this thread, maybe it can give more information. Looks like it's a kernel issue.

Tried to downgrade kernel, systemd, udev, but nothing from this doesn't work. And as I can see from some replies, it's not a problem with my hoster and virtualization only.

Solution 1

I encountered this issue of systemd[...]Time has been changed messages logged every five seconds in /var/log/syslog on a 16.04 server running under Windows 8.1 Hyper-V. To fix it, I disabled time synchronization on the Hyper-V side.

In Hyper-V Manager, I highlighted the VM, selected Settings, then Integration Services, unchecked Time synchronization, and clicked Apply.

The messages stopped instantly - no VM restart was required.

Solution 2

On HyperV/Azure, the message can be disabled by creating a filter for syslog.

1. create a new file in /etc/rsyslog.d

   vi /etc/rsyslog.d/15-excludetime.conf 
   

2. insert two lines

   :msg, contains, "Time has been changed" ~ 
   :msg, contains, "apt-daily.timer: Adding" ~
   

3. restart rsyslog

   systemctl restart rsyslog.service
   

Or you can disable the host time sync on VMBUS. I put the line under /etc/rc.local

echo 2dd1ce17-079e-403c-b352-a1921ee207ee > /sys/bus/vmbus/drivers/hv_util/unbind

Comments

[Russel FP]

I asked my provider to disable time synchronization and it fixed my problem. Thanks for your help!

[Adam Plocher]

I'm having this issue with a VM on Azure. While I wouldn't be surprised if it's running Hyper-V (or some variation of it) at its core, I don't believe that's an option for me :-/

[PatricF]

Such an easy fix. Thank you very much for this!