Synchronize the date and time on compute nodes

Question

I want to set a job on compute nodes but I checked and see time and date of compute nodes are not sync with the head nodes.

Head node shows: Wed Jul 22 16:57:32 PDT 2015
node 2 shows: Wed Jul 22 11:52:21 PDT 2015
node 3 shows: Wed Jul 22 12:00:38 PDT 2015

I want to sync it with the head node without any changes in time of head node (because I have some jobs in head node which are related to the time so it is important not to disturb them). Do you know how I can sync them with the head node?

Somebody told me to restart the ntpd service on the Compute Nodes using service ntpd restart command but I don't know whether this affects the head node or not?

Solution 1

Assuming you run SSH on the head node, you could simply retrieve the current time from it using the date command over SSH.

ssh user@host date

It's easiest and safest to authenticate with the head node using private key authentication, so you are not required to type in a password to login.

On the slave nodes you can set the current time using the obtained value using the following command as root:

date --set=STRING

where STRING is the string value obtained from the head node. This can be combined into one command using command substitution

date --set="$(ssh user@host date)"

I admit this is a bit of a poor man's time synchronization solution, but I don't see why it wouldn't work. Just turn off your time synchronization service altogether, if you have any, so the changes by your script aren't tinkered with, e.g.:

/etc/init.d/ntpd stop

You didn't provide much information about your systems, so this is as much as I can do. It's better to configure a proper time synchronization tool, but for that we need more information about your system, e.g. what distribution you are running.

Solution 2

If you have root access on the nodes, then to set up time synchronization. The most commonly used tool for this is ntpd, available in all but the most specialized Linux distributions (as well as other Unix systems). Ntpd periodically sends queries over the network to obtain the current time, measures the drift of the local clock and tunes the local clock to compensate for its skew. You can get better than millisecond precision over a LAN.

For best results, select one or a few nodes as servers, e.g. the head node(s). Install the NTP server package (e.g. ntp on Debian/Ubuntu/Mint as well as on Fedora/RHEL/CentOS) and configure the servers to be your head node(s) as well as some Internet servers. In the configuration file /etc/ntp.conf, you should have something like

server 0.fedora.pool.ntp.org
server 1.fedora.pool.ntp.org
server 2.fedora.pool.ntp.org
server 3.fedora.pool.ntp.org
server head1.mycluster.internal.example.com
server head2.mycluster.internal.example.com

(leave the lines set up by your distribution and add yours). On the head nodes, if there is a restrict line, make sure it allows the IP addresses of all the nodes. On the other nodes, you can have restrict 127.0.0.1.

Before you set up NTP, set the clocks to approximately the right time. Ntpd will not synchronize the time if it's more than about 20 minutes off (it assumes that you've set up the timezone incorrectly, or there's something very wrong about the clock or about the network). So first set the clock manually, e.g. first set the correct date and time on the local machine, then you can use

ssh somenode "date -d $(date +%m%d%H%M%Y.%S)"

to set the clock of somenode to a few tens of a second behind the local clock. Once you've done that, install ntp, or, if it's already installed, restart the daemon with service ntpd restart or whatever method your distribution uses.

Comments

[roaima]

If the head node's time is wrong won't that invalidate the results of your time dependent jobs running there?