Foreman displays a server as "out of sync", what could be the reason and what is the right way to troubleshoot such an issue?

Question

There's a puppet server in the company I work for with Foreman installed.

There are around 200 Linux machines which are being managed by the Puppet server.

When looking in the Foreman GUI, most server appear with a green icon saying "No changes" but there are some servers which appear with a yellow/orange icon with a tooltip saying "out of sync".

You can see in the following screen shot that most servers appear with green status and only server "sgproxy08" is out of sync:

Running puppet agent -t on the sgproxy08 server finishes properly:

[root@sgproxy08 ~]# puppet agent -t
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for sgproxy08.sg.company.com
Info: Applying configuration version '1436862601'
Notice: Finished catalog run in 49.90 seconds
[root@sgproxy08 ~]#

Now there are two questions:

1. What are the requirements for this icon to become green and in sync?
2. How can I troubleshoot such an issue?

Solution 1

I found the reason for the out-of-sync notifications in Foreman. It seems like in these servers there was a stuck process of puppet agent -t, is looks like so:

root  7531 21487  0 Aug09 ?   00:00:03 puppet agent: applying configurat

Once I killed this process and ran puppet agent -t again - problem solved.

Solution 2

Hosts may be out of sync due to host/server restart or network issues or puppet version mismatch between the server and agent.

For the version mismatch, you can upgrade puppet agent to match the server version and sync host via "puppet agent -t" .

If you have remote-execution and remote-execution-ssh plugin for foreman enabled, you can run the "puppet agent -t" command from Foreman console for out of sync host. Below are the steps :

1. Enabling remote-execution plugins in foreman server , may require server restart

   sudo foreman-installer --enable-foreman-plugin-remote-execution --enable-foreman-proxy-plugin-remote-execution-ssh
   

2. Syncing host

   • Goto “Hosts → All Hosts” and select the check box next to the out of sync host
   • A “Select Action” button will appear on top right of the hosts table. Select the “Schedule Remote Job” option from the action list.
   • In the displayed form, keep the default values and copy “sudo puppet agent -t” (without quotes) in “command” text box. You may need to provide full path to puppet (eg. /opt/puppetlabs/puppet/bin/puppet) if puppet binary is not in host's PATH. Click submit.
   • Once the command executes remotely in the host, it will sync with the server and Foreman should update the host status to sync after few minutes.

Solution 3

If you're on Foreman 1.8 or lower, your host will be shown as 'out of sync' only when Foreman has not received any reports for that host for 5 minutes longer than the setting 'puppet_interval' (check it in Administer > Settings). By default this would be 35 minutes. There's an 'outofsync_interval' setting that allows you to customize this on Foreman 1.9.

In order to start troubleshooting this (missing reports), I would look at /var/log/foreman/production.log on your Foreman boxes and check if you receive the report from that host when Puppet finishes its run.

Comments

[Govindarajulu]

Are the servers marked as 'out of sync' always the same ones?