Set different warning thresholds for Nagios on weekends

Question

I'm using Nagios to monitor CPU usage on my servers with the check_load plugin.

On weekends we have some batch jobs running and there are hardly any real users anyway, so I'd like to have the warning threshold a little higher.

I have seen solutions to completely disable notifications during these times, but I'd like to still get notifications if the load gets really high.

I've also see solutions to defince 2 services, one for weekday and one for weekends, but I'd like to keep the CPU load in a single chart for the whole week.

Any ideas or examples how I can have different thresholds in a single service ?

Solution

The way to do this depends on where your thresholds are defined. If they're defined on the host being checked, e.g., hard-coded in nrpe.cfg, then you need to keep two copies of nrpe.cfg and rotate them on a schedule. (And restart NRPE every time it swaps.) This can be done with a cron job and a simple bash script.

If you have NRPE accepting args, and are passing the thresholds in as ARG1, ARG2, etc., from the Nagios end, you have a few options.

You can define two nearly-identical services, with different/non-overlapping check periods and different thresholds passed as args.

Or you can use one service, and dynamically change the threshold arguments. It's complicated, but it can be done. It's called Adaptive Monitoring.

The basic idea is to submit external commands on a schedule to change object settings. In your case, you'd want to change the check_command to pass in different thresholds. Something like... change check_nrpe!check_load!5 10 to check_nrpe!check_load!15 30 or similar.

For adaptive monitoring, what you're looking for is the external commands that start with "CHANGE" in the command list, specifically CHANGE_SVC_CHECK_COMMAND in this case.

Comments

[naing linhtut]

Replacing the command incl. parameters would do. Is there an example config somewhere ? The documentation says that the feature won't be useful for 99% of the users...

[Keith]

Adaptive monitoring is an "advanced" topic, which is why the docs say 99% of people won't need it. Edited answer to include more info, if you really want to go that route.