How to calculate maximum system load average for my system?

Question

I have a situation where I wanted to monitor my servers load, for this I am trying to set a threshold, but I am not sure what is the maximum value my server can reach.

This is the current system load average:

0.23, 1.52, 2.69

sudo nproc --all returns value as 6

and here is the CPU details:

Does this mean my server can reach system average load upto 6?

Solution 1

There isn't really a maximum load average. It can increase based on other resources besides pure CPU processing such as waiting for IO a whack of other stuff.

But you'd expect see 1 per core on a fully utilized system when it is CPU bound.

4 core = 4 load avg

With hyperthreading, you'd hope for

4 core hyper-threaded = 8 load avg

But things can be more complicated, since they aren't real cores: Linux Load Averages and HyperThreads

What goes into figuring out load averages

https://superuser.com/questions/23498/what-does-load-average-mean-in-unix-linux

Load average is the average of the load number for a given period of time. It takes into account processes that are:

Actively running on a CPU.

Considered runnable, but waiting for a CPU to become available.

Sleeping: i.e., waiting for some kind of resource (typically, I/O) to become available.

Solution 2

The load average represents the number of processes READY to have CPU dispatched to it (i.e. not blocked for I/O or other things). Thus on a failing system you might see values in the 100s. So this value is independent on the number of cores but I guess theoritically this would be limited to the total number of processes the system can support.

Solution 3

CPU utilization would be an easier metric to set thresholds on than load average. Many monitoring systems source this from inputs like user + system time in vmstat

Your workload will vary, but high 90% tends to degrade user interactive response times.

Comments

[user4556274]

You can reach much higher than 6. load average (as in the uptime or top output) refers to the scheduling queue, not CPU utilization. blog.scoutapp.com/articles/2009/07/31/…

[shrish]

thank you for your input :) does this mean, we can't really depend on load average while monitoring system ? I want to get triggered email ( done by monitoring software) when the system reaches a level where it possibly cannot handle requests or cpu usage monitoring will be better in this case?

[Ryan Babchishin]

@shrish I would perhaps try monitoring at 1xcore, so 6 for you. And see how it goes. A load higher, unless normal for that system, could be worthwhile checking out at the very least. It's nice to always run your servers below max capacity anyways. But a system can often handle way more and remain stable. If the load is high because it ran out of memory and is swapping, bad. If the load is high because the CPU is busy, maybe OK temporarily. It depends.

[shrish]

I have noted your points :) thanks for your expert view on this. really appreciate your help :)

[EEAA]

There is no "normal" when it comes to Load Average.

[shrish]

This is kind of answer I was looking for :) thanks for the input.