How to prioritize SSH to work in the event of high CPU load

Question

I'm encountering a weird problem where a Fedora Linux VPS server reports 100% CPU, and effectively becomes unusable, but I don't know why because the high load prevents me from SSHing into it to see what's wrong.

How do I prioritize or configure SSH so that I'm still able to connect even if some process is consuming all other CPU?

Solution 1

If this is for real a CPU / Load related problem you can (re)nice the sshd process on your server to give it a higher priority. To make that persistent you can add the nice inside the sshd init script.

Solution 2

Create a script which logs processes:

#!/bin/bash
top -b -c -n 1 -d 1 > /tmp/top-$(date "+%Y-%m-%d_%H-%M-%S")

Setup a cron job which runs the script every minute:

*/1 * * * * /path/to/script.sh

Next time your system goes unresponsive, you'll have logging of what was doing it.

Solution 3

I was running the code that was hogging all the resources through Docker, so I just limited the amount of CPU that could be consumed by the container by starting it with

--cpus=7.5

Because I have 8 cores, this means I should always have half a core for processing SSH sessions into the actual server running the container (unless some other process takes up those resources too).

For good measure I also limited the container's memory usage with

--memory=4g

https://docs.docker.com/config/containers/resource_constraints/#cpu

Solution 4

If you don't mind my saying, I think you're asking the wrong question. You really want to find out what is using so may resources to the point of your not being able to log in. I have come across this problem numerous times and the solution is to track what is going on the server constantly. I came up with this solution which is as light as possible, reducing its chances of being stopped by a resource hog:

http://linuxtech.ie/wordpress/2012/09/05/finding-a-severe-resource-hog-on-your-server/

I hope that helps

View more solutions

Comments

[Robbie Mckennie]

I don't think this is really possible, why not attack the problem at it's source? Work out what's causing the cpu to max out and fix THAT problem.

[Oupkar Sandhu]

Maybe because he can't SSH in to attack the problem. Chicken meets egg. :)

[Cerin]

@RobbieMckennie, And how do you propose I do that?

[demure]

It sounds like it is a little late to force ssh to have priority. If you have 'physical' access, you might be able to get in. You might be about to run ssh user@host sudo reboot to try to make it restart, with less overhead then getting a full shell.

[Cerin]

@demure, I don't have physical access. I can reboot it, but that destroys any evidence as to what the problem was.

[Robbie Mckennie]

@Cerin top -d 15 -b will output a list of running processes, along with their respective cpu usage, every 15 seconds. Something like top -d 15 -b > log & will run in the background and append the list to the file "log" every 15 seconds.

[ewwhite]

This is true. Treating the symptom, not the cause. I've NEVER had to make changes to the priority of the SSH daemon on a production system. I think this lies more with the deficiencies of the VPS provider or another resource issue.

[suprjami]

For additional points, write another cron job which clears out old logging every few days so you don't fill up your filesystem. (hint: find and -mtime)