memory&swap are full, can't ssh; any option other than physical restart?

Question

by mistake, i executed some applications that used all memory (and i think swap) on my ubuntu server and its now crashed, SSH doesn't work and freezes. Do you know any other options other than following solutions:

1. physically restart the server.
2. wait until a process ends.

is there any way to remotely restart the server when ssh not working? i can still ping the server, so wondering if any reserved memory is there for killing unfriendly processes or for basic commands such as restarting the Os.

*The commands executed with "nohup" so they didn't end by closing ssh sessions.

Solution 1

No, your system is... pretty much done. Your only hopes are waiting for one to end, or having one get so large that the OOM killer sweeps in and does something about it.

Solution 2

It sounds like you system is doing what is called thrashing. What that means is your system is swapping pages in and out of memory for in use systems. That includes SSH.

You actually have a couple of options. The first option is of course to reboot the server, but it sounds like that is not much of an option for you because it's not local to you. If it's at a datacenter then you may be able to get them to walk up to it and reboot it. You have to check with your DataCenter.

Second option is also pretty simple. Your SSH session just hangs waiting for some stuff to be swapped out so it can allocate a tty for you. I have had this happen to me before, and you just have to wait for it to respond, it can take quite a while. Once you have a cli then you can kill or reboot the server.

The reason you can PING the server is actually pretty simple, the kernel's IP stack (IIRC) does not get swapped out that is why ICMP is able to respond.

A third option would require some setup before hand so I dont think it's a real option for you right now. You can setup a modem to allow you to dial in to a number and get a serial console. On the same note you can use IPMI Serial over LAN (SOL) to get a serial console. Also IPMI can allow you to reboot the server. Once again it sounds like those are not setup for you.

Comments

[cco]

I didnt say it was a reliable option. Also a couple of changes in your ssh client config can deal with that as well.

[John Gardeniers]

Changes on the client won't alter how long the server waits for a password and I think that's the real stumbling block.

[Ehsan]

thank you, so the only practical option is physical restart, otherwise i might come up with 100* years waiting...