SSH server not working (respawns until stopped)

Question

I have a running Ubuntu Server 10.04.1. When I tried to login to the server via ssh, I could not. Instead, I got connection refused error. I tried to ping the machine and I got reply! So, the clear reason is that SSH daemon is stopped.

After reboot, I was able to login to my server via ssh. After some time, I looked at my logs /var/log/syslog and found the following records:

Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2465) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2469) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2473) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2477) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2481) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2485) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2489) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2493) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2497) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2501) terminated with status 255
Jan 16 10:57:09 myserver init: ssh respawning too fast, stopped

I searched for a similar problem/solution. Some people said that this is caused by the SSH daemon trying to start before networking and they suggest to change ListenAddress in /etc/ssh/sshd_config to be 0.0.0.0. I think this is not the cause in my case, because my problem occurs after system is up and running.

Any idea what is causing this? This is Ubuntu Server and it should be running and accessed remotely using SSH.

UPDATE:

Here is the log snippet I found in /var/log/auth.log.

Jan 16 10:56:38 myserver sudo:     user : TTY=pts/0 ; PWD=/home/user ; USER=root ; COMMAND=/usr/bin/vim /etc/ssh/sshd_config
Jan 16 10:57:09 myserver sudo:     user : TTY=pts/0 ; PWD=/home/user ; USER=root ; COMMAND=/etc/init.d/ssh reload
Jan 16 10:57:09 myserver sshd[1465]: Received SIGHUP; restarting.
Jan 16 10:57:09 myserver sshd[2461]: Server listening on 0.0.0.0 port 22.
Jan 16 10:57:09 myserver sshd[2465]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2465]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2469]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2469]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2473]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2473]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2477]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2477]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2481]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2481]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2485]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2485]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2489]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2489]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2493]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2493]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2497]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2497]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2501]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2501]: fatal: Cannot bind any address.

It seems that this error started to appear after I reloaded the SSH daemon. Should I avoid using ssh reload and use ssh restart instead?

Solution 1

You should check to see what happened just before SSH started floundering in syslog. If the networking subsystem died, that could explain why sshd started failing.

I would also check /var/log/auth.log. It's sshd's log and it might give you a better error message.

Solution 2

I just had the same problem on my 12.04 box. I.e. same symptoms. Alas, it always happened when I introduced the ListenAddress clause with the inet and inet6 addresses in sshd_config. In short, this appears to be a symptom of a malformed sshd_config - although the log files didn't state anything like that.

Troubleshooting sshd

What I find generally very useful in any such cases is to start sshd without letting it daemonize. The problem in my case was that neither syslog nor auth.log showed anything meaningful.

When I started it from the terminal I got:

# $(which sshd) -Ddp 10222
/etc/ssh/sshd_config line 8: address family must be specified before ListenAddress.

Much better! This error message allowed me to see what's wrong and fix it. Neither of the log files contained this output.

NB: at least on Ubuntu the $(which sshd) is the best method to satisfy sshd requirement of an absolute path. Otherwise you'll get the following error: sshd re-exec requires execution with an absolute path. The -p 10222 makes sshd listen on that alternative port, overriding the configuration file - this is so that it doesn't clash with potentially running sshd instances. Make sure to choose a free port here.

This method has helped me many many times in finding issues, be it authentication issues or other types. To get really verbose output to stdout, use $(which sshd) -Ddddp 10222 (note the added dd to increase verbosity). For more debugging goodness check man sshd.

---

The main advantage of this method is that it allows you to check the sshd configuration without having to restart the sshd on the default port. Normally this should not interfere with existing SSH-connections, but I've seen it. So this allows one to validate the configuration file prior to - potentially - cutting off ones access to a remote server (for example I have that for some VPS and even for physical servers where I need to pay extra to get out-of-band access to the machine).

Solution 3

This appears to be the result of bug #687535, which was fixed recently in natty, and has been uploaded to both maverick and lucid as a proposed update.

https://bugs.launchpad.net/ubuntu/lucid/+source/openssh/+bug/687535

I'd encourage everyone to go there, try the test case (search for TEST CASE), and post your results both before and after installing the proposed fix. That will help the SRU team decide that verification has been done and release it as an update.

Solution 4

In /etc/ssh/sshd_config, make sure that all yes and no options are in lower case. For example, if you set, PermitRootLogin No, ssh will not start. It actually has to be PermitRootLogin no.

Solution 5

I had a similar issue with an Ubuntu 11.10 image on a Linode after restarting. ssh service would produce in syslog:

Mar 18 06:31:33 servername kernel: init: ssh main process ended, respawning
Mar 18 06:31:33 servername kernel: init: ssh main process (3419) terminated with status 255
Mar 18 06:31:33 servername kernel: init: ssh main process ended, respawning
Mar 18 06:31:33 servername kernel: init: ssh main process (3422) terminated with status 255
Mar 18 06:31:33 servername kernel: init: ssh respawning too fast, stopped

It's a test box, and it had about 60 days of uptime, so somewhere along the way, I installed something that appended to the bottom of sshd_config:

ClientAliveInterval 60
ClientCountAliveMax 60

Commenting those lines allowed ssh to start.

View more solutions

Comments

[Khaled]

Thanks! I found many entries in the auth.log file and I updated my question.

[Oli]

reload should be a valid action. It should trigger an internal restart (and it appears to have attempted that and just got stuck). Try reloading again and see if it gets stuck again.

[SpamapS]

indeed, reload should be valid, but there is a bug. See my answer for more info.

[Peter Laird]

Your direct invocation trick just saved my bacon. I had an error in my sshd_config file (generated from Chef) that I was able to solve using this technique. THANKS for taking the time to post it to everyone.

[David Foerster]

While this is a helpful guide, it's obviously not what led to OP's sshd not running correctly, as you can see from the very different error messages in their logs. -1