Ubuntu 16.04 LTS sshd restart problem

Question

I've choose Ubuntu 16.04 LTS as main OS on my production servers. It has the latest version of Linux kernel, frequent updates and seems to be stable.

However 2 days I can not resolve the problem with sshd service. In general it works good. The problems appears after system reboot. I'm able to login via SSH, but systemd says that sshd.service is not runnning / failed to start. Also I can not restart sshd service and I see the same error.

● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2017-03-11 12:17:42 UTC; 37s ago
  Process: 3535 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255)
 Main PID: 3535 (code=exited, status=255)

Mar 11 12:17:42 node2 systemd[1]: Starting OpenBSD Secure Shell server...
Mar 11 12:17:42 node2 sshd[3535]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Mar 11 12:17:42 node2 sshd[3535]: fatal: Cannot bind any address.
Mar 11 12:17:42 node2 systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a
Mar 11 12:17:42 node2 systemd[1]: Failed to start OpenBSD Secure Shell server.
Mar 11 12:17:42 node2 systemd[1]: ssh.service: Unit entered failed state.
Mar 11 12:17:42 node2 systemd[1]: ssh.service: Failed with result 'exit-code'.

I've reinstalled Ubuntu 16.04 LTS on server many times and tried to change sshd_config file as well to fix this. However it does not help. Here is my sshd_config:

# Package generated configuration file
# See the sshd_config(5) manpage for details

# What ports, IPs and protocols we listen for
Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
ListenAddress 0.0.0.0
Protocol 2

# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
#Privilege Separation is turned on for security
UsePrivilegeSeparation yes

# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 1024

# Logging
SyslogFacility AUTH
LogLevel INFO

# Authentication:
LoginGraceTime 120
PermitRootLogin no
StrictModes yes

RSAAuthentication yes
PubkeyAuthentication yes
#AuthorizedKeysFile     %h/.ssh/authorized_keys

# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /etc/ssh_known_hosts
RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes

# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Change to no to disable tunnelled clear text passwords
PasswordAuthentication yes

# Kerberos options
#KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no

#MaxStartups 10:30:60
#Banner /etc/issue.net

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

Subsystem sftp /usr/lib/openssh/sftp-server

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes
UseDNS no
GSSAPIAuthentication no

I get a lot of messages from my monitoring system about this issue. I just want to ensure that everything works properly and I will not have problems with my server in the future.

Can anybody help with this?

Thanks

Solution

Mar 11 12:17:42 node2 sshd[3535]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Mar 11 12:17:42 node2 sshd[3535]: fatal: Cannot bind any address.
Mar 11 12:17:42 node2 systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a

The sshd process is running, you can connect to the server, but systemd does not know about that (he does not know what process to kill before starting a new sshd process). Assuming this happened by starting sshd by hand or systemd lost a trace of the sshd process somehow, you should be able to resolve it with the following steps:

• Locate the main sshd process and kill it. Do not kill the children (also called sshd) otherwise you will cut yourself from the server!

  $ ps axf | grep sshd
   2208 ?        Ss     0:00 /usr/sbin/sshd
  18812 ?        Ss     0:00  \_ sshd: pi [priv]  
  18891 ?        S      0:01      \_ sshd: pi@pts/0   
  19123 pts/0    S+     0:00              \_ grep --color=auto sshd
  # kill -9 2208
  

• Start the sshd service:

  systemctl start sshd
  

Comments

[mobidev911]

Thank you for the detailed response. It seems that I've found the solution... You are right that systemd lost a trace of the sshd process. In my case it seems that there was a conflict between systemd and upstart. So I've removed /etc/init/ssh.conf file and it seems that everything started to work.

[mobidev911]

omg... the same bug again.