ssh refusing all connections after upgrade to 11.10

Question

I upgraded from 11.04 to 11.10 and upon completion found that I could no longer ssh into other computers that I routinely do so. There are several things I've checked:

1. Kerberos authentication is working fine, that's not the problem.
2. I tried restarting and reinstalling ssh, but neither helped.
3. I tried copying over all ssh related files from my laptop (with a properly function ssh in 11.04) and replace what is on my 11.10 malfunctioning OS, but that did not help.

4. I tried deleting the .ssh/known_hosts file. On my next attempt, I received the normal message about connecting somewhere for the first time, but was still refused a connection.

   jason:~$ /usr/sbin/sshd -ddd
   debug2: load_server_config: filename /etc/ssh/sshd_config 
   debug2: load_server_config: done config len = 682 
   debug2: parse_server_config: config /etc/ssh/sshd_config len 682 
   debug3: /etc/ssh/sshd_config:5 setting Port 22 
   debug3: /etc/ssh/sshd_config:9 setting Protocol 2 
   debug3: /etc/ssh/sshd_config:11 setting HostKey /etc/ssh/ssh_host_rsa_key 
   debug3: /etc/ssh/sshd_config:12 setting HostKey /etc/ssh/ssh_host_dsa_key 
   debug3: /etc/ssh/sshd_config:13 setting HostKey /etc/ssh/ssh_host_ecdsa_key 
   debug3: /etc/ssh/sshd_config:15 setting UsePrivilegeSeparation yes 
   debug3: /etc/ssh/sshd_config:18 setting KeyRegenerationInterval 3600 
   debug3: /etc/ssh/sshd_config:19 setting ServerKeyBits 768 
   debug3: /etc/ssh/sshd_config:22 setting SyslogFacility AUTH 
   debug3: /etc/ssh/sshd_config:23 setting LogLevel INFO 
   debug3: /etc/ssh/sshd_config:26 setting LoginGraceTime 120 
   debug3: /etc/ssh/sshd_config:27 setting PermitRootLogin no 
   debug3: /etc/ssh/sshd_config:28 setting StrictModes yes 
   debug3: /etc/ssh/sshd_config:30 setting RSAAuthentication yes 
   debug3: /etc/ssh/sshd_config:31 setting PubkeyAuthentication yes 
   debug3: /etc/ssh/sshd_config:35 setting IgnoreRhosts yes 
   debug3: /etc/ssh/sshd_config:37 setting RhostsRSAAuthentication no 
   debug3: /etc/ssh/sshd_config:39 setting HostbasedAuthentication no 
   debug3: /etc/ssh/sshd_config:44 setting PermitEmptyPasswords no 
   debug3: /etc/ssh/sshd_config:48 setting ChallengeResponseAuthentication no 
   debug3: /etc/ssh/sshd_config:63 setting X11Forwarding yes 
   debug3: /etc/ssh/sshd_config:64 setting X11DisplayOffset 10 
   debug3: /etc/ssh/sshd_config:65 setting PrintMotd no 
   debug3: /etc/ssh/sshd_config:66 setting PrintLastLog yes 
   debug3: /etc/ssh/sshd_config:67 setting TCPKeepAlive yes 
   debug3: /etc/ssh/sshd_config:74 setting AcceptEnv LANG LC_* 
   debug3: /etc/ssh/sshd_config:76 setting Subsystem sftp /usr/lib/openssh/sftp-server 
   debug3: /etc/ssh/sshd_config:87 setting UsePAM yes 
   debug1: sshd version OpenSSH_5.8p1 Debian-7ubuntu1 
   debug3: Incorrect RSA1 identifier 
   debug1: read PEM private key done: type RSA 
   debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048 
   debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048 
   debug1: private host key: #0 type 1 RSA 
   debug3: Incorrect RSA1 identifier 
   debug1: read PEM private key done: type DSA 
   debug1: Checking blacklist file /usr/share/ssh/blacklist.DSA-1024 
   debug1: Checking blacklist file /etc/ssh/blacklist.DSA-1024 
   debug1: private host key: #1 type 2 DSA 
   debug3: Incorrect RSA1 identifier 
   debug1: read PEM private key done: type ECDSA 
   debug1: Checking blacklist file /usr/share/ssh/blacklist.ECDSA-256 
   debug1: Checking blacklist file /etc/ssh/blacklist.ECDSA-256 
   debug1: private host key: #2 type 3 ECDSA 
   debug1: setgroups() failed: Operation not permitted 
   debug1: rexec_argv[0]='/usr/sbin/sshd' 
   debug1: rexec_argv[1]='-ddd' 
   debug3: oom_adjust_setup 
   Set /proc/self/oom_score_adj from 0 to -1000 
   debug2: fd 3 setting O_NONBLOCK 
   debug1: Bind to port 22 on 0.0.0.0. 
   Bind to port 22 on 0.0.0.0 failed: Permission denied. 
   debug2: fd 3 setting O_NONBLOCK 
   debug3: sock_set_v6only: set socket 3 IPV6_V6ONLY 
   debug1: Bind to port 22 on ::. 
   Bind to port 22 on :: failed: Permission denied. 
   Cannot bind any address. 
   

Maybe the problem is in that readout, but I'm not familiar enough with this output to know.

My laptop which still has Ubuntu 11.04 still can successfully log into the computers I need to, so the problem is definitely related to the upgrade of my desktop to 11.10.

=========================================================================

[edit:] I think I figures something out:

If I do a ssh -vv [email protected] (the computer I'm trying to log into), towards the end of the output I get:

Jason Nett 11:06:38 PM
debug1: Authentications that can continue: gssapi-keyex,gssapi-with-mic,keyboard-interactive
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug1: Next authentication method: gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentication succeeded (gssapi-with-mic).
Authenticated to computer.org.com ([xxx.xxx.xxx.xxx]:xx).

Notice: "gssapi-with-mic" is in the list of "Authentications that can continue" and is the one that succeeded. When I try on the machine that lost it's ability to ssh, this output is:

debug1: Authentications that can continue: gssapi-keyex,gssapi-with-mic,keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug1: Authentications that can continue: gssapi-keyex,gssapi-with-mic,keyboard-interactive
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (gssapi-keyex,gssapi-with-mic,keyboard-interactive).

So on this machine, "gssapi-keyex" and "gssapi-with-mic" are never attempted--according to the verbose output--and only "keyboard-interactive" is attempted. From my online searches, I gather that gssapi-with-mic has something to do with communicating my kerberos authentication, but I'm not quite sure where to go from here, at the moment.

Hopefully this extra info can help us rectify the issue quickly.

Solution 1

Looks like it 'might' be something to do with ipv6 and ipv4 both trying to bind to port22. Try disabling ipv6 in sshd_config.

ListenAddress 0.0.0.0
#ListenAddress ::

Solution 2

I'm getting the same problem, but if I log in to the console or via ssh with password authentication at least once then all subsequent ssh authentications with public key work ok. It's not a good workaround because a reboot kills the ability to login with ssh/public key, but it might help someone diagnose what's going on.

Comments

[bgvaughan]

user15346 executed sshd as a regular user, which is why it didn't have permissions to bind to a port.

[user15346]

Thanks. I checked this file again and by default both of these lines are commented out. So the one you say to comment out was already.

[user15346]

There seems to be no permutation of commenting/uncommenting these two lines that solves--or changes--my problem.

[duffydack]

hmm ok. The only other error I see is RSA. You say you reinstalled ssh, did you remove the keys as well?

[user15346]

There was never a .ssh/authorized_keys file to begin with, if that's what you're referring to.

[duffydack]

No, I mean either your id_rsa keys in .ssh and or the keys in /etc/ssh. When you remove ssh, remove /etc/ssh/ as well then reinstall it.

[user15346]

Good try, but this did nothing either.

[osirisgothra]

fyi, you should be able to execute as regular user for testing purposes however you cant when not. Also, make sure your key files are all at 600 permission, or else ssh wont even start up at all, this is important because xinetd (or inetutils-inetd or whatever superserver u use) will shut down because it thinks no services are available (technically, it is right, but doesnt bother to tell you directly). Furthermore, if you test make sure to use the -f and point to the right configuration file, use the absolute path like: '/usr/sbin/sshd -d -f /etc/sshd_config' (to be sure config's being used)!