SSH hangs after authentication

Question

When logging in to one of my servers over ssh, it just hangs after authentication. This is the output on the client with -v.

OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to host1 [10.6.27.64] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/identity type -1
debug1: identity file /home/user/.ssh/id_rsa type 1
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: loaded 3 keys
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'host1' is known and matches the RSA host key.
debug1: Found key in /home/user/.ssh/known_hosts:172
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No credentials cache found

debug1: Unspecified GSS failure.  Minor code may provide more information
No credentials cache found

debug1: Unspecified GSS failure.  Minor code may provide more information
No credentials cache found

debug1: Next authentication method: publickey
debug1: Trying private key: /home/user/.ssh/identity
debug1: Offering public key: /home/user/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = C
debug1: Sending env LC_ALL = C
Last login: Wed May 21 10:24:14 2014 from host2
This machine has been configured with kickstart
host1 in bcinf17 in bay 3 in rack D10-Mid

And in /var/log/secure on the server I see this (lucky I still have a session open):

May 21 10:27:31 host1 sshd[12387]: Accepted publickey for user from 1.1.11.239 port 34135 ssh2
May 21 10:27:31 host1 sshd[12387]: pam_unix(sshd:session): session opened for user user by (uid=0)

So nothing obvious going wrong. The client and server seem able to communicate. Nothing in /var/log/messages.

Plenty of disk space. Some paths are mounted (including home areas), but my still active shell can access them OK.

I can connect to other servers; only this one has the problem. I have tried restarting sshd. The config file for sshd looks like the default, so nothing in there. As far as I know, nothing has changed recently.

Trying to run a command (ssh host1 -t bash, or -t vi) also seem to hang, so don't think it's anything to do with my login scripts.

Have also tried logging in from other hosts in the same location and other locations, or from Windows via Putty, and logging in using password instead of key.

Not sure where else to look or what else to try.

This is a RHEL 6.4 server, 64 bit.

Solution 1

There are several things that can cause a hang right after SSH authentication.

However most of these will also carry other symptoms (the hang right after an SSH-auth is just the most visible symptom)

1. As Iain mentioned, any user login scripts.
   • ~/.bashrc, ~/.bash_profile, ~/.profile, ~/.kshrc and so forth
2. Too many processes running / restarting.
   • Something has fork()'d too many child-processes and the load (the 1/5/15 score) is too high.
3. There is an I/O wait problem.
   • Typically caused by a dying hard-drive (common) or a badly behaving NIC (rare).
4. A 3rd-Party PAM module hanging (eg: a non-standard Kerberos config)
   • Not always the module itself, but sometimes a service (like auditing) that has a full log-server somewhere.

Solution 2

Yet another trouble source might be ssh clients waiting for ssh-agent (all of those configured to use it, of course). If ssh'ing anywhere gets stuck at

debug1: SSH2_MSG_NEWKEYS received

then check ps auxw | grep askpass and its dialog(s) which might have escaped your attention.

PS: this is rather not the culprit here but I haven't googled more relevant questions up so far.

Solution 3

Does it connect directly if you use ssh -o GSSAPIAuthentication=no user@host?

If so, the system may be hanging at some point deciding on a GSSAPI method. For me, only one host did this, so I just disabled GSSAPI in ~/.ssh/config for that host:

Host badHostName
    GSSAPIAuthentication no

I learned this from http://germanrumm.eu/fixing-ssh-login-delay-how-to-disable-gssapi-with-mic-on-ubuntu-linux/ but never quite learned the cause.

Solution 4

Ancient post, but in case it helps anyone else I tracked down why my SSH was hanging after login (Debian 10). It was this line I had enabled and set to no in my sshd_config file:

PermitTTY no

View more solutions

Comments

[retromuz]

This solved the issue I was having. Thanks!

[jpierson]

Not sure if this was the underlying problem for me but I did notice that I was seeing the following error running the above command in WSL. This led me to rely on git-bash for Windows which didn't have the same problem. ``` Signal 11 (SEGV) caught by ps (3.3.12). ps:ps/display.c:66: please report this bug ```