What does it mean if SSH hangs after `Connection Established`?

Question

I use a Google Cloud VM and every once in a while I switch back to my terminal and see my ssh session has frozen. When I then try to reconnect

ssh -v  -i ~/.ssh/key  [email protected]

It shows this:

OpenSSH_7.4p1, LibreSSL 2.5.0
debug1: Reading configuration data /Users/UserName/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 53: Applying options for *
debug1: Connecting to host.domain [123.456.123.456] port 22.
debug1: Connection established.
debug1: identity file /Users/UserName/.ssh/ke> type 1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/UserName/.ssh/key-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4

So it appears a connection is established, but nothing else happens and I need to restart the VM. What does this mean?

I should note that I can ping the host with no problem, so it's not frozen or anything.

Solution

debug1: Local version string SSH-2.0-OpenSSH_7.4

When a client connects to an SSH server, the server starts the SSH protocol by sending a server version string in plain text to the client. With the OpenSSH ssh utility, the relevant debug lines look like this:

debug1: Local version string SSH-2.0-OpenSSH_7.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6

After the "local version" line, your client is waiting for the server to send its version string to the client. If the connection hangs here, it's because the client hasn't received the version string from the server.

In general there are several things that could cause this:

1. The client has connected to something that's not an SSH server. For example, an HTTP server wouldn't send anything to the client, because the HTTP protocol expects the client to send the first data.
2. The server is malfunctioning. For example the server might be overloaded, and the SSH server process isn't getting a chance to run.
3. The server is hung somehow. It might be stuck doing a DNS query on the client's IP address, for example.
4. Some network device is interfering with the TCP connection.

In your case, you're connecting to port 22 so it's safe to assume you're connecting to an SSH server process. It seems likely you're suffering from #2 (the server is malfunctioning), but it's not possible to say exactly what is wrong beyond that. You would need to get into the server and figure out what was happening at the time which prevented it from processing SSH connections.

Comments

[hyph]

Try increasing the verbose levels, i.e. -vv or even -vvv maybe that could give us more details to help.

[oarfish]

I used vvv as well, but didn't get any more output. I let the server be for a few hours, and now I can connect again. Will see if I can get more info next time it happens.

[oarfish]

Are there log files I should be looking at when I get access back?

[noorul]

@oarfish What was the issue? and which case was yours?

[oarfish]

I can't say I ever got to the bottom of this. Had to reboot the machine to get SSH to work again, so probably one of 2.-4. was the reason.