SSH tunnel to router periodically hangs. TCP problem?

Question

Something weird I've noticed since upgrading my office router to the Buffalo WZR-HP-G300NH. When I remote in using an SSH tunnel, the tunnel will "freeze" pretty often. The freeze lasts 1-2 minutes each, after which all the traffic that was stuck catches up. It's very annoying to be using VNC and have the session pause every few minutes.

I am using the following command from my local machine:

ssh -D 9000 [email protected] 

I know the whole router isn't freezing because I can open another SSH session and it's fine -- until it freezes, too. The freezing sessions are independent of each other. Both will periodically freeze, but not at the same time.

I can start a ping going in both sessions and neither loses a single packet or even shows any lag, even though the both freeze a couple times.

If I forward the VNC port straight through to the remote machine, the problem gets somewhat better leading me to believe this is more likely a TCP issue rather than SSH. But I'm not ruling anything out.

This behavior has persisted despite upgrading firmwares to the latest dd-wrt, including the build that was pulled a few weeks ago.

Is it a problem with dropbear? Or with MTU? QoS?

Solution

Two guesses:

1. Packet loss. Do not trust the "ping" you ran in parallel since the packet loss may be affecting only those TCP streams, for whatever reason. One simple way to detect it is to run $ netstat -s -p|grep "segments retransmited" frequently on both sides (ssh client and server) during the SSH session. See if the counter increases, in which case you experienced packet loss on TCP sessions in this machine. That said, the best way is to use tcpdump or wireshark to record what is happening and confirm there are re-transmissions occurring (they are usually marked with red by wireshark, but YMMV).

2. MTU problems. If you're trying to list the content of a large directory via SSH, for example, it may occur that the amount of data being transmitted would require it to be fragmented. The endpoints will use path MTU detection to determine the amount of data to send at a time. But in some cases, a firewall in the path may be blocking all ICMP packets and therefore breaking the path MTU detection, causing apparent freezes. This can be hard to confirm and diagnose. The easiest workaround that always works is to change the MTU of the network interface in either side to a low value, such as 1200 or 1000. This may hurt your performance so you'd better not use this forever.