SSH session through OpenVPN cuts off / locks up after a few lines

Question

I have a large number of identical fanless PCs running debian 6 (ARM). Most of these are connected via comcast and work ok. There are some that are connected to 'WiMax' modems and are having communication issues.

Specifically: if I ssh to one of these and try a command like 'ps -ax' I'll get about 3 lines back and then the session is locked up. If I let it sit, eventually it will close with a 'session closed by peer'.

What I've tried:

• ssh -vvv → no error messages
• ssh <user@host> 'command' → this will sometimes return the full output of the command. Sometimes it won't connect at all.

Suggestions on other things to try?

I've found that I can execute some commands successfully: e.g. hitting return a dozen times or more is ok. cd ~ and then lf works as does df -h. I can run df many times successfully but as soon as I try something with more output (e.g. ls /etc) it locks up.

Does it make a difference that I'm trying to communicate between these two hosts using OpenVPN?

Solution 1

You have the symptoms of an MTU problem: some TCP connections freeze, more or less reproducibly for a given command or URL but with no easily discernible overall pattern. A telltale symptom is that interactive ssh sessions work well as long as you don't run commands with large output. See Can't access select https sites on Linux over PPPoE for an explanation.

OpenVPN has several MTU-related option — search for “mtu” in the manual. I don't have enough experience to be confident as to which option you need to change. (It's even possible that you can change something in the Wimax modem configuration.) The most likely option to change is mssfix: try lowering the value until it fixes the problem. The default is 1450; something like around 1400 might fix your problem. Try openvpn --fragment 1200 -mssfix; if it helps, increase the value until it starts breaking.

Solution 2

Gilles' answer is completely correct, but there is also another potential cause for this.

There was a bug in version 2.3.0 of OpenVPN which would disconnect clients when sending large chunks of data: https://community.openvpn.net/openvpn/ticket/263

This issue only occurred when using TCP. UDP was completely unaffected.

Solution 3

We had the same problem, and it was indeed a MTU issue. However for us the problem was not in openVPN configuration but on the tun0 interface.

How we solved it: first find the maximum packet size that went through, with

ping <host> -s 1500 -M do

and reducing the 1500 value until a value was going through (for us, 1350).

Once the right value was found, change the tun0 interface with

sudo ip link set dev tun0 mtu 1350

as was proposed by Sebastian here. After that the VPN was going smoothly.

Comments

[Aaron Copley]

Make sure your path MTU isn't lower than your configured interface MTU. SSH does not fragment so, with DF bit set, packets are dropped instead. Read this for a much more detailed explanation.

[ethrbunny]

Tried setting MTU on all interfaces to 1400 with no apparent effect. Tested with ping -c 1 -s $((5000-28)) -M do machine-ip which returned 1500 - same as machine

[Aaron Copley]

Well... That was worth a shot.

[Marek Zakrzewski]

@AaronCopley ssh has a -T option. man ssh

[Marek Zakrzewski]

The comment was meant for ethrbunny.

[Aaron Copley]

Forgive my ignorance, but how does -T help in this instance?

[Gilles 'SO- stop being evil']

<Reads second paragraph> It's an MTU problem. <Reads further> Yep, MTU problem. See this thread for an explanation. I'm not voting to close as a duplicate because there one point that the other thread doesn't discuss: what you need to change in your VPN configuration to fix the problem.

[ethrbunny]

Am starting by setting mssfix 1200 on the server and restarting. So far so good. If this works I'll up it to 1300 or 1400 and see what happens. Too many clients to change all the remote configs though so server side will have to do.

[Aaron Copley]

I knew it was MTU!

[Admin]

Gilles gave a good answer (as always). Just to update it and make things easier - OpenVPN has an mtu-test option. From the manual: To empirically measure MTU on connection startup, add the --mtu-test option to your configuration. OpenVPN will send ping packets of various sizes to the remote peer and measure the largest packets which were successfully received. The --mtu-test process normally takes about 3 minutes to complete.