Firewall has begun dropping SSH response

Question

My firewall (Sonicwall NSA 2400) has begun dropping the SSH response from my remote linux server. When I initiate an SSH session from my internal IP it times out, and when I look at the firewall log, I see the following:

TCP connection dropped
source (remote Linux server IP), port 1200, interface X1
destination: (my internal IP), port 65050, interface X0
notes: TCP Port: 65050

Subsequent attempts fail, always with a different internal destination port. What am I missing here? I haven't added any rules to the Sonicwall's firewall, but the Linux server's SSHD (Ubuntu 10.10) has recently been updated.

Update:

A packet dump reveals that the packet's being dropped due to an invalid TCP flag.

IP Type: TCP(0x6), Src=[source ip], Dst=[10.10.10.11]
TCP Packet Header
TCP Flags = [RST,], Src=[1200], Dst=[51150], Checksum=0x76d7
Application Header
Not Known
Value:[0]
DROPPED, Drop Code: 22, Module Id: 26, (Ref.Id: _3694_uyHtJcpfngKrRmv) 1:1)

Hex:

0017c519 be3d0090 1a422c89 08004500 002c0000 00001606 *.....=...B,...E..,......*
82a34e10 18c10a0a 0a0b04b0 c7ce6c45 9d910000 00005004 *..N...........lE......P.*
000076d7 00007366 7463                                *..v...sftc              *

Does this mean that the Sonicwall's just being unusually fussy, or that something's misconfigured on the remote server? I can connect to it from a different remote LAN without any problem.

Solution 1

Ahh, the joys of figuring out why the bloody Sonicwall drops packets. Welcome to owning an NSA appliance.

First off, point at that drop message in the Sonicwall's admin page. You should see a pop up message, and included in that will be a 'message ID'. You can cross reference that value with the Drop Codes listed here (for firmware version 5.6.0.x). If you're using an older firmware, here's version 5.5.0.x or version 4.2.0.x. The codes are pretty vague, but it might help you get on the right path.

Alternately, you can do a packet dump, and the drop code and module ID will be listed in the packet capture interface when you look at the dropped packet.


--Christopher Karel

Solution 2

Is a problem in how Linux manage the MTU negotiation (tcp_mtu_probing). Most linux boxes comes with a mode where Linux doesn't negociate in an alternative way to the standard (which by other side is insecure, and thats why Sonicwall doesn't accept it). You can fix in your linux box by doing this:

echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing

So the problem is Sonicwall refuses to negociate MTU size by using the "first" and most standard method because is insecure, and Linux refuses to negotiate MTU size by using the "alternative" method, which is more secure but less efficient.

Result, you're fucked. This only happens in situations where the firewall cannot fragment by itself the package due a cyphered payload which needs to be recomposed in the source, not in transit. That's why seems to be a lot of issues with SSL, SSH and other crypted connections.

With OpenSSH you can "hack" the way it negotiate the need of resizing the payload, by touching some options in the openSSH client options, but you may found the problem by using other application, so it's better to fix the problem by altering how the kernel manage this situations, if not, you can fix SSH but later found problems with SSL or OpenVPN.

More information, from Sonicwall: https://support.software.dell.com/kb/sw3798 The SonicWALL does not honor or pass to the LAN MTU Path Discovery messages because there is no way to authenticate them and they can be used as a form of attack. For example, someone can initiate a denial of service attack against an unprotected server by simply sending a Path MTU Discovery packet to it that directs the server to limit packet size to 5 bytes instead of the normal 1500 bytes. This will slow the server down to a crawl because it has to process the same amount of data into many more packets, creating much overhead.

Comments

[Zaid Amir]

1200 is not SSH's port. If you changed, I really recommend you to move it back to its default, and avoid changing it without understanding the consequences. Also, 65050 is one of the last ephemeral ports available, it is very uncommon to see it in examples.

[urschrei]

Thanks! Have updated my answer with additional information.

[Christopher Karel]

Hmm...odd. Looks like the dropped packet has the RST flag set. That's an attempt by your server to kill the connection. That's probably the flag the Sonicwall is griping about. Is it possible the authentication has broken, or something else has gone awry due to the SSH upgrade? Maybe try some more troubleshooting to narrow the problem. Does it happen with certificates or passwords? Different client machines? Same machine, but connected to a different network?

[urschrei]

It's key-only authentication, and it works fine for the same user on a different machine on a different remote network, and it's fine for this machine when it's on the same LAN as the server.

[Christopher Karel]

Hmm...then it does look like this is largely due to the firewall. Assuming you don't have a iptables, or some other firewall, on the target, that might be blocking attempts from just that network? That would certainly explain the RST, and might have been caused by a package update/install at the same time that you touched SSH.

[urschrei]

I'll have a look around on the Ubuntu box, but I'm 99% sure that it's nothing there. I suppose I could run a packet dump on it while attempting to connect from the offending remote network, which will at least definitely isolate the problem to either the Sonicwall or the Ubuntu server.

[pspahn]

This question does not really apply to my situation, but this answer provides plenty of insight I hadn't considered.