SSH broken pipe, message authentication code incorrect

Question

I have set up a Ubuntu server that I plan on using to backup my macbook air using rsync. But every time I use rsync, or even scp, the connection drops, with one of the following errors:

packet_write_wait: Connection to 192.168.1.202: Broken pipe
packet_write_poll: Connection to 192.168.1.202: Broken pipe
packet_write_poll: Connection to 192.168.1.202: Protocol wrong type for socket

Now I have searched for other questions like this, and the usually people have this problem with long backups where the session times out. For me this always happens within 10 seconds of starting the file transfer. I get the same errors using scp and rsync. I guess it could be due to a faulty network connection, but I find it hard to believe that my connection to the server on the same LAN is that unstable. Anyone have any ideas?

Examples of the commands I have used that result in the errors:

scp -r /Users/Matt/Documents [email protected]:/media/matt/MattsBackups/

/usr/local/bin/rsync -av -e ssh /Users/Matt/Documents [email protected]:/media/matt/MattsBackups/

I did some more testing today, and strangely enough it was working fairly reliably from outside my LAN. So I tried again from inside my home network, and it still doesn't work.

Running

grep 'sshd' /var/log/auth.log

on the server shows the following error

fatal: ssh_dispatch_run_fatal: Connection from <My IP> port 49870: message authentication code incorrect

Some more detailed information on my setup:

Macbook Air OS X 10.11.5
OpenSSH_6.9p1, LibreSSL 2.1.8
rsync  version 3.1.2  protocol version 31

Ubuntu Server
OpenSSH_7.2p2 Ubuntu-4ubuntu1, OpenSSL 1.0.2g-fips  1 Mar 2016

I did notice the difference in ssh versions but I was hoping it wouldn't be an issue. I can try and install a newer version with homebrew.

UPDATE:

OK I just updated ssh with homebrew to

OpenSSH_7.2p2, OpenSSL 1.0.2g  1 Mar 2016

which appears to be the same version used on the Ubuntu box. However, Rsync still results in the error when I run the command. Heres the command I tried with the ssh -v flag:

/usr/local/bin/rsync -a -e '/usr/local/bin/ssh -v -c aes128-ctr -m hmac-sha1' /Users/Matt/Documents [email protected]:/media/matt/MattsBackups/

The output is:

OpenSSH_7.2p2, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug1: Connecting to 192.168.1.202 [192.168.1.202] port 22.
debug1: Connection established.
debug1: identity file /Users/Matt/.ssh/id_rsa type 1 
debug1: key_load_public: No such file or directory
debug1: identity file /Users/Matt/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/Matt/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/Matt/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/Matt/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/Matt/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/Matt/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/Matt/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu1
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 192.168.1.202:22 as 'matt'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:+zkrXNJENs5EobFwHa8wpMDe6zPDfj975qLcPp4b4sg
debug1: Host '192.168.1.202' is known and matches the ECDSA host key.
debug1: Found key in /Users/Matt/.ssh/known_hosts:1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/Matt/.ssh/id_rsa
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug1: Authentication succeeded (publickey).
Authenticated to 192.168.1.202 ([192.168.1.202]:22).
debug1: channel 0: new [client-session]
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug1: Sending command: rsync --server -logDtpre.iLsfxC . /media/matt/MattsBackups/MacAir/
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Connection to 192.168.1.202 closed by remote host.
Transferred: sent 145304, received 13032 bytes, in 0.1 seconds
Bytes per second: sent 1373019.8, received 123143.2
debug1: Exit status -1
rsync: [sender] write error: Broken pipe (32)
rsync error: error in socket IO (code 10) at io.c(820) [sender=3.1.2]

Solution 1

Well as a last resort, I found an old 10/100 ethernet card from a windows 98 PC, and installed it in the server. After configuring it, I have had no more errors, over about 30 GB of data. I guess the built-in ethernet chipset didn't work well with ubuntu. Or I had somehow configured it incorrectly.

Edit: While I never found the root cause of my problem, be sure to check out the comment thread under @sourcejedi's answer. Big thanks to @sourcejedi, @sneep and @dentarg.

Solution 2

It could be a bug in your SSH. There have been several examples of this over time. (You should definitely post the exact versions used at either end).

http://www.alcatelunleashed.com/viewtopic.php?t=25294

I can't work out why a remote network path would be more reliable, or any suggestion to work on that. It can be caused by buggy network boxes though...

http://community.ubnt.com/t5/EdgeMAX/clone-from-gitlab-com-through-ER-ssh-dispatch-run-fatal-message/td-p/1448285

If your Ubuntu server is recently installed and has all available updates installed, I'd be more suspicious of the software on the Mac client, that it's old and affected by some bug like this.

You could test different MACs etc. E.g.

scp -o MACs=hmac-md5

Note hmac-md5 is not considered weak (in the context of ssh) in the same way md5 is (e.g. in the context of HTTPS certificates). I expect mainly it's slower than e.g. [email protected]. However it should be better to use -etm modes if you can.

The links suggest you can prefer older options

MACs hmac-md5,hmac-sha1,[email protected],hmac-ripemd160

and it will avoid certain bugs. Once you find something you're happy with, you can configure it in /etc/ssh_config.

The Cipher could also be the problem. If you're using aes-gcm e.g. [email protected], maybe there's not a separate MAC at all. So you at least need to make sure of what ssh is doing by adding -v to your commands and looking for the MAC it says it's actually using.

Comments

[Serge]

would you mind please to add the command line causing problems to your question?

[Klaatu von Schlacker]

try running scp with the -v switch included for more verbose output. it might give clues to what is going wrong.

[Matt]

Unfortunately scp -v doesn't give any more information when the error occurs. Rsync -v results in the following: rsync: [sender] write error: Broken pipe (32) rsync error: unexplained error (code 255) at io.c(820) [sender=3.1.2]

[Lee-Man]

Can you 'ssh' to the Ubuntu server?

[Matt]

I can ssh into the server with almost no problems. Only one time has the broken pipe error occurred doing that.

[Matt]

Thanks for all your help. I will try your suggestions tonight. I saw that second link earlier and tried disabling TCP offloading, but it didn't seem to help. I also wouldn't be surprised if the FiOS router is causing issues.

[Matt]

Ok I tested hmac-md5, hmac-sha1, [email protected], and hmac-ripemd160, as well as a few other listed in the openssh_config man page and I still got broken pipe errors. I tested them using scp -r -o MACs=<MAC> and rsync -a -e 'ssh -m <MAC>. I am going to try to see if another cipher has an effect.

[Bohumir Zamecnik]

Yeah, I had a similar problem with some USB 1Gbit network card.

[sneep]

Old thread, but just wanted to say that disabling offloading (ethtool -K enX rx off tx off) worked for me.

[user2948306]

@sneep Thanks. IMO your comment is related to the accepted answer. It would also be more noticeable there (because the accepted answer is more highly voted and also shorter). Looking again, the first link I posted feels like a somewhat different bug/fault, because the question here is about failure after transmitting bulk data for a few seconds, not an immediate failure to connect.

[SeanJohn]

Thanks! This answer + blog.e-shell.org/270 ("big load") helped me: I had troubles transferring large ZFS filesystems (~8TB across ~30 filesystems) over SSH from one FreeBSD server (hardware from 2009) to a newer one (hardware from 2016). auth.log on the receiving server reported Corrupted MAC on input. and message authentication code incorrect from sshd. From ssh -vvvv between the servers I could see that [email protected] was used. By using [email protected] my Broken pipe errors disappeared.