rsync keeps disconnecting: broken pipe

Question

I've looked high and low through the settings, but I can't seem to figure out how to prevent KDE Kate 3.13 from auto-collapsing branches in the Documents pane when opening files. This is really annoying to me, since I then have to re-dig in to the collapsed file structures to figure out where the previous file I was working on is located.

Does anyone know where the secret sauce is on this one?

To reproduce this issue:

1. Have a Kate installation at or around version 3.13
2. Place the Documents pane in Tree view via Right Click->View Mode->Tree Mode
3. Open up several documents in different branches via File->Open.
4. Open up a file in a different branch than the currently exposed branches in the tree view.
5. Observe that the other branches in the Documents pane auto-collapse.

Workaround While Waiting For Fix:

1. Enable Console Tool and open Konsole interface (or use a regular Konsole instance)
2. execute "kate " to open the file without triggering the document collapse.

Solution 1

Your problem might be (lack of) memory. Back when 1GB was big for a server, rsync would fail on me for large datasets. Perhaps the algorithm has improved of the memory capacities has increased, but I haven't seen that problem in 8 years or so. So really, this is an outside shot, but one worth exploring. Try smaller datasets first. You might also try -- as a form on sanity check -- doing a tar-tar:

tar cf - $HOME | ssh ${server} tar xf -

If that also fails after a few minutes, it's not memory.

Solution 2

I have encountered this with rsync in the past as well. The solution that fixed it for me was running it from within a screen session, which was able to help maintain the connection to the remote server.

screen -LS rsync
[execute your rsync command]
Ctrl-A+D to detach from the session

You can check the status by running screen -x rsync (or whatever you decide to name the session if you do give it a name, which is not required). This will re-attach your current shell to that session. Just remember to detach from it again after you have checked the status so that it keeps running in the background.

You can also execute the command to run via screen in the background in one fail swoop by doing [someone please correct me if I'm wrong] screen -dm 'command'. You may want to man screen before trying that last one.

EDIT:

I am editing my answer because you have confirmed that screen provides no assistance in this scenario, but you replied to my comment suggesting to try scp and see what kind of results you get, to which you replied that oddly enough, it worked just fine.

So my new answer is this: use scp -- or ssh (with tar) -- instead of rsync

Granted, scp doesn't support the vast number of features as rsync, but you'd actually be surprised to discover just how many features that it does support that are almost identical to that of rsync.

Real world scenarios for scp and other alternatives to rsync:

Awhile back, I was tasked with creating a shell script that would pull logs from our production servers and store them locally on a web server so that developers could access them for troubleshooting purposes. After trying unsuccessfully to get the Unix team to install rsync on our servers, I came up with a workaround using scp that worked just as well.

That being said, I recently modified the script so that all it uses is ssh and tar -- GNU tar/gtar, to be exact. GNU tar supports many of the options that you will actually find in rsync, such as --include, --exclude, permission/attribute preservation, compression, etc.

The way I now accomplish this is by ssh-ing to the remote server (via pubkey auth) and using gtar -czf - [other options such as --include='*.log' and --exclude='*core*', etc.] -- this writes all of the info to stdout, which is then piped [locally] to tar -xzf so that no changes are made on the remote production server, and all of the files pulled as-is to the local server. It's a great alternative to rsync in this case. The only thing important thing neither tar nor scp support are incremental backups and the level of block-level error checking that rsync features.

The full command I am referring to when using ssh and tar would be something like this (remote is Solaris 10; local is Debian, for what it's worth):

cd /var/www/remotelogs
ssh -C user@remotehost "cd /path/to/remote/app.directories; gtar -czf - --include='*.log' --exclude='*.pid' --exlude='*core*' *" | tar -xz

In your scenario it would be the opposite -- tar -cf - locally, and pipe to remote server via ssh user@remotehost "tar -xf -" -- there is another answer that references this type of behavior but doesn't go into as much detail.

There are a few other options that I have included to speed things up. I timed everything relentlessly to get the execution time as low as possible. You would think that using compression with tar would be pointless, but it actually speeds things up a bit, as does using the -C flag with ssh to enable ssh compression as well. I may update this post at a later date to include the exact command that I use (which is very similar to what I posted), but I don't feel like getting on VPN at the moment since I'm on vacation this week.

On Solaris 10, I also use -c blowfish, because it is the quickest cipher to authenticate with and also helps speed things up a tad, but our Solaris 11 either don't support it or have this cipher suite disabled.

Additionally, if you choose to go with the ssh/tar option, it would actually be a good idea to implement my original solution of using screen if you are doing a backup that will take awhile. If not, make sure your keepalive/timeout settings in your ssh_config are tweaked just right, or this method will also be very likely to cause a broken pipe.

Even if you go with scp, I always find it to be a best practice to use screen or tmux when doing an operation of this sort, just in case. Many times I don't follow my own advise and fail to do this, but it is indeed a good practice to use one of these tools to ensure that the remote job doesn't screw up because of your active shell session getting disconnected somehow.

I know you want to figure out the root cause of your rsync issue. However, if this is really important, these are two great workarounds that you can experiment with in the meantime.

Solution 3

I was having the same problem on OSX El Capitan and fixed this by upgrading to rsync v3.11. The issue was happening for me on v2.6.9.

Solution 4

Kerberos is only for authentication, that should not cause any problems after you have created a successful connection.

Have you tried using the rsync daemon too?

Are your servers on the same network or do you have a firewall/router between?

You could try setup a netcat session between the servers, that is a simple way to try if you have any connection problems between your servers.

On the first server:

nc -lk <port-number>

And on the client

nc <server> <port-number>

You could leave the connection open, and see if the connection keep it, or if you loose the connection. You can also try write something on the client, see that it ends up on the other side.

Solution 5

the only time I've had a problem like this with rsync, I tracked it down to a spare ethernet port on another machine that had the same IP address as my target server. If rsync is flaky, it is almost surely a network reliability or (in my case) configuration problem.

View more solutions

Comments

[marekful]

Can you explain the problem in more detail? "auto-collapsing branches in the Documents pane when opening files"?? I never experienced auto-collapsing directories in the document pane.

[emanaton]

Sorry @MarcellFülöp -- I've added steps to reproduce to the OP.

[David C. Rankin]

In earlier versions of kate, that was controlled by Settings->Configure Kate->Plugins->Symbol Viewer->Automatically expand nodes in tree mode Good luck with the kde4 version...

[emanaton]

Sadly, playing with the Symbol Viewer setting (or enabling/disabling it) has no effect on this issue. =o/

[pfnuesel]

Maybe I should add that I use kerberos to authenticate on the remote server.

[roaima]

That's potentially very important. Please edit your question to include this information

[Jeff Schaller]

seeing an io error makes me wonder if the remote side's filesystem filled up?

[pfnuesel]

@dhag rsync fails every time after a few minutes. So if the syncing takes less time than that, it wouldn't fail. I don't see any pattern in the time when it stops. Kerberos authentication cannot be the issue, since I can still logon the machine without reinvoking kerberos. If I'm connected to the machine, I never disconnect.

[pfnuesel]

@JeffSchaller The file system is not filled up, there is plenty of space left.

[rubynorails]

Maybe a stupid question, but I have to ask simply to rule out the obvious -- are both servers on a wired Ethernet connection, or is one of them using WiFi?

[rubynorails]

I know you are looking for an rsync answer, but have you tried scp? If so, did you encounter any difficulties using it instead of rsync? I know rsync is preferable in most cases such as yours, but using scp may help narrow down the issues you are encountering.

[pfnuesel]

@rubynorails Interesting. That seems to work without problems.

[rubynorails]

@pfnuesel see my updated answer. It may not provide a solution to the root cause of your issue, but it should be able to provide an adequate workaround and some troubleshooting wiggle room in the meantime if you need to take awhile to figure it out.

[tim.rohrer]

@pfnuesel, did you ever solve this issue?

[pfnuesel]

@tim.rohrer No, it still persists.

[emanaton]

Oh snap! Thank you, @dhaumann! I'd looked but not found a bug report, so assumed it must be user-error. Clearly, I need to work on my search-word guesses. Cheers!

[pfnuesel]

I don't understand. Neither what's going wrong, nor what I'm supposed to do to find out what's writing on stdout.

[rubynorails]

@pfnuesel - at least it's good to know that you can rule it out.

[pfnuesel]

Unfortunately, I don't have root access on the server. This means I cannot run an rsync daemon or a netcat session.

[roaima]

@pfnusel you can run netcat on any port >1024 without needing root privileges

[roaima]

@pfnuesel if you copy the transcript of you logging in and post it here, someone may see what's up. Better, post your .profile or .bash_profile for review. You're looking for things like mesg or stty

[pfnuesel]

There's no mesg or stty in any of my dotfiles.

[roaima]

@pfnuesel anything else that writes to the terminal during login?

[pfnuesel]

No, but even if I add something that writes to stdout. It doesn't change anything.

[roaima]

@pfnuesel f you had added something writing to stdout it would have definitely broken rsync over ssh. That it didn't break it any differently suggests to me that this could be a reason. Can you pastebin your .bash_profile or .profile? Or a typescript (script) of the login process to the remote server?

[pfnuesel]

Maybe I misunderstand. I added an echo statement to .bash_profile. It didn't change anything. rsync was running fine for a few minutes, before crashing. Now I deleted .bash_profile and .profile. Same thing, it crashes after a few minutes.

[pfnuesel]

I'm running rsync 3.1.1.

[B Brendler]

You may want to check your router doesn't have packet flooding protection (or any similar protection) enabled. Are you connecting through any sort of VPN?

[pfnuesel]

That might be the problem. Unfortunately, I don't have access to the network devices. It works fine on other servers, though, so I'm guessing that this particular server has some sort of packet flooding protection.

[Gabriel Staples]

Why the downvote? This should be a comment not an answer, maybe? Anyone? Anyone?

[pfnuesel]

I can't reproduce the problem anymore, since I don't have access to that server anymore. But it's a reasonable answer and doesn't deserve the downvote.