Keep Remote Directory Up-to-date
Solution 1
How "real-time" do you want the syncing? I would still lean toward rsync since you know it is going to be fully supported on both platforms (Windows, too, with cygwin) and you can run it via a cron job. I have a super-simple bash file that I run on my system (this does not remove old files):
#!/bin/sh
rsync -avrz --progress --exclude-from .rsync_exclude_remote . remote_login@remote_computer:remote_dir
# options
# -a archive
# -v verbose
# -r recursive
# -z compress
Your best bet is to set it up and try it out. The -n (--dry-run)
option is your friend!
Keep in mind that rsync (at least in cygwin) does not support unicode file names (as of 16 Aug 2008).
Solution 2
lsyncd seems to be the perfect solution. it combines inotify (kernel builtin function which watches for file changes in a directory trees) and rsync (cross platform file-syncing-tool).
lsyncd -rsyncssh /home remotehost.org backup-home/
Quote from github:
Lsyncd watches a local directory trees event monitor interface (inotify or fsevents). It aggregates and combines events for a few seconds and then spawns one (or more) process(es) to synchronize the changes. By default this is rsync. Lsyncd is thus a light-weight live mirror solution that is comparatively easy to install not requiring new filesystems or blockdevices and does not hamper local filesystem performance.
Solution 3
What you want to do for linux remote access is use 'sshfs' - the SSH File System.
# sshfs username@host:path/to/directory local_dir
Then treat it like an network mount, which it is...
A bit more detail, like how to set it up so you can do this as a regular user, on my blog
If you want the asynchronous behavior of winSCP, you'll want to use rsync combined with something that executes it periodically. The cron solution above works, but may be overkill for the winscp use case.
The following command will execute rsync every 5 seconds to push content to the remote host. You can adjust the sleep time as needed to reduce server load.
# while true; do rsync -avrz localdir user@host:path; sleep 5; done
If you have a very large directory structure and need to reduce the overhead of the polling, you can use 'find':
# touch -d 01/01/1970 last; while true; do if [ "`find localdir -newer last -print -quit`" ]; then touch last; rsync -avrz localdir user@host:path; else echo -ne .; fi; sleep 5; done
And I said cron may be overkill? But at least this is all just done from the command line, and can be stopped via a ctrl-C.
kb
Solution 4
To detect changed files, you could try fam (file alteration monitor) or inotify. The latter is linux-specific, fam has a bsd port which might work on OS X. Both have userspace tools that could be used in a script together with rsync.
Solution 5
I have the same issue. I loved winscp "keep remote directory up to date" command. However, in my quest to rid myself of Windows, I lost winscp. I did write a script that uses fileschanged and rsync to do something similar much closer to real time.
How to use:
- Make sure you have fileschanged installed
- Save this script in /usr/local/bin/livesync or somewhere reachable in your $PATH and make it executable
- Use Nautilus to connect to the remote host (sftp or ftp)
- Run this script by doing livesync SOURCE DEST
- The DEST directory will be in /home/[username]/.gvfs/[path to ftp scp or whatever]
A Couple downsides:
- It is slower than winscp (my guess is because it goes through Nautilus and has to detect changes through rsync as well)
- You have to manually create the destination directory if it doesn't already exist. So if you're adding a directory, it won't detect and create the directory on the DEST side.
- Probably more that I haven't noticed yet
- Also, do not attempt to synchronize a SRC directory named "rsyncThis". That will probably not be good :)
#!/bin/sh
upload_files()
{
if [ "$HOMEDIR" = "." ]
then
HOMEDIR=`pwd`
fi
while read input
do
SYNCFILE=${input#$HOMEDIR}
echo -n "Sync File: $SYNCFILE..."
rsync -Cvz --temp-dir="$REMOTEDIR" "$HOMEDIR/$SYNCFILE" "$REMOTEDIR/$SYNCFILE" > /dev/null
echo "Done."
done
}
help()
{
echo "Live rsync copy from one directory to another. This will overwrite the existing files on DEST."
echo "Usage: $0 SOURCE DEST"
}
case "$1" in
rsyncThis)
HOMEDIR=$2
REMOTEDIR=$3
echo "HOMEDIR=$HOMEDIR"
echo "REMOTEDIR=$REMOTEDIR"
upload_files
;;
help)
help
;;
*)
if [ -n "$1" ] && [ -n "$2" ]
then
fileschanged -r "$1" | "$0" rsyncThis "$1" "$2"
else
help
fi
;;
esac
Related videos on Youtube
Jake McGraw
Cofounder & CTO at The Block Been double-clawing my way to the top with PHP since 2004.
Updated on June 23, 2020Comments
-
Jake McGraw almost 4 years
I absolutely love the Keep Remote Directory Up-to-date feature in Winscp. Unfortunately, I can't find anything as simple to use in OS X or Linux. I know the same thing can theoretically be accomplished using changedfiles or rsync, but I've always found the tutorials for both tools to be lacking and/or contradictory.
I basically just need a tool that works in OSX or Linux and keeps a remote directory in sync (mirrored) with a local directory while I make changes to the local directory.
Update
Looking through the solutions, I see a couple which solve the general problem of keeping a remote directory in sync with a local directory manually. I know that I can set a cron task to run rsync every minute, and this should be fairly close to real time.
This is not the exact solution I was looking for as winscp does this and more: it detects file changes in a directory (while I work on them) and then automatically pushes the changes to the remote server. I know this is not the best solution (no code repository), but it allows me to very quickly test code on a server while I develop it. Does anyone know how to combine rsync with any other commands to get this functionality?
-
jfs over 15 yearsgetdropbox.com (mentioned by Scott) will detect file changes in the dropbox's directory and its subdirectories (while you work on them) and then automatically will push the changes to all computers which are linked to your dropbox account.
-
will about 4 yearsDis you look at SyncThing (
https://syncthing.net/
)?
-
-
FaceBro about 10 yearsThis is better, it will only sync the files on changes detected.
-
Anton Rudeshko over 9 years
-
ThorSummoner almost 9 yearsUhh, your blog link doesn't go into any detail on treating sshfs like a network mount.
-
ThorSummoner almost 9 yearsIs this bi-directional?
-
Shautieh over 8 yearslooks like what I was looking for since a few weeks ago!! I tried unison, some fat file sharing servers, sshfs: all were no good. IMHO rsync isn't worth having a look at for this task (unless we want to spend time configuring it along with inotify or equivalent)... lsyncd looks good so far!
-
einpoklum over 8 years@dwj, perhaps you might want to change your answer to use SSH, as rsync directly might sometimes be a closed port (more often than SSH).
-
einpoklum over 8 yearsWhat happens if the changes happen while your computer is not connected to the remote one? Will lsyncd keep a record of those changes for later?
-
Chris Smith over 8 yearsWhat kind of syncing will this do? Keep remote directories up-to-date with local ones? The other way around? Or will it sync both directions and keep the newest one?
-
nick fox over 7 yearsWhat advantage does sshfs have over syncing directly to the remote host in this situation? it seems like an unnecessary overhead and in my experience sshfs can occasionally disconnect.
-
rivu over 3 yearsis there an answer to the uni/bi directional question?