rsync --iconv option on Mac not working (sync from remote Linux server to local Mac)

Question

I want to use rsync to backup data from a remote Linux server to my local Mac. And I want to initialise this operation on my local Mac. All works fine except that there is a special character problem: every time I re-run the rsync operation (after the initial sync), the files with special characters are first deleted and then re-synced. As far as I understand, there is a problem with different character sets, and the preferred solution seems to be to use the --iconv option:

You can use rsync's --iconv option to convert between UTF-8 NFC & NFD, at least if you're on a Mac. There is a special utf-8-mac character set that stands for UTF-8 NFD. So to copy files from your Mac to your NAS, you'd need to run something like:

rsync -a --iconv=utf-8-mac,utf-8 localdir/ mynas:remotedir/

This will convert all the local filenames from UTF-8 NFD to UTF-8 NFC on the remote server. The files' contents won't be affected.

The problem is that this only works 'one way' for me, namely when syncing from the Mac to the linux. But I want to 'go the other way', i.e. sync FROM the linux machine TO the Mac. And I want to initialise the operation from my local Mac. But when I try:

rsync -av --delete --iconv=utf-8,utf-8-mac mynas:remotedir/ localdir/

I receive an error:

iconv_open("UTF-8", "utf-8-mac") failed
rsync error: requested action not supported (code 4) at rsync.c(118) [sender=3.0.9]
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.1]

I am at loss to understand why this does not work. My rsync version on the Mac is updated from 2.6.9. to 3.1.1. using Macports. Note that the operation works then when I (on the Mac, nota bene) initiate a rsync FROM the Mac TO Linux:

rsync -av --delete --iconv=utf-8-mac,utf-8 localdir/ mynas:remotedir/

But going the other way' from the mac – which is what I want to do – does not work.

Strangely enough, testing to initiate the sync from the linux machine renders this strange message:

rsync: on remote machine: --iconv=UTF-8-MAC: unknown option
rsync error: syntax or usage error (code 1) at /SourceCache/rsync/rsync-45/rsync/main.c(1333) [server=2.6.9]
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]

including, note, the very strange claim [server=2.6.9], although I have updated to 3.1.1 on the Mac. For some reasons, it looks as my linux machine 'sees' only the original rsync version on the Mac.

Any suggestion on how to solve this?

UPDATE October 23: Per @Lee Johnson's excellent suggestion (see below), initiating the sync from the linux server now works. For completeness, I have now tried all combinations, and an interesting pattern emerges:

ON MAC:

WORKS: Files from Mac to Linux

FAILS: Files from Linux to Mac

ON LINUX

WORKS: Files from Linux to Mac

FAILS: Files from Mac to Linux

In other words, the --iconv option seems to work only one way, with files from the local machine to the remote, not the other way around. It looks like a bug to me, but maybe that is the way it is SUPPOSED to work?

Anyone able to share light on this?

Solution 1

After a lot of experimenting, and very much due to @Lee Johnson's helpful suggestions, I finally found out the solution, which now strikes me as embarrassingly obvious. Much due to a comment I read when researching the problem, I thought you were supposed to specify the character set in the order of transformation; but it seems as that is not the correct syntax. Rather, one should

ALWAYS use --iconv=utf-8-mac,utf-8 when initialising the rsync from the mac, and ALWAYS use --iconv=utf-8,utf-8-mac when initialising the rsync from the linux machine, no matter if I want to sync files from the mac or linux machine.

Then it works like magic!

Solution 2

Did you recently upgrade to OS X Yosemite? I had the same problem, before I remembered that I had updated /usr/bin/rsync with version 3.1. When I upgraded to Yosemite, this got replaced with the old 2.6.9 version.

In my own case, I fixed the problem on the Mac by relinking my 3.1 rsync back into /usr/bin:

sudo -s
cd /usr/bin
mv rsync rsync-2.6.9
ln -s /usr/local/bin/rsync .
exit

Solution 3

The way I've solved this issue on Mac OS Big Sur:

Install rsync new version with brew:

$:brew install rsync

display rsync path list:

$:rsync list rsync

/usr/local/Cellar/rsync/3.2.3/bin/rsync
/usr/local/Cellar/rsync/3.2.3/bin/rsync-ssl
/usr/local/Cellar/rsync/3.2.3/share/man/ (3 files)

backup the old version binary command link:

$:mv /usr/local/bin/rsync /usr/local/bin/rsync2

make the new version binary command link:

$:ln -s /usr/local/Cellar/rsync/3.2.3/bin/rsync /usr/local/bin/rsync

Comments

[Alkesh Kumar]

when using a custom rsync (eg from homebrew) on the mac and calling it from linux, it is necessary to specify the correct path using --rsync-path="/usr/local/bin/rsync"

[Daniel]

I was excluding .DS_Store from syncs and because of this OSX couldn't delete directories with these files inside. I setup the character sets with --iconv, the rsync path on the mac with --rsync-path (I'm using homebrew), and then had to add --delete-excluded so that the stubborn directories could be deleted.

[Nick The Swede]

Thanks a million, that solves the mystery of why I got that 2.6.9. message. (On my Mac, though, the Macport installed version is in /opt/local/bin/rsync, but changing the link to that sport worked magic.) Unfortunately, I want to initialise the sync from my MAC machine, so this only helps me as far as understanding that my Linux machine can figure out what to do. So why does it not work when initialised from my Mac? That is, the "rsync -av --delete --iconv=utf-8,utf-8-mac mynas:remotedir/ localdir/"

[Nick The Swede]

Let me also say that unfortunately I have too low a reputation to be able to +1 your helpful answer, and since it still does not work the way I want it to, I cannot tick it as solved. Gold star in my mind, in any case (and I promise to go back and +1 you as soon as my rep reaches above 15)!

[Lee Johnson]

Are you saying that it still doesn't work from the OS X side, even with rsync 3.x running? I don't think that --iconv is supported in 2.6.9; even if rsync is just shipping the option off to the remote host for handling, it needs to recognize the option on the OS X side. What does which rsync; rsync --version tell you, from an OS X terminal?

[Nick The Swede]

That's correct. As you can see in the error message (third grey quote in the question), it does recognise that I am using 3.1 on the mac: [Receiver=3.1.1], and claims that the action is not supported, although it evidently works from the Linux side, as well as when I sync, from the Mac, files on the mac to the linux server. But from the mac, files from the linux to mac does not work. So strange (to my noob eyes at least).

[Lee Johnson]

When you're trying this from Linux, what happens if you force the executable path with something like --rsync-path=/opt/local/bin/rsync to get your known 3.1.1 version on the Mac side?

[Nick The Swede]

Thanks,but there is no need for that, since it does recognise the 3.1.1. version, thanks to your link solution (and indeed, nothing happens when I add the path option). But returning to the problem, it did some more experimenting, and finally I managed to solve the remaining problem, which had to do with order of character set specification!

[Achim Lammerts]

UTF8-MAC is a pseudo-charset and it's not available per se with iconvlib on a Linux system, even not with the latest 3.1.1 version in Ubuntu 14.04 LTS. That's not working if you try to initiate the sync on Linux.

[J. Doe]

brew install rsync solved this for me. Appreciate the insight.