rsync not working between NTFS/FAT and EXT

Question

I have music that I play in my car, from an FAT32 USB stick. The folder which I use to put songs on is stored on my EXT4 hard drive. I add/remove/retag songs regularly and occasionally want to rsync the changes to the USB stick. But for some unknown reason (maybe permissions?), rsync copies all the files every time rather than just changed ones. I am calling rsync like:

rsync -vrlptgD source dest

How can I make it work like I want it to (i. e. know when a file hasn't been changed and don't copy it)?

Solution 1

Javier Rivera's answer works, but it takes quite long for rsync to check and compare all file checksums. I found that using the following option worked better for me:

rsync -rtv --modify-window=1 /source /dest

The --modify-window=1 switch allows for a variance of ±1s on the timestamps. With this option enabled, timestamp comparison will be more lenient and look over the minuscule time differences between NTFS/FAT and Unix file systems.

Source (ger): http://www.kai-hildebrandt.de/tutorials/rsync.html

P.S.: Please be aware that DST will cause full file transfers twice a year. See here for further details and possible solutions.

Solution 2

Timestamps in FAT32 are too different from unix ones to rely on them to check for file changes, you should use also the -c switch, it will force rsync to compare all the files to detect changes instead of relying in timestamps. It will work, but it's slower.

Finally, there are a couple of options in your command that can't work with FAT32 file systems.

• -l will preserve links, FAT32 has no concept of links
• -p will try to preserve permission, again no permissions on FAT32
• -t will try to preserve modification timestamps, there is only one timestamp on FAT32
• -g will try to preserve group ownership, again not supported by FAT32
• -D will try to preserver special files and devices, you now what comes here.

As htorque comments, the invalid options will no hurt you, they just will do nothing. But you must add -c switch.

This:

rsync -vrc source dest

should work (at least it works on my computer).

Solution 3

I was having a similar issue under OSX, and Glutanamate's answer didn't help. Some of the files differ by an hour; this might be because I tend to cross timezones relatively often. Other files are off by a day or even a month. I'm not sure why this is. Checksumming on some of the files with widely differing timestamps shows that they are, indeed, identical.

In any case, it looks like the --size-only option, which tells rsync to ignore timestamps, will work for my purposes. -c / --checksum (as mentioned by Javier) also works, but takes a bit longer. I timed it and it took about a minute to compare checksums for the GB or so in the subdirectory I'm working with. Of course the speed at which this happens will be dependent on the slowest drive in the system; in my case, that's the SD card in my phone. However, that was after I'd already been doing some file manipulation (including checksumming), so many of the files may have already been copied into RAM cache.

Solution 4

You should also avoid using the popular -a option. My recommendation on FAT32 is

 rsync -vrc --delete --progress --no-p source  destination

--no-p : no permission

--delete : delete unmatched files and folder in destination(if you really want this)

--progress : show progress during transfer. It is good for large files.

Solution 5

That's too many flags (-vrlptgD) you're using. Remember, rsync is a Linux utility and doesn't work with Fat32 and NTFS effectively.

You'd have to hunt for tricks to be able to use it.

Try:

rsync -rvh --size-only --progress --delete /path/to/ext4/ /path/to/fat32/

More info here

View more solutions

Comments

[wim]

This is great. I found also the option --iconv helpful when going from source linux to dest MacOS

[Luc]

For some reason I need to use modify-window=2 because =1 still copies all files. Copying from NTFS to FAT32. After that it's fine though.

[Antony]

Have an upvote from me although in additional to -vrc, I also do -vrcz

[alexandre1985]

I added the --size-only flag to skip files that match in size

[David Foerster]

@alexandre1985: That will omit files that changes their content but not their size. You should only use it when you only want to back up files that always change size when they change content, e. g. because data is only appended to them but never overwritten.

[Gringo Suave]

-c is --checksum for any folks wondering.