Robocopy falsely marks files as newer

Question

I have two NTFS formatted drives on different machines and I do a robocopy mirror between them like this:

robocopy "D:\My Documents" "X:\Backup\My Documents" /MIR /V /NP /XD .* /XO /XA:SHTO /FAT /Z /ETA /DST /R:3 /W:30

But it will falsely mark files as newer, but I haven’t altered it, and it is a lots of files so I don’t think windows have altered them either.

I am running the robocoy in a steady loop and it takes about 15 minutes, so I think something is a bit wrong, anyone have this issue?

Example output:

• same 17548 draft1.jpg
• same 98108 draft2.jpg
• same 104 Spiral3DCopy1.mtl
• Newer 104 Sphere3DCopy1.mtl
• Newer 345985 swamp1.jpg
• Newer 75209 test.jpg

I have done a compare and no differences are found:

C:\>fc D:\Documents\"My Documents"\3d\Sphere3DCopy1.mtl X:\Backup\"My Documents"\3d\Sphere3DCopy1.mtl
Comparing files D:\DOCUMENTS\MY DOCUMENTS\3D\Sphere3DCopy1.mtl and X:\BACKUP\MY DOCUMENTS\3D\SPHERE3DCOPY1.MTL
FC: no differences encountered

Solution

We found that using the /FFT Robocopy switch will reduce the timestamp to about 2 seconds difference when comparing file changes. It eliminated the 2 hours copying everything when the changed files would normaly take under 3 minutes.

RoboCopy /? says to use this switch to "assume FAT File Times (2-second granularity)."

Thanks to the folks at connetrix.

http://www.conetrix.com/Blog/post/Robocopy-Incorrectly-Detects-Files-as-Newer-When-Copying-Between-Different-File-Systems.aspx

Try this:

robocopy "D:\My Documents" "X:\Backup\My Documents" /FFT /MIR /V /NP /XD .* /XO /XA:SHTO /FAT /Z /ETA /DST /R:3 /W:30

It worked for us.

Comments

[FooBee]

Is the time set right on both machines?

[Cinaird]

Yes, it is, not all files is marked "newer" about 20% maybe

[Trevor Sullivan]

Is there anything common about the filetypes that are getting marked newer?

[Cinaird]

@TrevorSullivan: No, it is randomly distributed among my most common filetypes in the folder. Here is a sample of the most common filetypes: mtl, obj, jpg, png, pdf, pdf, odt, html, txt, psd, rar

[designarti]

Might help if you included some output from the command so we can check your assumptions.

[Cinaird]

@LarrySilverman Good observation but no, they are in the same building, and the only reason I have /FAT /DST /XO is to overcome the “Newer” bug, but obviously it does not work. I have tried with, without and all the combination of them.

[designarti]

Is robocopy the only thing that thinks the old files have become "freshened"? I wonder if you watched D:\My Documents with something like SysInternals procmon if you could detect some other process touching those files.

[Cinaird]

@LarrySilverman I’m pretty new to this, so bear with me if I don’t do things right. I run the backup script twice after each other and look in the Process Monitor it doesn’t show any *.mtl files ass accessed, but shouldn’t it show at least twice? And the Modified date Is still 2008-04-07. I think I have missed something.

[designarti]

Are you absolutely sure there are no processes that might be modifying these files on either side? It could be something touching the files on the sending side, making them seem newer (antivirus?); or something touching files on the receiving side, modifying to make them seem older.

[designarti]

I thought perhaps there was an option to only copy files if the size had changed, ignoring the timestamps, but I'm not seeing a switch for that in the robocopy /? help output. Perhaps I'm thinking of rsync.

[designarti]

This might be useless to you if you're trying to script this, but have you seen/tried SyncToy from Microsoft? It's a GUI file sync tool, rather than command line.

[Cinaird]

@LarrySilverman, Actually I’m not sure how to be absolutely sure, but I think the files are untouched what I can see in Procmon, maybe it’s on the receiving end. The option to only copy files if the size had changed would be perfect, but sadly I haven’t found it either. I think my best option is to test with some other synchronization software. Thanks for all the help.

[Cinaird]

@LarrySilverman ok, now i have done a fc and a compare, and both say no differences

[designarti]

Take a look at stackoverflow.com/questions/2871931/… -- Is it possible your /XO is competing with /MIR? Perhaps try getting rid of /MIR and explicitly setting /XO.

[designarti]

And one more idea. Try running it with /L. This will do a "what if" scenario, without actually making changes. Let robocopy tell you which files it thinks are newer. Then pick one or two and go check the timestamps on both machines and see if they are actually different. Honestly, you've got a very odd situation, and it makes me think you've got some rogue program changing timestamps on you.

[Cinaird]

@LarrySilverman Ok, no I did a test I should have done much earlier. I tried a different destination. Same computer as the source, but different hard drive. It worked like a charm, so now I know the problem is my destination. The destination is a Boxee box, so I don’t have so much control. Thaks for all the help. I tested what you suggested, but there was no difference on the files timestamp, and sadly just /XO did not do the trick. I will try some new settings on the box and hopefully it will work.

[designarti]

You said both source and destination are NTFS-formatted partitions. You have an NTFS-formatted partition mounted to a Boxee Box?

[Cinaird]

@LarrySilverman I formatted the disc as NTFS and then mounted it, so if Boxee Box dose'nt formated it the, it should be NTFS, else i don't know