How do I force Robocopy to overwrite files?
Solution 1
From the documentation:
/isIncludes the same files./itIncludes "tweaked" files.
"Same files" means files that are identical (name, size, times, attributes). "Tweaked files" means files that have the same name, size, and times, but different attributes.
robocopy src dst sample.txt /is # copy if attributes are equal
robocopy src dst sample.txt /it # copy if attributes differ
robocopy src dst sample.txt /is /it # copy irrespective of attributes
This answer on Super User has a good explanation of what kind of files the selection parameters match.
With that said, I could reproduce the behavior you describe, but from my understanding of the documentation and the output robocopy generated in my tests I would consider this a bug.
PS C:\temp> New-Item src -Type Directory >$null
PS C:\temp> New-Item dst -Type Directory >$null
PS C:\temp> New-Item src\sample.txt -Type File -Value "test001" >$null
PS C:\temp> New-Item dst\sample.txt -Type File -Value "test002" >$null
PS C:\temp> Set-ItemProperty src\sample.txt -Name LastWriteTime -Value "2016/1/1 15:00:00"
PS C:\temp> Set-ItemProperty dst\sample.txt -Name LastWriteTime -Value "2016/1/1 15:00:00"
PS C:\temp> robocopy src dst sample.txt /is /it /copyall /mir
...
Options : /S /E /COPYALL /PURGE /MIR /IS /IT /R:1000000 /W:30
------------------------------------------------------------------------------
1 C:\temp\src\
Modified 7 sample.txt
------------------------------------------------------------------------------
Total Copied Skipped Mismatch FAILED Extras
Dirs : 1 0 0 0 0 0
Files : 1 1 0 0 0 0
Bytes : 7 7 0 0 0 0
...
PS C:\temp> robocopy src dst sample.txt /is /it /copyall /mir
...
Options : /S /E /COPYALL /PURGE /MIR /IS /IT /R:1000000 /W:30
------------------------------------------------------------------------------
1 C:\temp\src\
Same 7 sample.txt
------------------------------------------------------------------------------
Total Copied Skipped Mismatch FAILED Extras
Dirs : 1 0 0 0 0 0
Files : 1 1 0 0 0 0
Bytes : 7 7 0 0 0 0
...
PS C:\temp> Get-Content .\src\sample.txt
test001
PS C:\temp> Get-Content .\dst\sample.txt
test002
The file is listed as copied, and since it becomes a same file after the first robocopy run at least the times are synced. However, even though seven bytes have been copied according to the output no data was actually written to the destination file in both cases despite the data flag being set (via /copyall). The behavior also doesn't change if the data flag is set explicitly (/copy:d).
I had to modify the last write time to get robocopy to actually synchronize the data.
PS C:\temp> Set-ItemProperty src\sample.txt -Name LastWriteTime -Value (Get-Date)
PS C:\temp> robocopy src dst sample.txt /is /it /copyall /mir
...
Options : /S /E /COPYALL /PURGE /MIR /IS /IT /R:1000000 /W:30
------------------------------------------------------------------------------
1 C:\temp\src\
100% Newer 7 sample.txt
------------------------------------------------------------------------------
Total Copied Skipped Mismatch FAILED Extras
Dirs : 1 0 0 0 0 0
Files : 1 1 0 0 0 0
Bytes : 7 7 0 0 0 0
...
PS C:\temp> Get-Content .\dst\sample.txt
test001
An admittedly ugly workaround would be to change the last write time of same/tweaked files to force robocopy to copy the data:
& robocopy src dst /is /it /l /ndl /njh /njs /ns /nc |
Where-Object { $_.Trim() } |
ForEach-Object {
$f = Get-Item $_
$f.LastWriteTime = $f.LastWriteTime.AddSeconds(1)
}
& robocopy src dst /copyall /mir
Switching to xcopy is probably your best option:
& xcopy src dst /k/r/e/i/s/c/h/f/o/x/y
Solution 2
This is really weird, why nobody is mentioning the /IM switch ?! I've been using it for a long time in backup jobs. But I tried googling just now and I couldn't land on a single web page that says anything about it even on MS website !!! Also found so many user posts complaining about the same issue!!
Anyway.. to use Robocopy to overwrite EVERYTHING what ever size or time in source or distination you must include these three switches in your command (/IS /IT /IM)
/IS :: Include Same files. (Includes same size files)
/IT :: Include Tweaked files. (Includes same files with different Attributes)
/IM :: Include Modified files (Includes same files with different times).
This is the exact command I use to transfer few TeraBytes of mostly 1GB+ files (ISOs - Disk Images - 4K Videos):
robocopy B:\Source D:\Destination /E /J /COPYALL /MT:1 /DCOPY:DATE /IS /IT /IM /X /V /NP /LOG:A:\ROBOCOPY.LOG
I did a small test for you .. and here is the result:
Total Copied Skipped Mismatch FAILED Extras
Dirs : 1028 1028 0 0 0 169
Files : 8053 8053 0 0 0 1
Bytes : 649.666 g 649.666 g 0 0 0 1.707 g
Times : 2:46:53 0:41:43 0:00:00 0:41:44
Speed : 278653398 Bytes/sec.
Speed : 15944.675 MegaBytes/min.
Ended : Friday, August 21, 2020 7:34:33 AM
Dest, Disk: WD Gold 6TB (Compare the write speed with my result)
Even with those "Extras", that's for reporting only because of the "/X" switch. As you can see nothing was Skipped and Total number and size of all files are equal to the Copied. Sometimes It will show small number of skipped files when I abuse it and cancel it multiple times during operation but even with that the values in the first 2 columns are always Equal. I also confirmed that once before by running a PowerShell script that scans all files in destination and generate a report of all time-stamps.
Some performance tips from my history with it and so many tests & troubles!:
. Despite of what most users online advise to use maximum threads "/MT:128" like it's a general trick to get the best performance ... PLEASE DON'T USE "/MT:128" WITH VERY LARGE FILES ... that's a big mistake and it will decrease your drive performance dramatically after several runs .. it will create very high fragmentation or even cause the files system to fail in some cases and you end up spending valuable time trying to recover a RAW partition and all that nonsense. And above all that, It will perform 4-6 times slower!!
For very large files:
- Use Only "One" thread "/MT:1" | Impact: BIG
- Must use "/J" to disable buffering. | Impact: High
- Use "/NP" with "/LOG:file" and Don't output to the console by "/TEE" | Impact: Medium.
- Put the "/LOG:file" on a separate drive from the source or destination | Impact: Low.
For regular big files:
- Use multi threads, I would not exceed "/MT:4" | Impact: BIG
- IF destination disk has low Cache specs use "/J" to disable buffering | Impact: High
- & 4 same as above.
For thousands of tiny files:
- Go nuts :) with Multi threads, at first I would start with 16 and multibly by 2 while monitoring the disk performance. Once it starts dropping I'll fall back to the prevouse value and stik with it | Impact: BIG
- Don't use "/J" | Impact: High
- Use "/NP" with "/LOG:file" and Don't output to the console by "/TEE" | Impact: HIGH.
- Put the "/LOG:file" on a separate drive from the source or destination | Impact: HIGH.
Solution 3
I did this for a home folder where all the folders are on the desktops of the corresponding users, reachable through a shortcut which did not have the appropriate permissions, so that users couldn't see it even if it was there. So I used Robocopy with the parameter to overwrite the file with the right settings:
FOR /F "tokens=*" %G IN ('dir /b') DO robocopy "\\server02\Folder with shortcut" "\\server02\home\%G\Desktop" /S /A /V /log+:C:\RobocopyShortcut.txt /XF *.url *.mp3 *.hta *.htm *.mht *.js *.IE5 *.css *.temp *.html *.svg *.ocx *.3gp *.opus *.zzzzz *.avi *.bin *.cab *.mp4 *.mov *.mkv *.flv *.tiff *.tif *.asf *.webm *.exe *.dll *.dl_ *.oc_ *.ex_ *.sy_ *.sys *.msi *.inf *.ini *.bmp *.png *.gif *.jpeg *.jpg *.mpg *.db *.wav *.wma *.wmv *.mpeg *.tmp *.old *.vbs *.log *.bat *.cmd *.zip /SEC /IT /ZB /R:0
As you see there are many file types which I set to ignore (just in case), just set them for your needs or your case scenario.
It was tested on Windows Server 2012, and every switch is documented on Microsoft's sites and others.
tbl
Updated on August 21, 2020Comments
-
tbl almost 4 years
In general, Robocopy ignores files for which lastwrittendate and filesize are the same. How can we escape this design? I'd like to force overwriting with Robocopy.
I expected that dst\sample.txt should be written test001. But these file are recognized as the same files by Robocopy and not overwritten. The "/IS" option is not effective in this case.
New-Item src -itemType Directory New-Item dst -itemType Directory New-Item src\sample.txt -itemType File -Value "test001" New-Item dst\sample.txt -itemType File -Value "test002" Set-ItemProperty src\sample.txt -Name LastWriteTime -Value "2016/1/1 15:00:00" Set-ItemProperty dst\sample.txt -Name LastWriteTime -Value "2016/1/1 15:00:00" ROBOCOPY.exe src dst /COPYALL /MIR Get-Content src\sample.txt, dst\sample.txt > test001 > test002 ROBOCOPY.exe src dst /COPYALL /MIR /IS Get-Content src\sample.txt, dst\sample.txt > test001 > test002