ROBOCOPY command to do an incremental backup

Question

I am overwhelmed by the ROBOCOPY documentation. I want to do an incremental backup of my local files to the network drive (M). I want it to run as quickly as possible, with no log file and with as little text as possible to the screen. My files are all somewhere inside one folder (MyFolder) that has many nested subfolders. By "incremental" I mean "only copy what is new or changed." I don't want to delete any historical files in the destination, but if I've changed a file I only want the newer version. And if I have not changed a file, then I don't want to copy it over the existing backed-up file. Is below correct? (Running Win7 Enterprise.)

robocopy C:\MyFolder M:\MyFolder /z /np /xo /e

Please, no suggestions for anything but robocopy. I'm not allowed to install anything. And I don't care about security stuff. I have people for that (whether I want them or not). ;-)

Solution 1

I have been successfully using a variant of the following script for a few years now:

robocopy C:\source M:\destination /MIR /FFT /R:3 /W:10 /Z /NP /NDL

Parameters explained

• The /MIR option (equivalent to /E /PURGE) stands for "mirror" and is the most important option. It regards your source folder as the "master", causing robocopy to copy/mirror any changes in the source (new files, deletions etc.) to the target, which is a useful setting for a backup.

• /FFT is a very important option, as it allows a 2-second difference when comparing timestamps of files, such that minor clock differences between your computer and your backup device don't matter. This will ensure that only modified files are copied over, even if file modification times are not exactly synchronized.

• /R:3 specifies the number of retries, if the connection should fail, and /W:10 specifies a wait time of 10 seconds between retries. These are useful options when doing the backup over a network.

• /Z copies files in "restart mode", so partially copied files can be continued after an interruption.

• /NP and /NDL suppress some debug output, you can additionally add /NS, /NC, /NFL to further reduce the amount of output (see the documentation for details). However, I would suggest to print some debug output during the first runs, to make sure everything is working as expected.

Additional useful parameters mentioned by other users

• /XJD excludes "junction points" for directories, symbolic links that might cause problems like infinite loops during backup. See Brian's comments for details.

• /MT[:N] uses multithreading and can speed up transfers of many small files. For N, a value of 2-4 times the number of cores should do on a normal machine. Commented by Zoredache on the original question.

Edit in response to Granger's comment:

If you really want to keep files that exist on the destination, but not on the source side, simply replace the /MIR option with /E. However, I would strongly suggest to use /MIR when you want to use the destination for incremental backups. Otherwise any files that have been renamed or moved at the source will clutter up the destination, meaning you get duplicates. I usually create a subfolder "backup" on the destination which contains a 1:1 copy of my source folder tree. That way you can still keep around historical files next to the backup folder and remove or reorganize them safely later on.

Solution 2

I like to use the following:

robocopy "C:\Users\<user>" "F:\robocopy\<user>" /XJD /R:0 /XA:SH /E /ZB /XO /XD "Downloads" "AppData" /LOG:robocopy.log /TEE

I run this as Administrator so backup mode (/b option) can make copy of files in use.

Other options not related to incremental nature of backup are:

/XD to exclude directories from backup.

/XJD to exclude junction points ("My Music").

/R:0 to set retry on failed attempts to 0.

/XA:SH to skip hidden and system files.

To remove as much logging info as possible, append the following options /NP /NS /NDL /NFL /NC.

Comments

[David Marshall]

You need a /s or /e option to copy subdirectories.

[dmm]

@DavidMarshall: oops. I actually knew that. Added. So, am I good now?

[David Marshall]

Yes. I've been using something similar.

[dmm]

Testing it now. There is still a bunch of stuff being written to the screen. How do I get rid of it, in future runs?

[Zoredache]

One option that can make a big difference performance wise is to have multithreaded transfers. /MT[:N] I have a 4 core CPU so I typically use something in the 8-16 rage for my number of threads. I figure 2-4 threads per core. You should certainly test with different values on your hardware. Multithreaded helps the most when you have a lot of small files. If you have a small number of large files it won't help nearly as much.

[KCD]

FYI I always use /L first as a dry run to see what robocopy will do and watch out for Extras that may get deleted depending on your options

[jiggunjer]

FYI Robocopy has a GUI tool to help

[Shabeer Sher]

Does this command allow to copy files with permissions? robocopy C:\source M:\destination /MIR /FFT /R:3 /W:10 /Z /NP /NDL

[Granger]

Except /MIR (/PURGE) will delete files at the destination which no longer exist at the source. The question asked explicitly stated that was not wanted.

[pederpansen]

@Shabeer No, Robocopy by default only copies data (D), attributes (A) and timestamps (T). Other values have to be specified with the "/copy:<value>" parameter, e.g. "/copy:DATS" for including ACL permissions. But since I copy to a Linux-based NAS, that doesn't make much sense for me.

[ejbytes]

Nice! I've been using robocopy too. But I got some good ideas reading this. Maybe a good idea to implement something like this: IF (TODAY DestFileLastDate > 30Days AND SourceFile does not EXIST in DESTINATION) DO {MIRROR}. I'll be trying that soon perhaps.

[Gary - Stand with Ukraine]

Warning: If you do not add /XJ or /XJD, robocopy may try to create an infinitely nested directory, stopping only when you run out of disk space. Such a directory can be deleted by calling robocopy /purge c:\[emptydirectory] c:\[corruptdirectory] (windows explorer cannot delete such directories).

[pederpansen]

@Brian Thanks for the hint. I've never had problems with junction points, but I guess it won't hurt to avoid them just in case. Added it in my answer.

[Gary - Stand with Ukraine]

@pederpansen: It tends to happen when copying user folders (or copying an entire drive). Usually the application data folder is the culprit, as it has a junction point for application compatibility reasons. Try running "cd C:\Users\[USERFOLDER]\AppData\Local\Application Data\application data\application data\" from a command prompt if you want to see a sample infinite path. This works even on Windows 10.