Looking for a way to copy a file that does not lock the file being copied

Question

I am aggregating together data that is being generated at a bunch of disparate sources, in a mixed environment of various versions of windows (xp/7) and one linux (centos). The data is available via network shares. The data is being produced by different software at each location, creating and appending to files at varying intervals. Most of the data sources are closed systems I do not have administrative access to, which really complicates the situation.

I am needing to periodically attach to the network shares and collect together the data into a central location, used for further distribution. The data is being generated and collected 24/7, for weeks at a time. This will be run from a Windows 2008 server machine.

The only hassle I am having with organizing this, is copying a file from a remote source without locking the file. I had initially set up a system that worked well using robocopy, until my testing found that the data source software would fail if it tried to append to a file currently being copied.

Unfortunately using vss is not an option as they are not volumes that are shared.

So, I am after some way to copy the file that will simply not lock the file while it is being copied. I am handling the files by checking modify times and sizes before and after copy, and flagging those with differences. Not having the exact most recent files is not an issue, but data loss at the source is.

I have looked into various backup software solutions but all that I have tested still lock files when they can't use vss.

I don't want to have to implement my own file copy software for something that should be pretty simple.

Solution 1

Not sure if XXCOPY can do what you want so you may want to read a post regarding its access to locked files. Otherwise, it sounds like XCOPY (and RoboCopy) on steroids. Not free for business use, although it is still being supported (latest version released on 12/27/2013)

Addition: The documentation says that XXCOPY itself does not lock the file while it copies it.

Solution 2

Just to add another option, I have found FastCopy to copy files in shared read/write mode.

It seems fairly mature and has been around since 2004, and is open source under the BSD license found here. It is seeming like a good alternative, as not all the extra bells and whistles that XXCOPY provides are required.

Solution 3

FileCopy.exe claims to be able to do this but it is rather old (2004) so I don't know if it will work for you. In addition, it isn't clear whether the copy would remain non-blocking when used over a network share.

It is, of course, quite dangerous to do non-blocking copies which is why most utilities don't support it. If the master application updates the file while you are copying, you are likely to end up with a corrupted copy.

Comments

[Daniel R Hicks]

This is a tricky job, if you want an atomic view. It's done either by making a shadow copy (a la Unix) or by enabling a journal on the file. But if you can tolerate a changing view then it's a simple matter of opening a shared read that allows simultaneous write (if the OS permits that mode).

[JdeBP]

Non-blocking is a rather different concept when it comes to opening files. The right terminology is sharing mode, in this particular case with a sharing mode that allows reading and writing (and possibly FILE_SHARE_DELETE as well, although the doco indicates probably not) by other processes that open the file.

[Steve]

Great, thanks for that - XXCOPY looks like a viable way to go. the flag /SHRW opens in shared mode, which is just what I am looking for. Just tested it now, and my acquisition systems are able to continue appending to the data file during the copy :)

[Scott - Слава Україні]

Also, he says he is “checking modify times and sizes before and after copy, and flagging those with differences” -- that should detect most copies that are corrupted by concurrent modification of the source file.

[Julian Knight]

Deserves a +1 @JSanchez, good find. I've used it in the past and had completely lost track of it.

[Steve]

Thanks @JSanchez, XXCopy does do what I want. The only sticking point is their licensing, which requires a license for every machine that xxcopy touches. This might be problematic as a portion of the machines that make up the environment would belong to external visiting groups, rotating through every month or so. That however is an issue separate to what I outlined. Thanks again!