copy|move sparse files on NTFS with Windows

Question

How can I copy or move a sparse file from one NTFS volume to another NTFS volume while maintaining the sparseness using the native tools included with Windows? If there is no way to do this with the native tools, then what is a free application from a trusted vendor that will do this? Surely there is a command prompt command or powershell command that will do this.

I have a file that represents 250GiB that is taking up 20GiB of disk space and I would like it to continue only taking up 20GiB when I move it. Thanks.

Solution 1

You cannot do it with standard Windows tools.

But there is an utility. Go to http://www.flexhex.com/docs/articles/sparse-files.phtml, download and unpack sparse.zip.

Then run from from the command line:

cs.exe c:\src.dat d:\dest.dat

The file will be copied preserving sparseness.

Solution 2

Use Cygwin and mount the filesystem with the sparse option. Then Unix commands that support sparse files, such as cp, dd conv=sparse, and rsync -S, will correctly create or copy the file as a sparse one. As one would expect, simple output redirection will not create a sparse file.

See the following demonstration.

$ mount -o sparse D: /tmp/mnt
mount: warning - /tmp/mnt does not exist.
$ cd /tmp/mnt
$ dd conv=sparse if=/dev/zero seek=10000 of=sparse count=1
1+0 records in
1+0 records out
512 bytes copied, 0.0101909 s, 50.2 kB/s
$ ls -lh sparse
-rw-rw-r--+ 1 dds None 4.9M Sep  5 13:05 sparse
$ du -h sparse
0       sparse
$ cp sparse sparse-cp
$ dd conv=sparse if=sparse of=sparse-dd
10001+0 records in
10001+0 records out
5120512 bytes (5.1 MB, 4.9 MiB) copied, 0.0500354 s, 102 MB/s
$ rsync -S sparse sparse-rsync
$ cat sparse >sparse-fail-cat
$ cat sparse | dd conv=sparse of=sparse-cat-dd
$ ls -lh sparse*
-rw-rw-r--+ 1 dds None 4.9M Sep  5 13:05 sparse
-rw-rw-r--+ 1 dds None 4.9M Sep  5 13:15 sparse-cat-dd
-rw-rw-r--+ 1 dds None 4.9M Sep  5 13:06 sparse-cp
-rw-rw-r--+ 1 dds None 4.9M Sep  5 13:11 sparse-dd
-rw-rw-r--+ 1 dds None 4.9M Sep  5 13:19 sparse-fail-cat
----rw----+ 1 dds None 4.9M Sep  5 13:06 sparse-rsync
$ du -h sparse*
0       sparse
0       sparse-cat-dd
0       sparse-cp
0       sparse-dd
4.9M    sparse-fail-cat
0       sparse-rsync

Comments

[kobaltz]

What kind of sparse file is it? Would you be able to use a utility to shrink it after it's moved? If it is a VM Image, it should be fine to copy and retain just the space that is already allocated.

[Michael Yasumoto]

It is a file system level sparse file, as in an NTFS file with the 'P' extended attribute set on it. It is not an application level sparse file which adds sparse support (and overhead) regardless of whether the file system already supports sparseness or not such as with a vmdk file. You're suggesting I transfer 250GiB, then spend the CPU to compress that, and then uncompress back to a 250GiB file whenever I want to actually access its contents. I'd rather move the small 20GiB file and have it stay 20GiB and remain accessible to me in its destination. Much less network, CPU, and storage use.

[Charles Roberto Canato]

I'll surely try this in the next few days, when I'll have to try to move a good lot of sparse files from a host to another. Would you happen to know if this sparse mount would also work onto a network mapped dir (basically, could I achieve the same by mounting the drive of another computer in the network)? Of course, I'm ready to move the new HDD to the same system and do it, if I have to, but that could save a hardware move.

[piojo]

To make this run on WSL, changed the fsutil invocation to fsutil.exe sparse setflag "$(wslpath -w "$to")". But I'm still ironing out the difficulties--it seems to work copying sparse files to and from a Windows directory but not from a WSL directory to a Windows directory.