Is (Ubuntu) Linux file copying algorithm better than Windows 7?

Question

Windows Copying is a real mess ever since Windows Vista. Even Microsoft claims they've improved the performance, from a user perspective, it's not quite visible. Even with single file the copying window appears too much time for 'Calculating' and then finishing the copy(Even after 100% completion some times the dialog remains active).

At the same time, I was backing up some files in Ubuntu Linux. I felt it's really fast. Might be a feeling caused by faster UI updates.

I read an informative post from Jeff Atwood few years back on Windows File Copying. but what my specific questions are

• Is (Ubuntu) Linux file performance is better than Windows-7?
• Are both algorithms, Windows and Linux is making use of multiple threads and pipelining mechanism to improve the speed? If yes, which one is better?

Solution 1

The standard copy builtin in Windows isn't very good, but Windows comes with two advanced commands that are far more efficient: xcopy and robocopy. Especially when you need full control over advanced filesystem features like ACLs or alternate data streams, robocopy is the tool of choice.

As an example, if you are copying a large directory structure, you may wish for the target to be unfragmented. The standard invocation of robocopy is already quite good at that, but since directory entries are updated as subdirectories get created, you can still get fragmented directories. To prevent that, you can run robocopy in two passes, the first of which only creates the target directory structure without copying any file content, and the second to actually copy the file bodies.

Altogether, robocopy is very efficient and fully aware of NTFS features.

Solution 2

Some of the speed you see from Ubuntu Linux is probably file caching. As far as I know, file copy in Linux makes no effort to minimize the use of file cache.

With Linux when you copy a file that is smaller than about 50% of your RAM, the entire file is copied into file cache and then written to disk as time permits. This appears to be very fast, but it forces other more useful files out of cache. These might be web cache files, emails, configuration files, font files, etc.

Windows XP used to copy without regard for file cache also. But in Windows Vista, Microsoft changed Explorer so that when it copies it uses a limited amount of file cache. This reduces the impact of large file copies on the rest of the system. But it also reduces the speed of even relatively small file copies to the speed of the hard disk.

Solution 3

In Windows, use Teracopy - it will solve the overall stupidity problem that Explorer has.

In Ubuntu, I found file copying to be very fast.

Still the "File Opertion" dialog does not go away when you are using removable drives. If you try to safely eject the drive immediately after the dialog disappears, you will get some error message. Data gets through though.

Update: Windows file operations are slow probably of all the integration with services and registry. Nautilus (in Gnome) is just a simple process I guess or at least does not have the load/interruptions that Explorer has.

Comments

[Kerrek SB]

In Windows use xcopy, or even robocopy. Much better. For example, with the latter you can make two passes, first to create all the directories but not copy any files, and second to copy the actual files, thus causing no fragmentation on the target.

[sehe]

Or RichCopy (blogs.technet.com/b/ken and download)

[new123456]

Depends also on the filesystem - your system presumably using something like ext3/4, which is competing with NTFS here. You might also try reiserfs and xfs - both of which are optimized for fast operations with small files.

[CarlF]

@KerrekSB, please make your comment an answer so I can upvote it. (I did just +1 the comment, but you know what I mean.) That's the answer.

[GabrielB]

It should be noted (albeit 8 years later!) that Robocopy /CREATE, although very useful to replicate a complete directory structure with empty files, does not by itself prevent fragmentation of files on the target. The structure is created in the MFT, but then the files' contents can be recorded anywhere. Normally even in one pass if Robocopy is running mono-threaded and copies one file at a time, there should be no fragmentation anyway (and running it multi-threaded is a bad idea as it severely reduces the performance on a HDD to HDD copy, since the mechanical drive has to “grind” a lot more).

[sehe]

Solve the stupidity by adding another GUI that must stay open :) In ubuntu, my experience is that you can dismiss the copy/move dialog and it will go to the system tray, showing a notification when done. Not sure about how Natty with Unity desktop does that

[sarat]

Is NTFS better efficient than ext3/4?

[sarat]

Yea Ubuntu using the file caching mechanism. I also noticed the browsing is slightly slow while copying was in progress. I also noticed that the speed is consistently improved especially while copying large number of files

[sarat]

How a two pass robocopy help us to reduce fragmentation, as the real content will be copied later. The disk allocation is entirely controlled by file manager no?

[sarat]

I am doing some tricks like using Beyond Compare to copy files. I.e I can run multiple copying operations to maximize the bandwidth usage. But I am not sure how fragmented it will be

[BZ1]

Teracopy gets itself loaded with Explorer. You do not have to keep it open. So, Teracopy is loaded on to memory without any user effort and it works unobtrusively. Ubuntu does not need anything like that because it has all that intelligence built into Nautilus i.e., "merge" and "skip all" options.

[Kerrek SB]

@Sarath: Regarding fragmentation: The directory entries grow as new files and subdirs are added to the directory. If you just naively create the directory and start putting files in, eventually the original directory record will be too small and has to be extended, but now there's no space because you already have the file data following it. With the /CREATE option, empty files and subdirs are created, which occupy no space and thus allow the directory entries to be written out in full contiguously.

[Kerrek SB]

@Sarath: As for comparing NTFS to Ext, I can't say, because NTFS is not open and the question of efficiency is rather vague. NTFS is certainly a fine piece of technology with a large number of features. There isn't really a choice, though, because you'll use NTFS in Windows and Ext (or BTRFS) in Linux and that's it.

[sehe]

+1: thanks for bringing TeraCopy to my attention then!

[sarat]

I was using a free and open source tool called fastcopy. [ipmsg.org/tools/fastcopy.html.en]

[sarat]

I agree that moving is pretty fast within the partitions. But copying always wanted to have a duplicated date in the disk and create new inodes. Also I doubt moving files between partitions is costly because there's physical file movement happening. Correct me if I am wrong.

[Nandakumar Edamana]

I think this is the answer that addresses the question almost completely.

[GabrielB]

And it should be noted that Robocopy, although it's aware of most NTFS features (even the “sparse” attribute is preserved in my tests), does not preserve the “compressed” attribute, or NTFS compression. So if a partition is almost full and contains many NTFS-compressed files, and it copied to a target of equal capacity with Robocopy, there might not be enough space.

[jjmontes]

@sarat Copying files between partitions (usually) does indeed involve deleting and writing them to a different place in the disk.