Slow File Copy observed copying 40GB files across network to iSCSI device

Question

Here's a curious ones for the gurus:

Setup:

Source Machine: Windows Server 2003 R2 machine with local hard drive. VHD file of 40GB. 1 x 1Gbps network card, Cat6 cable, switch.

Target Machine: Windows Server 2008 R2 machine with iSCSI connection to iSCSI target on separate machine (1TB, RAID5). 1 x 1Gbps network card, Cat6 cable, connected to same switch as for Source Machine. Second 1Gbps network card, Cat6 cable, connected via isolated switch to the iSCSI target.

Switches are Netgear JGS524 model (web managed).

If I copy from the Win2003R2 machine to Win2008R2 machine local drive I get 40GB in 45 minutes, 36 seconds.

If I copy from the Win2008R2 machine to the iSCSI target (local drive to iSCSI target) I get 40GB in 37 minutes 56 seconds.

If I copy from the Win2003R2 machine to the iSCSI target via the Win2008R2 machine I get 40GB in 3 hours, 50 minutes, 24 seconds.

All copies were done via the following command issued on the Win2008R2 box:

XCOPY <source> <target> /J

XCOPY /J - Copies using unbuffered I/O. Recommended for very large files.

So, what's the bit I'm missing here? Why does a back-to-back copy take in total 1 hour, 23 minutes, 32 seconds when a "straight through" copy take almost 3 times as long?

Switches show no errors, network hovers around the 3% utilisation mark for the duration of the copy (whereas the "back-to-back" copies are around the 25% utilisation mark).

What have I missed?

Solution

Could it be the 'unbuffered' copy the problem? It is possible that Windows does some tricks that can speed up the copy the the source/target is a local disk, but it reverts to a safer behaviour if it is using two net devices.

I have played with disk testing in Unix, and the OSes can play lots of tricks with the disk subsystem. Good luck.

Comments

[MrGigu]

Are you using Jumbo Packets? Not that it should matter that much, but it will drastically reduce the amount of processing required by the 2k8R2 machine to forward the data to the iSCSI

[Rick]

No, Jumbo Frames are not in use. We initially saw this with Jumbo Frames turned on and thought that was the issue. So, we disabled them (restarted all, including the switch), brought it back to "plain vanilla" TCP and are still seeing this bizarre behaviour.

[Helvick]

Have you got hardware flow control enabled on the switch and server NICs? And are you 100% certain you're running GigE Full Duplex on both ends of the W2K8 box, you seem to be given the standard copy performance but it's worth checking.

[Rick]

I have just checked both - yes, hardware flow control enabled across all devices and also full duplex. What's curious is that A+B <> C (ie. if I copy from 2k3r2 to local + local to iSCSI it's way faster than 2k3r2 to iSCSI).

[xenny]

How are the NICs in the 2008R2 machine arranged? If they're both on a PCI bus, you'll have a performance hit simply because the bus isn't fast enough to handle the traffic. More generally, 40GB in 38 minutes is miserable for gigabit, it's only 18MB/sec, which isn't even twice as fast as 100BaseT. Either you're using very slow drives, or there's a performance problem you've not identified yet somewhere in the system.

[Rick]

Could be. We deliberately used the /J switch to remove any buffering - but we may have tripped over some other "undocumented" feature with this setting. When we didn't use the /J the copies actually took longer - but I don't have the actual figures handy on this.