10Gbps iSCSI network transferring at only 2Gbps

Question

I have a performance issue with my SAN. I have a full 10Gbps private network setup. I have the following setup

1 - Mac with Connected with via Fiber Transciever 10GbE
1 - Windows 7 PC with 10GbE
1 - HP Switch with All ports are 10GbE
2 - Quantum StorNext Servers with 10GbE
1 - Dell Compellent Solution with 2 Controllers connected to the network at 10GbE
All servers, switches, and computers have staticly assigned IPs

Compellent Config
Tier 1 - 10K SAS Drives in RAID 10
Tier 2 - 7200 SAS Drives in RAID6 (12 groups of 13 Drives in Each)

Additional info
Windows 7 Client TCP Offload Options
IPv4 Checksum Offload - Enabled
TCP Checksum Offload - Enabled
UDP Checksum Offload - Enabled
Large Send Offload - Enable
Jumbo Packet - 9014Bytes Enabled

I mounted the StorNext volume on my Windows 7 PC and my Lion Workstation. Unfortunately all my transfer speeds are at around 2Gbps or 2.8Gbps (if I'm really lucky). I was looking to get at least 5Gbps speeds out of this setup, but I'm averaging about 2Gbps or a little above 250MBps transfer rates on file copying. When I map a LUN directly to the boxes and then format it natively with either HFS (with journaling) on the MAC or NTFS on the Windows 7 PC, and then copy a file, I get about 180MBps. So my performance on a directly mapped LUN is slower than my StorNext Volume. Any suggestions? Has anyone seen degraded performance on iSCSI with 10GbE? Any help would be Awesome! Thanks!

Solution 1

1.) Jumbos -might- help if you're seeing a lot of processor load for interrupt traffic but if TCP is operating correctly it should be able to ramp well past 2G on a 10G link. I've seen plenty of 10GE links running above 90% without jumbos enabled.

2.) If you do use jumbos, enable the same size on every NIC and every switchport in the VLAN and/or broadcast domain. PMTU works when packets cross routers and mixing MTU values within the same network will lead to nothing but misery.

3.) I'm not particularly familiar with the Procurve gear but TCP traffic can be tricky at high speeds if there are any questions about buffer availability. I've seen other testing where this has manifested (without apparent TCP drops) as a huge cut in performance that ended up being fixed by actually reducing buffer sizes.

4.) Make sure that the actual TCP settings (1323, SACK, etc) are all configured consistently. The operating systems in question should be fine out of the box but I don't know much about the storage node. It might be worth digging into - either in terms of settings on the device or via a protocol trace (wireshark or tcpdump) to observe window sizing and any retransmissions going on.

5.) Try eliminating as many variables as you can - even getting down to a crossover cable between one of your storage nodes and a single workstation - to further isolate the issue. Don't be afraid to disable some of the offloads you mentioned as well, as they've been known to cause issues from time to time.

Solution 2

Not the answer you'll be looking for, but our Compellent SAN has been nothing but a disappointment performance wise. I too had similar performance numbers (Before I get jumped on... this is on a dedicated 10G iSCSI switch using Jumbo frames, and everything was validated by both CoPilot and Dell's "Tiger Team". Our other SANS (EMC and many Equallogics) have comparably blistering performance). What got us out of this particular performance bottleneck was replacing a faulty controller.

Have you had co-pilot look over the configuration? If properly set up, the SAN will be reporting home anyway, and they may already have it flagged.

What interface speeds do you have on those client machines? Any interesting network setups? Up to date drivers? What does CPU/Memory look like in Resource Monitor while doing a transfer test?

Solution 3

If you're using 1500 MTU, piece of advice #1, use jumbo frames/packets at ~9000. It does seem you have Jumbo packets enabled to some extent (based on "Jumbo Packet - 9014Bytes Enabled") so it is also possible you're dealing with packets getting fragmented as well. Make sure everything along your path is set to allow the 9014 byte packets.

Comments

[Christopher Perrin]

This is probably a dumb question but is your storage system capable of transfering more than 2.8 Gbps?

[Mike Pennington]

So many possibilities... have you looked at CPU utilization? MTU size? TCP Offload settings? Packet loss? NIC driver issues?

[SpacemanSpiff]

What are the type (SAS/SATA), size, rotational speed of the drives? how many of them are there and what RAID configuration are you running them in?

[cohortq]

I have a Tier 1 Filled with 5TB of 10K SAS spinning drives in RAID 10. My Tier 2 is all 7200rpm drives in 12 groups of 13 drives in each All in RAID6. I haven't measured packet loss. My MTU Sizes are the standard 1500bytes. CPU Utilization is really low. CPU on my StorNext servers are really low, like around 15%. My Compellent Doesn't give me CPU metrics, but I'm hitting a max of 2,000 IOPs with IOMeter. The Max speed we were able to hit briefly with IOMeter was 600MBps, but that wasn't fully sustainable. Not sure what my TCP Offload settings are either.

[David Schwartz]

I think you're hitting packet per second limits. A larger MTU will likely help. (Is your MTU 1500 or 9014?)

[SpacemanSpiff]

What kind of switching, is it configured for jumbo frames as well? How about flow control?

[cohortq]

I don't have my switches configured for Jumbo Frames. They're pretty much just configured with the IP's for SAN segment they are on. Flow Control is still default as well. I'm not sure my Compellent is configured for Jumbo frames either.

[rnxrx]

MTU / PPS limits will tend to manifest in terms of excessive interrupt counts and attendant high CPU, as will a large amount of TCP overhead. Jumbos are certainly a good thing, but shouldn't make this large of a difference. Try to confirm performance with something other than iSCSI (iperf or similar) to verify the underlying network.

[Michuelnik]

What exactly helped from the accepted answer?

[cohortq]

Really I tried everything on the accepted answer. JUMBO frames was what I thought was going to make the biggest difference, but really I've never been able to get past the 2Gbps limit. Only on IOMeter with 22 workers running for a multi-threaded write am I able to get above 8Gbps. But I know nothing that can do multi-threaded filesystem writes.

[cohortq]

I have everything connected on a separate 10GbE network. I have 2 Controllers and they are each connected to a dedicated HP ProCurve on 10GbE. How was your Compellent configured? What do you think was the main bottleneck in performance after the Controller replacement?