How to enable multipath I/O for NFS-backed VMware ESXi storage?

Question

Right now I have 3 VMware ESXi hosts using 4Gb fibre channel to our NetApp for storage. I'd like to switch to NFS over 10Gb ethernet.

Each ESXi server has two 10Gb ethernet ports and each controller on my NetApp has two 10Gb ethernet ports. The only piece left that I need to get is the ethernet switches.

I'd like to have two ethernet switches for redundancy, so that if one switch dies, storage will still work, identical to the dual-switch fibre channel multipath I/O I have now.

But how do you do the same thing for NFS over ethernet? I know how to handle the ESXi side and the NetApp side of the equation, it's just with the switching side I don't quite know what to do.

I know how to do a LACP trunk/etherchannel bonding, but that doesn't work between physically separate switches.

So, can you recommend a pair of Cisco switches to use for this purpose and which Cisco IOS features I'd use to enable this kind of multipath NFS I/O? I'd like the switch to have at least 12 10Gb ports each. I know these switches will be mega-expensive, that's fine.

Solution

My firm just expanded our Cisco 4507 chassis switch by adding and another supervisor engine and 6-port 10GbE line cards to accommodate the storage network (VMWare and NexentaStor/ZFS). I know it's not the multiple-switch arrangement, but was a good way to get the number of ports we needed. Elsewhere in the industry, it seems as though Cisco Nexus and 4900M are popular for the solution you're requesting.

Comments

[DigiSage]

Would very much love to hear from someone that is actually using 10Gb ethernet NFS NetApp storage with ESXi in production....

[DigiSage]

I got a Cisco rep from CDW on the phone and we figured this out. The industry standard way of doing this, with Cisco anyway, is by taking advantage of VPC, "virtual port channel", available in the Nexus 5000 and 7000 series switches: cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/… It basically lets you do LACP across physically separate switches. The specific switch that seems to fit best here is the Nexus 5548. I now consider this question "answered".

[JamesRyan]

If you had to figure anything out then it is hardly an industry standard.

[DigiSage]

Thanks for the response, but this doesn't actually address multipath I/O over ethernet, as far as I can tell. I asked NetApp for info and they sent me details on the NetApp side of things, the ESXi side of things, but nothing about the switching in between.

[DigiSage]

Thanks for the switch recommendation, but that's just half the equation, I also need to know which IOS features / configuration to use to implement multipath I/O.

[user2751502]

This addresses accessing your Netapp filer over multiple paths. It's an active/passive, rather than active/active configuration, but it's still multipath. The vif configuration doesn't require any specific configuration on the switches; all of the logic is handled by the filers failover machinery. If you take out one of your switches, the filer would fail over to the second interface and continue providing service through the second switch.

[DigiSage]

Like I said in my original post, I know how to handle the netapp side of things and the ESXi side of things. It's the switching in the middle where I need help.

[user2751502]

And I think I've addressed that both in my answer and explicitly in my previous comment. But if you're unhappy with this answer, that's fine. I've got other things to work on.

[DigiSage]

Are you suggesting creating a vif on the netapp for two ports, and connecting one of each port to separate switches, with no special configuration on the switch, and that will somehow work?

[user2751502]

I am suggesting that you give it a shot. In theory, if you lose a switch, your ESXi hosts will ARP for the netapp and the netapp will respond over the second interface (through the remaining switch). I don't know if this will work in practice, but it seems as if it might. If you don't have the luxury of experimentation, Netapp support may be able to answer specific questions about this configuration...I won't have the luxury of a free filer to try this with until autumn.

[DigiSage]

I think the major problem with that will be (if it works) the time delay it takes for the arp request/receive to happen. That's why some switches support things like etherchannel across physically separate switches, but I think that's only in stacked switches. My ultimate goal in posting this question was to hear from someone saying "This is the industry standard way it's done, and how I implemented it: ", etc. I'd be happy to experiment, but I have to know what hardware to buy first, and I can't know that until I know exactly the best way of making this work, as not all switches will have it.

[user2751502]

If you're still in the market for switches, Cisco's 3750-X switches are (a) stackable and (b) allow you to create aggregate channels (LACP, etherchannel, etc) across the stacked switches. This document discusses exactly what you want, I think.

[DigiSage]

Thanks larsks, I've considered the 3750X and the problem is you can only get 2 10Gb ports per switch. I need, at a minimum, 4, preferably 8.