How can I setup PCI passthrough for my HighPoint 1144A USB 3.0 card under ESXi 5.1?

Question

I have an Apple Mac Pro (MacPro5,1) running VMware ESXi 5.1.0 (799733) with a HighPoint RocketU 1144A USB 3.0 PCI-express card installed. Given that VMware ESXi 5.1 doesn't support USB 3.0 for host-attached devices, I am trying to attach the USB 3.0 PCIe card to a virtual machine using DirectPath I/O PCI passthrough.

It took me a number of different configurations, but I finally managed to configure the HighPoint RocketU 1144A USB 3.0 PCI Express card for DitectPath passthrough:

The first few times I tried, after rebooting I still received the "The host needs to be rebooted to apply configuration changes" warning, even after rebooting the host. Finally I found that by moving the USB 3.0 card to the 3rd PCIe slot, I was able to get the passthrough configuration to stick.

I also note that the device shows up as a "ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller" when it actually should have 4xASM1042 chips on it. I also had to pass-through the PLX Technology, Inc PEX 8609 8-lane, 8-Port PCI Express Gen 2 Switch to make the configuration changes stick.

Now, however, I am stuck trying to attach the ASMedia Technology ASM1042 device to a FreeBSD (FreeNAS) virtual machine. When I add the device as a PCI passthrough device to the virtual machine, I am unable to power on the virtual machine:

VMware ESX unrecoverable error: (vcpu-0) PCIPassthruChangeIntrSettings: 09:00.1 failed to register interrupt (error code 195887105)

I tried passing through only the ASMedia ASM1042, only the PLX PEX 8609, and both the two devices. None of these three configurations worked.

Googling for that error code led me to this forum post in German which by using Google Translate I was able to extract the following information:

You have to disable MSI and forcing the card to get a INTx interrupt access, we go. Can you read here on page 5. http: / / www.vmware.com/pdf/vsp_4_vmdirectpath_host.pdf you following parameters have to manually add to your VM config: pciPassthru0.msiEnabled = "FALSE" I had to run the whole thing also virtualisert to ESXi and then I walked the closed system of ESXi but with the time on your nerves because you problems but not actually really engage. can I have now migrated to KVM (Proxmox VE) where the rich and Cine S2 V5.5 also successful at the next VM (with MSI). The whole runs super stable.

Following those instructions I added

pciPassthru0.msiEnabled = "FALSE" pciPassthru1.msiEnabled = "FALSE"

to my .vmx file, and this did enable me to boot the FreeBSD 8.3 (FreeNAS 8.3.1) VM... however the virtual machine hung at boot here:

I expect that I need to make some other change. The VMware PDF document they linked to describes "linked devices", and I expect there's some sort of linking going on whereby the 4xASM1042 chips (because the 4 ports on the USB 3.0 card each have their own channel) need to be linked through the switch... or something... but I am unsure of technically what is going on or how to proceed.

How can I resolve these problems and get passthru working for the HighPoint 1144A USB 3.0 card under EXSi 5.1?

Solution 1

Honestly, you're wayyyy outside of the norms for an ESXi server setup. I'm certain that this would not be a supported configuration in any way.

I understand that you're probably trying to get to an all-in-one (ZFS?) storage solution, but I doubt that you'll find a lot of mindshare on this type of arrangement.

The two things I can recommend are checking the VMware Hardware Compatibility List (HCL) for notes on the HighPoint USB card and also upgrading your ESXi to the current build, #914609.

Solution 2

It can work, as I've shown with screenshots from Nov 2011, using ESXi 5.0: http://TinkerTry.com/usb3passthru

Verify the checkboxes you've turned on, if you're still trying to get this going. Yeah, none of this is supported or on HCL lists, but it's passed through anyway, so the hypervisor doesn't really care, and the Highpoint 1444U should work.

I'm having pretty good luck with ESXi 5.5 as well (passthrough on 5.1 was poor).

I'll be posting more articles about this soon.

Comments

[wfaulk]

I am →this← close to voting to close so that no one will be tempted to ever try doing this ever again.

[Josh]

Care to explain @wfaulk? I seriously do believe that USB 3.0 support to a VM isn't that unreasonable of an expectation!

[wfaulk]

@Josh: I have never seen PCI passthrough actually work; it would be way simpler to just have SAS or SATA or even eSATA storage than argue with USB3; a file server using removable storage seems like a horrible idea; I would never suggest running a file server as a VM; I have no idea why you would buy a Mac Pro and not run MacOS on it (though I am surprised to find that it is actually on VMware's HCL). I'm sure I could come up with more reasons, but do I really need to?

[Josh]

We are running Mac OS X in a VM. We did this for budget reasons, we couldn't afford two servers, and the USB3.0 drives are from an earlier system which I am trying to gain access to. Also, one reason to use them: Offsite backups.

[Josh]

@ewwhite Sadly the newest version of ESXi still doesn't work. I saw screen captures others posted of this card in VMware ESXi and it shows as 4 USB controllers where mine only shows as 1. I am about to give up on this and just use USB2 for backup.

[Admin]

This is a terrible idea but a great question. Good luck and godspeed!

[Josh]

Yes, what I am trying to do is run a ZFS machine with access to USB 3.0 devices...

[Josh]

Oh, great, I should try with ESXi5.5 then! Thanks!