pfSense with Mellanox ConnectX-2 10GBit NICs

Question

I recently picked up two Mellanox ConnectX-2 10GBit NICs for dirt cheap. I'm trying to get these to be recognized by a pfSense box. The BIOS recognizes the NICs, no problem whatsoever. However, pfSense definitely doesn't. It's not showing up at all.

I've heard that I can install the driver myself or at least throw some kind of magical peanut butter some place to make it work out. Sadly, the only evidence of this I have is from this post about FreeNAS. This wasn't applicable to my install though. I also managed to find a post on NAS4Free, but I could not find the ISO mentioned in the post, nor what version of BSD it was based off of. Maybe I missed out on something on the NAS4Free site, but it's not there.

I've tried a few snapshots and the current 2.2.6 official release. No luck.

Can anyone point me in the right direction with this?

Solution 1

Alright. I spent a little while figuring this out.

It turns out that Mellanox has basically dropped all support for this NIC. However, not all is lost. You can still use the legacy versions for ConnectX-3 to actually utilize the NIC (or so says the release notes for the driver I downloaded). You can find the driver webpage here, and the actual driver download right here. It's under ConnectX-3 -> Archive Versions -> 2.1 -> FreeBSD -> All -> All on the actual driver site.

Fortunately it comes with instructions on how to install the driver. Since pfSense doesn't come with suitable environment for compiling, you'll have to set up a separate machine to compile. I installed FreeBSD 10.3 (with source code; it's an option in the installer) in a virtual machine, which is the same release that my version of pfSense is using. Be sure to use the same release of FreeBSD as the one used in your pfSense box. It might cause problems otherwise.

I installed gcc using pkg install gcc. Unfortunately, at this point, I wasn't able to actually use said drivers I downloaded from the Mellanox site. However, FreeBSD does still have support for the drivers, they're just not enabled by default. If you find you get an error when building the drivers, you can find the two modules you need to compile in /usr/src/sys/modules. There will be two directories of interest, mlxen and mlx4. There will be a Makefile in both of these directories. Do the usual make && make install procedure that you'd do with any other driver. It'll drop the modules you need in /boot/kernel. The module names are mlx4.ko and mlxen.ko. Copy both of these modules to your pfSense box (into the same directory: /boot/kernel). Afterwards, you can add these two lines to your /boot/loader.conf file:

mlx4_load="YES"
mlxen_load="YES"

Save the file and restart your pfSense machine.

For me, the NIC now shows up and allows itself to be configured as a LAN interface.

Solution 2

Edit & FreeNAS-9.10-STABLE-201605240427 Note!

Recently I upgraded to latest FreeNAS-9.10-STABLE-201605240427 build and upon restart got a kernel panic referencing mlxen module. My [limited knowledge] guess is that mlx driver is now included as part of that FreeBSD build and so, these steps are no longer needed.

So, if you are stuck with a kernel panic:

1. Reboot and at GRUB prompt press 'e' to edit boot settings
2. This will bring up a simple editor, find lines referencing mlx driver loading and comment them out with a #.
3. Press F10 to continue booting
4. Remove module loader statements from System Tunables.

---

@Lildirt's solution works on FreeNAS too (FreeNAS-9.10-STABLE-201605021851). Short story long, I picked up a two-card pack for $50 and a direct attach cable so that I could do iSCSI between FreeNAS and my other server running ESXi. ESXi recognized the card no problem; much like pfsense in OP's situation, FreeBSD did not know what it is.

My steps differed in that:

1. Created a FreeBSD jail to do work in
2. Installed gcc (pkg install gcc)
3. Downloaded the tar mentioned (wget http://www.mellanox.com/downloads/Drivers/MLNX_OFED_FREEBSD_V2.1.6.tar.gz)
4. Untar
5. Navigate to directory containing the makefiles (each time for mlx4 and mlxen)
6. make && make install this will install into /boot/kernel with respect to jail's root (so in my case, my actual path was /mnt/zfspoolname/jails/mytempjail/boot/kernel) (also each time for both modules)
7. Now copy both .ko files (modules) to FreeNAS's /boot/kernel (remember to exit the jail, duh)
8. Add the module loader statements via FreeNAS web ui (System -> Tunables -> Add Tunable); This essentially adds them to /boot/loader.conf.local
9. Reboot FreeNAS
10. You should now be able to add a new network interface, mine was named mlxen0