Connect two infiniband cards to each other without a switch

Question

Is it possible to connect two Mellanox ConnectX-2 cards on two seperate computers to each other without a switch in between? I am just trying to learn to use verbs programming without having to spend $ on an expensive switch.

Do I need a special cable or do the standards one work ok? On standard ethernet I would need a cross connect cable.

Solution

Yes, you can directly connect them. But you must ensure that on at least one of the machines you run the opensm subnet manager.

From the software side, if you were using Ubuntu 14.04, install these packages:

sudo apt-get install opensm infiniband-diags librdmacm-dev libibverbs-dev libmlx4-dev

Then add these to /etc/modules

mlx4_ib
ip_ipoib
rdma_ucm

Add an IP address to the card. edit /etc/network/interfaces

auto ib0
iface ib0 inet static
  address 10.0.0.1
  netmask 255.255.255.0

Reboot.

Opensm will automatically start. Repeat the above for the other machine but use a different IP: 10.0.0.2.

All going well you should be able to ping 10.0.0.2 from 10.0.0.1 and vice versa.

Run some tests:

sudo ibnodes

Oh, there is a bug in scripts for ibnodes and several others. You can fix them by editing them. Mostly they are shell scripts that point to /usr/local/sbin when they are actually calling apps in /usr/sbin.

If you find one, go right ahead and edit it and change IBPATH accordingly. I have never gotten around to filing a bug report against Ubuntu for that. Hopefully someone else will do it. It's been a problem for ages.

Output will look something like:

Ca  : 0x001a4bffff0c9374 ports 2 "HP Lion Cub DDR 128MB"
Ca  : 0x001a4bffff0c446c ports 2 "HP Lion Cub DDR 128MB"
Ca  : 0x001a4bffff0c4438 ports 2 "HP Lion Cub DDR 128MB"
Switch  : 0x000b8cffff006aa8 ports 24 "MT47396 Infiniscale-III Mellanox Technologies" base port 0 lid 2 lmc 0

Comments

[hookenz]

No. Because it varies depending on the OS. I can give some instructions from Ubuntu.

[Ivan]

That would be great. CentOS is probably what I will use, but following the ubuntu instructions might get me in the ballpark.

[hookenz]

Added Ubuntu instructions. I think CentOS should be similar but I'm not sure what the package names are.

[Ivan]

Great thanks. BTW, does opensm imply ipoib? Or would this still be a raw infiniband ip addresses for verbs programming (or RDMA library abstraction)?

[hookenz]

The module ib_ipoib is what you need loaded in /etc/modules (ubuntu) to enable ipoib. Generally you'll want to use it. It makes it much easier for node location. If you also want good examples of verbs programming see here: thegeekinthecorner.wordpress.com/category/infiniband-verbs-r‌​dma

[Ivan]

I see. Hmmm, I am confused, because I thought that IPoIB was a protocol that you could not use rdma verbs with, just standard TCP style programming? My confusion. Thanks re:gitc, I am aware of it. One of the few sites that has examples. Wish there were more...

[hookenz]

You can use them side by side. ipoib uses the verbs interface. Think of it more as IP over Infiniband verbs. Where as IP in traditional networking could be considered as IP over ethernet.

[Ivan]

Gotcha, so I assume thte greek examples would work with your configuration above?

[Ivan]

Thanks. I think that all that is needed to run a "software switch" is opensm so if everything else comes up it should be relatively easy to set up opensm I would think.