NIC Teaming with Broadcom on HyperV 2008 R2

Question

I have a HyperV cluster made up of 3 hosts. Each host is connected to both of my Nexus 5548 switches running in an etherchannel. LACP on the switch and NIC teaming using Broadcom 802.3ad on the server side. This gives me 2GB of bandwidth and also provides fault tolerance.

The problem I am having occurs when I perform a live migration. Before the live migration both Nexus switches show the MAC of the VM in the ARP table. After the migration one switch shows the MAC of the VM and the other shows the MAC of the HyperV host which it moved to.

I ran a packet capture and saw the HyperV host send a gratuitous ARP with the IP of the VM and the MAC of the host instead of the MAC of the VM. I lose layer 3 connectivity when this happens. I have to manually clear the ARP entry from the switch or wait about 7 minutes for it to correct itself.

I did some looking around and people are having similar issues when dealing with NIC teaming using Broadcom. Has anyone seen this? Any advice?

-------- Edit added below

I am only having this problem when teaming using Link Aggregation 802.3ad. The Broadcom teaming options are...

• Link Aggregation (802.3ad)
  
• Smart Load Balancing (TM) and Failover
  
• SLB (Auto-Fallback Disable
  
• Generic Trunking (FEC/GEC) / 802.3ad-Draft Static

I switched to Smart Load Balancing and the VM Live Migrates without losing any network connectivity. However, the ARP tables on the Nexus switches are in sync but they show the MAC address of the Host and not the VM. This is opposite of what I thought it would do. Shouldn't the ARP tables of the switches show the MAC of the VM? If not and they are suppose to show the MAC of the host, why?

Solution 1

Okay. After 3 weeks of intense battle I finally got it all figured out.

I opened a case with Broadcom support and after going back and forth for a few days here is the response I received from a Broadcom software developer.

---

Seeing a (vm_ipv4, host_mac_addr) or (vm_ipv6, host_mac_addr) pair on a remote station or switch is okay for many networks. Basp will handle the muxing of the addresses and make sure the vm's get the correct packets.

However, a few network configurations require the ip address to map to the vm_mac_addr. If you want to use the vm_mac_addr instead of the host_mac_addr, you need to add the HyperVMode registry key listed below.

If it is desired not to mux the vm's mac address, that is to not replace it with a mac address of a nic, you need to set the following registry entry on the host the vm is running on and reboot the system for it to take effect. Setting this registry entry results in a less efficient operation by basp, but is necessary in certain setups, such as when there is a dhcp server that requires the vm's mac address be used in order to assign an ip address. By setting this registry entry, you should also see the actual vm's mac address in the arp table of all the remote stations the vm is connected to.

This registy entry can be added after a team has been created, but before the hyper-v virtual adapter using the team has been created. A value of 1 will configures basp to use a vm's mac address instead of the mac address of a nic in a slb team when load balancing flows. A value of 0, or not present, configures basp to mux/demux the vm's mac address with a mac address of one of the nics in the team.

Set the "1" in the registry path to the correct team number that this mode is to be applied to and reboot the system. Note that this registry entry will be deleted when the team is deleted, and must be recreated each time the team is created. Future versions of BACS will have a check-box that will set this registry entry and prevent the need for adding it when a team is created and having to reboot the system. This registry entry is only available on >=basp6-1.5.1.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ Blfp\Parameters\1] "HyperVMode"=dword:00000001

PS. Besides this registry entry being only available on >=basp6-1.5.1, it is only applicable to slb team types, and does not apply to lacp or gec teams.

---

So with that said I gave it a try. First, I applied created the registry key. Then, I changed the Virtual Switch to Private Virtual Machine Network mode. I gave the server a quick reboot. Finally, when the server came back online I configured the Virtual Switch back to External Mode and choose the BASP Virtual Adapter. I tested Live Migrations and everything worked perfectly and the ARP tables in the Nexus showed the VM_IP and the VM-MAC. YEY!!!!

Solution 2

Here is the setting within SCVMM. I was having issues like you until I turned on Trunk mode for the Host. If you right click on the host in SCVMM --> Properties --> Networking tab

Comments

[Peter Lindqvist]

Interesting question, sounds like a bug, but who is responsible for it?

[Greg Askew]

Can you provide your nic model number and driver version?

[tony roth]

Personally I won't team with Broadcom nics/drivers. I know that doesn't help but the intels have a feature set that matches up better with hyper-v.

[Mike]

I have a 3 node cluster setup with Broadcom teaming. I was having a similar issue until I setup the host with VLAN trunking enabled within SCVMM. The VM was fine until I migrated then it would go offline indefinitely. Unfortunately I am not able to look at my install right now. I can get you the info in the morning

[Ruisu]

The model number of my NIC is 5709C and the BASP is driver version 1.5.3.0. I am using BACS 4 to manage the teams.

[Philip]

What firmware version are you running? The BCM5709 had "issues" prior to a Dec 2010 update.

[Philip]

@tonyroth What features specifically from Intel NICs match better with Hyper-V?

[TomTom]

I hate that. TIME FOR SERVER 8 - nic teaming on OS level ;) Just installed forst test server today (a spare for running high perofrmance stuff for testing, and we decided to test hyper-v 3 on it).

[SpacemanSpiff]

Interesting, he didn't even mention using VLANs?

[Ruisu]

I am using VLANs. My setting was already set to Trunk mode but I was missing a couple of VLANs on there which I corrected. I still have the same issue but only when teaming using 802.3ad. I'll describe more by editing my question.

[SpacemanSpiff]

oooooh upvote upvote upvote... I could see this burning a lot of people.