Physical switch not learning MAC address of virtual machines

Question

I have a virtual environment running vSphere 5.1. I have two ESXis with a vDS between them. There is, for the purposes of the discussion, just one portgroup on that vDS configured to use a VLAN. The uplink of each host is made of two teamed NICs, which are both connected to the same physical switch. On another port of the physical switch there is a firewall with one leg in the same VLAN as the portgroup.

I have observed that occasionally, when creating and starting a new VM on host1, the switch will not learn this VM's MAC address (even if I force a gratuitous ARP using arping). When this happens, the VM will not be able to communicate with the firewall, or even with other VMs (that use the same portgroup) on host2. However, it can still communicate with other VMs on host1 that are connected to the same portgroup.

Furthermore, if there is a VM on host1 that is working fine and can communicate externally, and I vMotion it to host2, then the switch will not learn that the MAC address has moved to a different port (again, even after I issue arping from the VM), and as a result this VM will lose connectivity to the firewall.

To add to the confusion, and this is just driving me crazy, if I run a "ping -t " from the firewall, while a VM is starting up, then the switch will learn the VM's MAC address and the VM will be able to communicate.

The "notify switches" option is turned on both at the portgroup and the uplink.

What could I be missing here? I am willing to consider crazy ideas. Tomorrow I will try rebooting the hosts just in case the vDS configuration is out of sync...

Solution 1

For what it's worth, this turned out to be a physical switch issue. There's a very long thread at https://communities.vmware.com/thread/421560 that proved immensely helpful (if only by showing us that we were not alone in this...).

Thank you all for the help!

Solution 2

My advice to you is to upgrade your ESXi installations. Things like this sometimes fall into the bug category, and according to VMware, your build number (838463) is from October 2012. The current release is build 1312873.

It makes sense to patch if you can, since you're working with a very old build.

Also see: Are VMware ESXi 5 patches cumulative?

Comments

[ewwhite]

Please provide details about the hosts, ESXi patch level and switching equipment involved...

[telewin]

@ewwhite Hosts are running ESXi 5.1 build 838463. I'll have to get back to you about the switching equipment. Are you implying this could be a hardware issue in the switch?

[ewwhite]

No, just need to know the switch make/model to help troubleshoot. Cisco != HP Procuve != Linksys.

[telewin]

@ewwhite Definitely Cisco, this much I can say. Did you have specific troubleshooting steps in mind? I'd be happy to try them tomorrow...

[ravi yarlagadda]

I'm inclined to think this is some kind of problem with the switch, actually... sounds like packets are going though the port without the MAC being learned, which would definitely be a problem with the switch. Can you check whether that's happening?

[telewin]

@ShaneMadden I agree - the switch can definitely be at fault here. I am not sure, though, how I can isolated the problem to the switch and rule out the vDS. It seems that the only way to do this is sniff the traffic on the switch itself, no?

[ravi yarlagadda]

@telewin Awesome, glad you tracked it down!