VMWare ESXi virtual machine can contact the gateway but not the DNS server

Question

I am having a bit of a strange issue. I have a VMWare ESXi server with two virtual machines running on it. They are running just fine and can communicate on the network without a problem.

I am now trying to add a third. I am installing Ubuntu 8.04 Server. I assign it a static IP address and it's a fresh installation. Once installed I can ping the gateway but I cannot ping the DNS server. It's on the same network with the other two VMs which are communicating just fine. I have tried to reinstall the operating system but it still fails to connect.

Here is /etc/network/interfaces

auto eth0
iface eth0 inet static
    address 192.168.1.23
    netmask 255.255.255.0
    network 192.168.1.0
    broadcast 192.168.1.255
    gateway 192.168.1.1
    dns-nameservers 208.67.222.222 #opendns
    dns-search mydomain.com 

Here is route

Destination | Gateway     | Genmask       | Flags | Metric | Ref | Use | Iface
localnet    | *           | 255.255.255.0 | U     | 0      | 0   | 0   | eth0
default     | 192.168.1.1 | 0.0.0.0       | UG    | 100    | 0   | 0   | eth0

Since I'm running this behind a FortiGate this is what the sniff command gives me when I try to ping 208.67.222.222

arp who-has 192.168.1.1 tell 192.168.1.23
arp reply 192.168.1.1 is-at MAC
192.168.1.23 -> 208.67.222.222: icmp: echo request
192.168.1.23 -> 208.67.222.222: icmp: echo request
192.168.1.23 -> 208.67.222.222: icmp: echo request
192.168.1.23 -> 208.67.222.222: icmp: echo request
192.168.1.23 -> 208.67.222.222: icmp: echo request

As you can see it looks like I never get a response. One interesting thing I notice is the arp reply's MAC doesn't look right. I have cleared the FortiGate's ARP cache though and checked the entry and it seems correct. The MAC it lists is the one for the router. However if I ping from a different virtual machine that is also Ubuntu 8.04 with a nearly identical configuration I get this.

192.168.1.22 -> 208.67.222.222: icmp: echo request
208.67.222.222 -> 192.168.1.22: icmp: echo reply
192.168.1.22 -> 208.67.222.222: icmp: echo request
208.67.222.222 -> 192.168.1.22: icmp: echo reply
192.168.1.22 -> 208.67.222.222: icmp: echo request
208.67.222.222 -> 192.168.1.22: icmp: echo reply

So, what could I be missing?

Thanks.

Solution

I found the problem. The issue was how the SSL VPN was setup. I had configured the VPN to use the same IP range. So for example I had configured the regular local network to be

192.168.1.1

Then I configured the SSL to have the range of

192.168.1.[100-105]

When that happened it also put in a default gateway of 0.0.0.0 for the ssl vpn. So in order to fix my problem I assigned the SSL VPN to use this

192.168.2.[100-105]

Then I changed the default gateway for that to be 192.168.2.1. Everything worked fine after that.

Comments

[Nathan Palmer]

Watching the external interface I see nothing pinging from .23. Pinging from .22 I see the expected result.

[joeqwerty]

Have you checked the ARP cache on the switch, router, and firewall? Can you ping from the router or firewall to .23? Is there a rule on the firewall that's blocking outbound traffic from .23?

[Nathan Palmer]

The ARP cache looks ok. I have even tried to clear it. I cannot ping from the router to .23. The rules are the same for both .20 and .23 and there is one rule that is switch -> wan1 for external traffic. I don't have a wan1 -> wan1 rule however.

[Nathan Palmer]

Ok. This is strange. I swapped the machines back so .20 had it's original IP back. But then I gave the other machine .24 and it works. Then I realized. All the VMs that are working have even IPs. .10, .20, .22 and .24. Any ideas?

[joeqwerty]

Is it a managed switch? If so, are there any ACL's on the switch that may be causing the problem? Whether the ip address is even or odd shouldn't make any difference. Since you can't ping .23 from the router try tracing the route from the router to .23 to see what path it's trying to take.

[Helvick]

Odd/even issues like this may point to a problem with NIC Teaming on the ESXi host - do you have multiple nics on the VM Port Group\vSwitch?

[Justin]

I don't think the problem is with NIC Teaming in this case. From the sniff on the Fortigate we know that: the icmp packet is making it from the VM to the Fortigate. The icmp packet is NOT making it from the internal interface on the Fortigate to the external Interface. I believe this points to a problem with the rules in the Fortigate.

[Nathan Palmer]

It is using 1 port on the NIC. I am currently not Teaming. There is 1 rule that goes from the internal network to the outside and it's a NAT'd all->all. How does this describe the even/odd issue?