KVM virtual machine unable to access internet

Question

I have KVM set up to run a virtual machine (Windows Home Server 2011 acting as a build agent) on a dedicated server (CentOS 6.3). Recently, I ran updates on the host, and the virtual machine is now unable to connect to the internet.

The virtual network is running through NAT, the host has an interface (eth0:0) set up with a static IP (virt-manager shows the network and its IP correctly), and all connections to that IP should be sent to the guest.

The host and guest can ping one another, but the guest cannot ping anything above the host, nor can I ping the guest from anywhere else (I can ping the host). Results from the guest to another server under my control and from an external system to the guest both return "Destination port unreachable". Running tcpdump on the host and destination shows the host replying to the ping, but the destination never sees it (it doesn't even look like the host is bothering to send it on at all, which leads me to suspect iptables). The ping output matches that, listing replies from 192.168.100.1.

The guest can resolve DNS, however, which I find rather odd. The guest's network settings (connection TCP/IPv4 properties) are set up with a static local IP (192.168.100.128), mask of 255.255.255.0, and gateway and DNS at 192.168.100.1.

When originally setting up the vm/net, I had set up some iptables rules to enable bridging, but after my hosting company complained about the bridge, I set up a new virtual net using NAT and believe I removed all the rules.

The VM's network was working perfectly fine for the last few months, until yesterday. I haven't heard anything from the hosting company, didn't change anything on the guest, so as far as I know, nothing else has changed (unfortunately the list of packages updated has since fallen off scrollback and I didn't note it down).

Update:

iptables -L:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere            udp dpt:domain 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:domain 
ACCEPT     udp  --  anywhere             anywhere            udp dpt:bootps 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:bootps 
ACCEPT     tcp  --  main-domain          anywhere            tcp dpt:mysql 

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             192.168.100.0/24    state RELATED,ESTABLISHED 
ACCEPT     all  --  192.168.100.0/24     anywhere            
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere            reject-with icmp-port-unreachable 
REJECT     all  --  anywhere             anywhere            reject-with icmp-port-unreachable 
ACCEPT     all  --  anywhere             192.168.100.128     
ACCEPT     all  --  anywhere             guest-subdomain

iptables -t nat -L:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DNAT       all  --  anywhere             guest-subdomain to:192.168.100.128 

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  tcp  --  192.168.100.0/24    !192.168.100.0/24    masq ports: 1024-65535 
MASQUERADE  udp  --  192.168.100.0/24    !192.168.100.0/24    masq ports: 1024-65535 
MASQUERADE  all  --  192.168.100.0/24    !192.168.100.0/24    
SNAT       all  --  192.168.100.128      anywhere            to:guest-ip 

Solution 1

These lines in the FORWARD chain:

ACCEPT     all  --  anywhere             192.168.100.128     
ACCEPT     all  --  anywhere             guest-subdomain

should be before the reject rules. Such that:

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             192.168.100.128     
ACCEPT     all  --  anywhere             guest-subdomain
ACCEPT     all  --  anywhere             192.168.100.0/24    state RELATED,ESTABLISHED 
ACCEPT     all  --  192.168.100.0/24     anywhere            
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere            reject-with icmp-port-unreachable 
REJECT     all  --  anywhere             anywhere            reject-with icmp-port-unreachable 

You can delete them and use the following to insert the rules at the top.

iptables -I FORWARD -d guest-subdomain -j ACCEPT
iptables -I FORWARD -d 192.168.100.128 -j ACCEPT

Also the SNAT rule is redundant since you already have MASQUERADE rules.

Solution 2

Restarting libvirtd may help..I faced same issue once and just restarted libvirtd. iptables rules corresponding to libvirtd are sets again and I can ping outside world.

sudo service libvirtd restart

Comments

[Admin]

It does sound a bit like an iptables problem, although you said you can perform DNS lookups from the guest. What is the configured IP of the DNS server on the guest? From the host, can you share with us the itpables configuration? Also see davychiu.com/blog/11-nat-with-iptables-and-kvm.html and rcritical.blogspot.co.uk/2011/01/… and ubuntuforums.org/showthread.php?t=1529341

[Admin]

@javano The guest is using the host for DNS (guest settings have been in the question, hidden in paragraph 4), so that part makes good sense. Posted the iptables conf, which is very similar to the recommended settings in your links.

[Admin]

@javano Results of a ping with tcpdump are already in the question. Haven't run other protocols yet.

[ssube]

The lines given didn't seem to add the rules at the top (possibly my mistake), but after removing and recreating the REJECT rules, things appear to be working.

[kumar]

I faced same issue first time - I just restart libvirtd in host - and everything worked.

[Jones G]

this worked for me!