Openstack, neutron, Can't ping external network
12,573
I know it is late... but let me try to answer. Not being able to ping the external network might be related to the virtualization and promiscuous mode but I think that step should not work. I will try to explain why. For a very good explanation on how the networking is setup you can check out this presentation: https://www.hastexo.com/system/files/neutron_packet_flows-notes-handout.pdf
You have multiple parallel network spaces involved here.
First you have the global or main routing table of the host. In the case of the network node you have 3 ethernet interface, two with and IP address and one without.
Second you have your external network. This is not configured on your network host. If you would run
netstat -rn
you would not see any route towards your external network. This means that when you send you ICMP packets to the external gateway they would go the default gateway set on the network node and not be answered. This network lives in the neutron router's network namespace.
Third you have the tenant network which also does not live in your main network space but in the same neutron router namespace.
To exemplify I will show you how it looks in my setup. I have an all in one node that is running inside a Virtualbox machine and it is running Ubuntu Trusty. My interfaces are eth0 for admin (also my default gateway connected to a NAT network in Virtualbox), eth1 for the tunnels, eth2 is my external net (also attached to a NAT network with the range 172.16.100.0/24).
Here is how the routing table of the Ubuntu VM looks like:
root@columbo:~# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.0.1.1 0.0.0.0 UG 0 0 0 eth0
10.0.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
192.168.56.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
So no sign of my external net (172.16.100.1). If I try to ping it I get nothing:
ping 172.16.100.1
PING 172.16.100.1 (172.16.100.1) 56(84) bytes of data.
^C
--- 172.16.100.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2004ms
Now you can see what network namespaces you have using the command ip netns list
root@columbo:~# ip netns list
qrouter-e53979a8-8bab-4da5-9b57-58dba6d5db7b
qdhcp-120a6fde-7e2d-4856-90ee-5609a5f3035f
qdhcp-b7ab2080-a71a-44f6-9f66-fde526bb73d3
Inside a namespace you can run most of your normal commands - see bellow
root@columbo:~# ip netns exec qrouter-e53979a8-8bab-4da5-9b57-58dba6d5db7b netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 172.16.100.1 0.0.0.0 UG 0 0 0 qg-c69702a9-ae
10.255.1.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-e706aba1-46
172.16.100.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-c69702a9-ae
and
root@columbo:~# ip netns exec qrouter-e53979a8-8bab-4da5-9b57-58dba6d5db7b arp -an
? (10.255.1.12) at fa:16:3e:40:50:36 [ether] on qr-e706aba1-46
? (172.16.100.1) at 52:54:00:12:35:00 [ether] on qg-c69702a9-ae
? (10.255.1.14) at fa:16:3e:75:5f:b4 [ether] on qr-e706aba1-46
and now ping will work (inside the namespace)
root@columbo:~# ip netns exec qrouter-e53979a8-8bab-4da5-9b57-58dba6d5db7b ping 172.16.100.1
PING 172.16.100.1 (172.16.100.1) 56(84) bytes of data.
64 bytes from 172.16.100.1: icmp_seq=1 ttl=255 time=2.55 ms
64 bytes from 172.16.100.1: icmp_seq=2 ttl=255 time=0.555 ms
^C
--- 172.16.100.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.555/1.554/2.553/0.999 ms
I can also ping the router which has the ip 172.16.100.50:
root@columbo:~# ip netns exec qrouter-e53979a8-8bab-4da5-9b57-58dba6d5db7b ping 172.16.100.50
PING 172.16.100.50 (172.16.100.50) 56(84) bytes of data.
64 bytes from 172.16.100.50: icmp_seq=1 ttl=64 time=0.065 ms
64 bytes from 172.16.100.50: icmp_seq=2 ttl=64 time=0.065 ms
^C
--- 172.16.100.50 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.065/0.065/0.065/0.000 ms
Same for my tenant network
root@columbo:~# neutron net-list
+--------------------------------------+---------------+------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------------+------------------------------------------------------+
| 120a6fde-7e2d-4856-90ee-5609a5f3035f | SecondVlan | 5432f1c9-0bb6-4619-b897-65d301071f72 5.5.5.0/25 |
| f2597437-a005-44ad-9ce2-168fbc331e56 | outside_world | 3fe35e71-53d7-4432-8c82-a06856b79316 172.16.100.0/24 |
| b7ab2080-a71a-44f6-9f66-fde526bb73d3 | SERVER_VLAN_1 | 87d769f1-5cf3-48cf-8741-44a01479ff3e 10.255.1.0/24 |
+--------------------------------------+---------------+------------------------------------------------------+
the ping of the tenant router in the SERVER vlan
root@columbo:~# ip netns exec qrouter-e53979a8-8bab-4da5-9b57-58dba6d5db7b ping 10.255.1.1
PING 10.255.1.1 (10.255.1.1) 56(84) bytes of data.
64 bytes from 10.255.1.1: icmp_seq=1 ttl=64 time=0.050 ms
64 bytes from 10.255.1.1: icmp_seq=2 ttl=64 time=0.064 ms
^C
--- 10.255.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.050/0.057/0.064/0.007 ms
I hope it is useful for somebody and not totally flawed.
Author by
jazzsir
Updated on June 18, 2022Comments
-
jazzsir almost 2 years
I have followed the Icehouse doc to install a 3 node environment my openstack nodes are builded as virtual machines using libvirt in CentOS
http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-networking-neutron.html
-controller-node : 10.0.0.11 (management network) -network-node : 10.0.0.21 (management network), 10.0.1.21 (data network) -compute-node : 10.0.0.31 (management network), 10.0.1.31 (data network) -external network : 192.168.125.0/24 -demo network : 172.30.1.0/24In my network node, external NIC is configured witout IP as per the documentation.(PROMISC="yes")
DEVICE="eth1" TYPE="Ethernet" ONBOOT="yes" PROMISC="yes" BOOTPROTO="none" HWADDR="52:54:00:D3:92:E2" UUID="7f8a9e99-fbd0-4c59-900f-2369c9e8f780"However, after adding networking service. I cannot ping the external network in a below stage.
1) inside(tenant router gateway on network-node) -> outside(external geteway)
[root@network-node ~]# ping 192.168.125.254 PING 192.168.125.254 (192.168.125.254) 56(84) bytes of data. From 192.168.125.54 icmp_seq=1 Destination Host Unreachable From 192.168.125.54 icmp_seq=2 Destination Host Unreachable From 192.168.125.54 icmp_seq=3 Destination Host Unreachable2) outside -> inside
[root@desktop ~]# ping 192.168.125.150 PING 192.168.125.150 (192.168.125.150) 56(84) bytes of data. --> receive no responseThis is the info of my environment:
1) controller node
###The status of the port for external is "DOWN"### [root@controller-node]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:91:4e:06 brd ff:ff:ff:ff:ff:ff inet 10.0.0.11/24 brd 10.0.0.255 scope global eth0 inet6 fe80::5054:ff:fe91:4e06/64 scope link valid_lft forever preferred_lft forever [root@controller-node]# [root@controller-node]# neutron net-list +--------------------------------------+----------+-------------------------------------------------------+ | id | name | subnets | +--------------------------------------+----------+-------------------------------------------------------+ | 9cebb2a6-fd73-4ef7-81d2-188652f57ecd | demo-net | c66648c9-c34b-4806-af39-3c982378a411 172.30.1.0/24 | | e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | ext-net | a1e1fcc6-d596-4959-8923-9b46d64445af 192.168.125.0/24 | +--------------------------------------+----------+-------------------------------------------------------+ [root@controller-node]# neutron subnet-list +--------------------------------------+-------------+------------------+--------------------------------------------------------+ | id | name | cidr | allocation_pools | +--------------------------------------+-------------+------------------+--------------------------------------------------------+ | a1e1fcc6-d596-4959-8923-9b46d64445af | ext-subnet | 192.168.125.0/24 | {"start": "192.168.125.150", "end": "192.168.125.159"} | | c66648c9-c34b-4806-af39-3c982378a411 | demo-subnet | 172.30.1.0/24 | {"start": "172.30.1.2", "end": "172.30.1.254"} | +--------------------------------------+-------------+------------------+--------------------------------------------------------+ [root@controller-node]# [root@controller-node]# neutron port-list +--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+ | id | name | mac_address | fixed_ips | +--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+ | 9810105a-edf5-41bc-a140-81ccf71f6bc4 | | fa:16:3e:34:fd:cb | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} | | 98c762ea-d7f7-4c1d-9b74-73efc9990236 | | fa:16:3e:cb:0c:11 | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.1"} | | f5eec840-e629-448b-ba9a-fbcd60501247 | | fa:16:3e:ae:a6:fa | {"subnet_id": "c66648c9-c34b-4806-af39-3c982378a411", "ip_address": "172.30.1.2"} | +--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+ [root@controller-node]# [root@controller-node]# neutron port-show 9810105a-edf5-41bc-a140-81ccf71f6bc4 +-----------------------+----------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+----------------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | os-network | | binding:profile | {} | | binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} | | binding:vif_type | ovs | | binding:vnic_type | normal | | device_id | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 | | device_owner | network:router_gateway | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "a1e1fcc6-d596-4959-8923-9b46d64445af", "ip_address": "192.168.125.150"} | | id | 9810105a-edf5-41bc-a140-81ccf71f6bc4 | | mac_address | fa:16:3e:34:fd:cb | | name | | | network_id | e5f7b93c-475c-4c9d-95e4-8d1cf7728013 | | security_groups | | | status | DOWN | | tenant_id | | +-----------------------+----------------------------------------------------------------------------------------+ [root@controller-node]# [root@controller-node]# neutron router-show demo-router +-----------------------+-----------------------------------------------------------------------------+ | Field | Value | +-----------------------+-----------------------------------------------------------------------------+ | admin_state_up | True | | external_gateway_info | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} | | id | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 | | name | demo-router | | routes | | | status | ACTIVE | | tenant_id | c94f1dc5870a4d06a8b6ba947e1ac554 | +-----------------------+-----------------------------------------------------------------------------+ [root@controller-node]# [root@controller-node]# neutron router-list +--------------------------------------+-------------+-----------------------------------------------------------------------------+ | id | name | external_gateway_info | +--------------------------------------+-------------+-----------------------------------------------------------------------------+ | 8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 | demo-router | {"network_id": "e5f7b93c-475c-4c9d-95e4-8d1cf7728013", "enable_snat": true} | +--------------------------------------+-------------+-----------------------------------------------------------------------------+ [root@controller-node]# [root@controller-node]#2) network node
[root@network-node ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:f6:31:07 brd ff:ff:ff:ff:ff:ff inet 10.0.0.21/24 brd 10.0.0.255 scope global eth0 inet6 fe80::5054:ff:fef6:3107/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:d3:92:e2 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fed3:92e2/64 scope link valid_lft forever preferred_lft forever 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:48:c8:65 brd ff:ff:ff:ff:ff:ff inet 10.0.1.21/24 brd 10.0.1.255 scope global eth2 inet6 fe80::5054:ff:fe48:c865/64 scope link valid_lft forever preferred_lft forever 5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN link/ether ea:8e:aa:ad:57:60 brd ff:ff:ff:ff:ff:ff 6: br-ex: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether a2:f1:0b:6b:34:4f brd ff:ff:ff:ff:ff:ff inet6 fe80::a0f1:bff:fe6b:344f/64 scope link valid_lft forever preferred_lft forever 9: br-int: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether 32:a4:53:15:fc:4f brd ff:ff:ff:ff:ff:ff inet6 fe80::30a4:53ff:fe15:fc4f/64 scope link valid_lft forever preferred_lft forever 12: gre0: <NOARP> mtu 1476 qdisc noop state DOWN link/gre 0.0.0.0 brd 10.0.0.31 13: gretap0: <BROADCAST,MULTICAST> mtu 1476 qdisc noop state DOWN qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff 22: tun0@NONE: <POINTOPOINT,NOARP> mtu 1476 qdisc noqueue state DOWN link/gre 0.0.0.0 peer 10.0.0.31 inet 10.0.1.21 peer 10.0.1.31/32 scope global tun0 24: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether de:a8:a4:b1:b1:46 brd ff:ff:ff:ff:ff:ff inet6 fe80::dc0e:8cff:fe67:d352/64 scope link valid_lft forever preferred_lft forever [root@network-node ~]# [root@network-node ~]# ovs-vsctl show 23804a8f-7c89-4422-9b9f-67bf26a34c51 Bridge br-int fail_mode: secure Port br-int Interface br-int type: internal Port "qr-98c762ea-d7" tag: 1 Interface "qr-98c762ea-d7" type: internal Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port "tapf5eec840-e6" tag: 1 Interface "tapf5eec840-e6" type: internal Bridge br-ex Port "eth1" Interface "eth1" Port br-ex Interface br-ex type: internal Port "qg-9810105a-ed" Interface "qg-9810105a-ed" type: internal Bridge br-tun Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Port "gre-0a00011f" Interface "gre-0a00011f" type: gre options: {in_key=flow, local_ip="10.0.1.21", out_key=flow, remote_ip="10.0.1.31"} Port br-tun Interface br-tun type: internal ovs_version: "1.11.0" [root@network-node ~]# [root@network-node ~]# ip netns list qdhcp-9cebb2a6-fd73-4ef7-81d2-188652f57ecd qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 [root@network-node ~]# [root@network-node ~]# ip netns exec qrouter-8ae4b1fa-fb60-4690-bbe2-febbfbcf7555 route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 172.30.1.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-98c762ea-d7 192.168.125.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-9810105a-ed 0.0.0.0 192.168.125.254 0.0.0.0 UG 0 0 0 qg-9810105a-ed [root@network-node ~]#Any hints would be much appreciated!
Cheers,
hbseo