how do I build a bridge between two virtual machines?

Question

I have used virt-install to create two CentOS 7 virtual machine guests on a CentOS 7 host computer. One virtual machine is called webvm, and hosts web sites on a private network. The other virtual machine is called datavm and has the sole purpose of being the virtual database server for the web apps hosted on webvm. How can I set up networking so that datavm ONLY allows data connections from webvm, and so that those data connections happen WITHIN the physical server box? I want to make sure that the database transactions between webvm and datavm do not travel across the local network.

Note that bridge networking already links the host OS to each of the guest OS'.

The local area network ip of webvm is 10.0.0.6 and the network ip of datavm is 10.0.0.5. The connection string from a typical web app hosted on webvm is:

jdbc:mysql://localhost:3306/somedb?autoReconnect=true

You can see that localhost refers to webvm. We apparently need to set up a NEW, second bridge network with 10.1.1.x addresses, so that the connection string would become jdbc:mysql://10.1.1.1:3306/somedb?autoReconnect=true.

Whatever new bridge networking code we add must not conflict with the pre-existing bridge network.

So how do I set up a one-to-one, exclusive data connection between datavm and webvm?

---

UPDATED WORK IN PROGRESS:

---

@derobert suggested the following steps:

1.) Add a second bridge to the host. 
2.) Add a second network interface to webvm, connected to the new host bridge. 
3.) Add a second network interface to datavm, connected to the new host bridge.
4.) Configure the new network interfaces inside each guest.  

Towards this end, I got a baseline by running the following in the HOST:

[root@localhost ~]# nmcli con show
NAME               UUID                                  TYPE            DEVICE 
bridge-slave-eno1  c36fd051-cacc-4e91-944f-a98f4fee26ff  802-3-ethernet  eno1   
bridge-br0         d472bc86-0f75-4dd5-bfee-5b8208b3fed2  bridge          br0    
System eno1        abf4c85b-57cc-4484-4fa9-b4a71689c359  802-3-ethernet  --     
vnet1              ea985e89-94fb-403c-af33-7daefb378ca5  generic         vnet1  
vnet0              06deb20d-b0b7-4233-8abc-cbb285165082  generic         vnet0  
[root@localhost ~]# 

Then I ran the following inside webvm:

[root@localhost ~]# nmcli con show
NAME  UUID                                  TYPE            DEVICE 
eth0  71bf7ff1-7574-4364-8c83-5878ed30d028  802-3-ethernet  eth0   
[root@localhost ~]# 

Then I ran the following inside datavm:

[root@localhost ~]# nmcli con show
NAME  UUID                                  TYPE            DEVICE 
eth0  d976f7ca-ab7f-4fd0-ab2b-6213815bd1a1  802-3-ethernet  eth0   
[root@localhost ~]# 

I then implemented the following commands on the HOST:

[root@localhost ~]# nmcli con add type bridge ifname br1
Connection 'bridge-br1' (8b9fd6d9-bcb4-4e1c-85ab-55905d08667e) successfully added.
[root@localhost ~]# nmcli con show
NAME               UUID                                  TYPE            DEVICE 
bridge-slave-eno1  c36fd051-cacc-4e91-944f-a98f4fee26ff  802-3-ethernet  eno1   
bridge-br0         d472bc86-0f75-4dd5-bfee-5b8208b3fed2  bridge          br0    
System eno1        abf4c85b-57cc-4484-4fa9-b4a71689c359  802-3-ethernet  --     
bridge-br1         8b9fd6d9-bcb4-4e1c-85ab-55905d08667e  bridge          br1    
vnet1              ea985e89-94fb-403c-af33-7daefb378ca5  generic         vnet1  
vnet0              06deb20d-b0b7-4233-8abc-cbb285165082  generic         vnet0  
[root@localhost ~]# virsh
Welcome to virsh, the virtualization interactive terminal.
virsh # list
 Id    Name                           State
----------------------------------------------------
 2     public4-centos7                running
 4     data-centos7                   running

virsh # attach-interface data-centos7 bridge br1
Interface attached successfully

virsh # attach-interface public4-centos7 bridge br1
Interface attached successfully

virsh # 

I then logged in to each of the virtual machines separately, and the new connections to the bridge network were shown with the name Wired connection 1, as follows:

In the web vm:

[root@localhost ~]# nmcli con show
NAME                UUID                                  TYPE            DEVICE 
Wired connection 1  44f1f791-0d86-4587-8a2d-48dfa217ee99  802-3-ethernet  ens7   
eth0                71bf7ff1-7574-4364-8c83-5878ed30d028  802-3-ethernet  eth0   
[root@localhost ~]# nmcli con modify 'Wired connection 1' ipv4.addresses "10.1.1.2"

And in the data vm:

[root@localhost ~]# nmcli con show
NAME                UUID                                  TYPE            DEVICE 
Wired connection 1  448101d7-1f8f-4b78-ad90-7efd5be23b08  802-3-ethernet  ens7   
eth0                d976f7ca-ab7f-4fd0-ab2b-6213815bd1a1  802-3-ethernet  eth0   
[root@localhost ~]# nmcli con modify 'Wired connection 1' ipv4.addresses "10.1.1.1"  

But then ping 10.1.1.1 from the web vm failed (Destination Host Unreachable), and ping 10.1.1.2 from the data vm also failed (Destination Host Unreachable).

In web vm, the contents of vi /etc/sysconfig/network-scripts/ifcfg-Wired_connection_1 are:

HWADDR=52:54:00:8F:3B:14
TYPE=Ethernet
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME="Wired connection 1"
UUID=44f1f791-0d86-4587-8a2d-48dfa217ee99
ONBOOT=yes
IPADDR=10.1.1.2
PREFIX=16
PEERDNS=yes
PEERROUTES=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes

In data_vm, the contents of vi /etc/sysconfig/network-scripts/ifcfg-Wired_connection_1 are:

HWADDR=52:54:00:1F:FE:27
TYPE=Ethernet
BOOTPROTO=dhcp
IPADDR=10.1.1.1
PREFIX=32
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME="Wired connection 1"
UUID=448101d7-1f8f-4b78-ad90-7efd5be23b08
ONBOOT=yes

What else do I type to finish what @derobert suggested? Remember that all data traffic needs to stay INSIDE THE PHYSICAL BOX, so that the new bridge will have to include new ip addresses for datavm and webvm to use ONLY in the new bridge.

As per @garethTheRed's comments, I typed ip route in the web vm and got the following:

[root@localhost network-scripts]# ip route
default via 10.0.0.1 dev eth0  proto static  metric 100 
10.0.0.0/24 dev eth0  proto kernel  scope link  src 10.0.0.6  metric 100 
10.1.1.0/30 dev ens7  proto kernel  scope link  src 10.1.1.2 
10.1.1.2/31 dev ens7  proto kernel  scope link  src 10.1.1.2  metric 100 
169.254.0.0/16 dev ens7  scope link  metric 1003 
[root@localhost network-scripts]# 

I then typed ip route in data vm and got the following:

[root@localhost network-scripts]# ip route
default via 10.0.0.1 dev eth0  proto static  metric 100 
10.0.0.0/24 dev eth0  proto kernel  scope link  src 10.0.0.5  metric 100 
10.1.1.0/31 dev ens7  proto kernel  scope link  src 10.1.1.1  metric 100 
10.1.1.0/30 dev ens7  proto kernel  scope link  src 10.1.1.1 
169.254.0.0/16 dev ens7  scope link  metric 1003 
[root@localhost network-scripts]# 

Stripping the ifcfg-* file down to the 6 lines in the answer caused a failure when I tried systemctl restart network. I think it may be due to the hardware or uuid arguments, but that is just a guess. When I restored the ifcfg-* files to include @garethTheRed's edits in addition to the extra arguments shown above, systemctl restart network then ran without error, but the pings failed.

Solution

Two things could be the cause of this:

One possible cause is the way you've built the private network (using the bridge on the host). It would be much safer and easier to configure this using virt-manager. If your host is CLI only, install it on a remote desktop/laptop and connect over SSH to the host.

Once installed, connect to the hypervisor, right click on it's name in the list and choose 'Details'. On the 'Virtual Networks' tab you can add a new network by clicking on the '+' button (lower,left). The wizard will guide you through the process, but make sure you un-check the options for IPv4 and IPv6 addresses (you don't need them as it's a point-to-point link) and choose the radio button for 'Private network'. Continue with the wizard and exit.

If you're a die-hard command line only person, then the above can be carried out using the virsh net-define command line interface. Create an XML file such as the example below (virbr2 is an unused bridge name - use brctl show to list yours):

<network>
    <name>private</name>
    <bridge name="virbr2" />
</network>

Then import it with:

# virsh net-define <XML filename>

Once the above is done, you can edit each VM to use this new private network (you'll have to reboot the VMs for this to take effect). Once you've edited the VM configuration, you'll be ready to log in to each one and configure the OS with the relevant IP details (from your OP). But, read on first...

Secondly, you have no routes between the two VMs pointing to the new interfaces. Another This is because you've configured the IP address with a /32 prefix.

When editing the connection use the format a.b.c.d/p to set the prefix; otherwise, without a prefix, it will default to /32:

# nmcli con edit "Wired connection 1"
nmcli> set ipv4.addresses 10.1.1.1/30
nmcli> save
nmcli> quit
# systemctl restart network

Configuring the network manually also works:

NM_CONTROLLED=no
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
IPADDR=10.1.1.1
PREFIX=30

Carry out a similar procedure on the other VM (10.1.1.2/30) and it should work.

Note: You could use /31 but only if you changed the IP addresses to 10.1.1.0 and 10.1.1.1. This may cause problems as the first address in the range is usually reserved for the network address and the last for broadcast. As you only have two addresses with /31, you'd have none left for the hosts. Best stick with /30, which gives you 4 addresses - two reserved and two for your hosts.

Comments

[psusi]

I just noticed you are using centos and not a debian based distribution, and I'm pretty sure that /etc/networking/interfaces is a Debian thing.

[RabT]

[root@localhost ~]# cd /etc/networking/interfaces -bash: cd: /etc/networking/interfaces: No such file or directory

[RabT]

I need a command line approach, so that this can be done with shell scripts later on. Are you willing to look at the terminal output from my UPDATED WORK IN PROGRESS section of the OP, and give specific code suggestions? Thank you.

[RabT]

If I set up a firewall rule inside datavm to only accept 3306 traffic from webvm, then all that traffic would go through the local network instead of staying inside the physical box. Also, creating a separate ip for datavm and webvm to use in the second bridge would likely enable the traffic to stay inside the physical box. This is sensitive data.

[OJS]

oooh my apologies, i couldnt post a comment sorry

[psusi]

@CodeMed, no, it wouldn't. Traffic from one vm to another when they are connected via a virtual bridge in the host doesn't go out on the lan.

[RabT]

Yes, of course, because they have their own ip addresses, as shown in the UPDATED WORK IN PROGRESS in my OP. Are you willing to help resolve the remaining obstacles to making that approach work? The OP specifies that approach.

[RabT]

You can build rep by asking good questions. The resulting rep will enable you to make comments.

[garethTheRed]

The IP address of the original interfaces is 10.0.0.x. Is this on a traditional class A network or have you used a (e.g.) class C (/24 or 255.255.255.0) net-mask? I've noticed that you have different prefixes on the two new connections - it's worth tidying them up :-)

[RabT]

@garethTheRed Thank you. The 10.0.0.x addresses are with the same comcast xfinity modem that we have discussed in other postings. By contrast, the 10.1.1.x addresses I just made up myself because I thought they would not conflict. I just changed prefixes to both be 32 in the ifcfg-* files, but the pings both fail. I do not know what class network it is. I have been learning by trial and error. Is there more that I can do or show you?

[garethTheRed]

@CodeMed - What are you using as the hypervisor? VirtualBox? KVM? I removed all the extra lines (many of which are for IPv6) from mine when I tested on VirtualBox just then and it worked. I'm certain systemctl restart network worked for me. Check if the route containing 10.1.1.0 is displayed when you run ip route on the guests.

[garethTheRed]

I noticed that your web_vm has an IP address of 10.1.1.2 but additionally, you have a route to 10.1.1.2/31 in web_vm's routing tables. You shouldn't have that there.

[garethTheRed]

let's move to @chat chat.stackexchange.com/rooms/30308/codemeds-woes before we get kicked into one.

[psusi]

@CodeMed, if the netmask on one connection is 255.0.0.0, aka a class A, aka a /8, then it conflicts with 10.*.*.*. Also before I meant that even if the vms are not on their own dedicated network, traffic between them will still not go out on the lan. Other hosts on the lan can talk to both vms, but when the vms talk to each other, it doesn't go over the lan.

[RabT]

I did a deep study of the RHEL7 Networking Guide, and I have re-framed my new question with a narrower focus. I think I just need a small adjustment to be able to use nmcli to give a specific public IP to a CentOS 7 server box connected to a Comcast Cisco DPC3941B cable modem. Are you willing to comment? Here is the link: unix.stackexchange.com/questions/347960/…