Using two network interfaces for FreeBSD VM

routing freebsd virtualbox

5,326

What you are doing is pretty strange:
1) You have same subnet reachable from different interfaces which will make your life a bit difficult for some types of server software
2) When you put 192.168.1.0/24's interface down you should mangle routes:

sudo route add -net 192.168.1.0/24 -iface em1

One of pretty straightforward solutions for your problem is to use link aggregation technique.

There is lagg(4) device with failover mode for that. There is config from my notebook which always stays on fastest available link:

ifconfig_em0="ether 00:26:c7:xx:xx:xx"
ifconfig_wlan0="WPA"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto failover laggport em0 laggport wlan0 DHCP"

in your case you should have lagg with em0 and em1. Also note that I set MAC of em0 same as my wifi card.

As another solution you can set up an IGP between host and guest OS (e.g., OSPF).

5,326

tony19

Updated on September 18, 2022

Comments

tony19 over 1 year

How do I turn on/off one network interface without affecting the other in FreeBSD?

I have a FreeBSD 8.2 virtual machine (Virtual Box 4.1.4), with two network interfaces, one bridged to my Ethernet card and another to WiFi on the host OS (Mac OSX Lion 10.7.2). In the VM, I assigned the first two network interfaces (em0 and em1) to static IP addresses, and set the default router IP to my router's IP, as shown in the config here:

/etc/rc.conf :

inetd_enable="NO"
sshd_enable="YES"

ifconfig_em0="inet 192.168.1.109 netmask 255.255.255.0"
ifconfig_em1="inet 192.168.1.129 netmask 255.255.255.0"
defaultrouter="192.168.1.1"

netstat -r :

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.1.1        UGS         2       71    em0
127.0.0.1          link#3             UH          0        0    lo0
192.168.1.0/24     link#1             U           2       78    em0
192.168.1.109      link#1             UHS         0        0    lo0
192.168.1.129      link#2             UHS         0        0    lo0

With the configuration above, I can ssh to em0 or em1 from the host OS, but if I take down em0 with:

sudo ifconfig em0 down

...em1 loses network access, as if it were also taken down. That is, the host OS can't ssh into em1; and the guest OS can't ping google.com or ping 192.168.1.130 (IP of host OS).

I don't have this problem in Ubuntu Server 11.10: I can setup a static IP for my WiFi bridge and a dynamic IP for my Ethernet bridge; and take either of them down without affecting the other.

BEFORE em0 down

$ ifconfig -a
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
    ether 08:00:27:16:4d:32
    inet 192.168.1.109 netmask 0xffffff00 broadcast 192.168.1.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
    ether 08:00:27:3f:a7:44
    inet 192.168.1.129 netmask 0xffffff00 broadcast 192.168.1.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=3<RXCSUM,TXCSUM>
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
    inet6 ::1 prefixlen 128 
    inet 127.0.0.1 netmask 0xff000000 
    nd6 options=3<PERFORMNUD,ACCEPT_RTADV>

$ arp -an
? (192.168.1.129) at 08:00:27:3f:a7:44 on em1 permanent [ethernet]
? (192.168.1.130) at c4:2c:03:2b:73:d1 on em0 expires in 314 seconds [ethernet]

AFTER em0 down

$ ifconfig -a
em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 08:00:27:16:4d:32
        inet 192.168.1.109 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 08:00:27:3f:a7:44
        inet 192.168.1.129 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
        
$ arp -an
? (192.168.1.129) at 08:00:27:3f:a7:44 on em1 permanent [ethernet]
? (192.168.1.130) at c4:2c:03:2b:73:d1 on em0 expires in 190 seconds [ethernet]

Output (as requested):

@host : ~/tmp
$ ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
    options=3<RXCSUM,TXCSUM>
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
    inet 127.0.0.1 netmask 0xff000000 
    inet6 ::1 prefixlen 128 
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
en0: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
    options=2b<RXCSUM,TXCSUM,VLAN_HWTAGGING,TSO4>
    ether c4:2c:03:2b:73:d1 
    inet6 fe80::c62c:3ff:fe2b:73d1%en0 prefixlen 64 scopeid 0x4 
    inet 192.168.1.130 netmask 0xffffff00 broadcast 192.168.1.255
    media: autoselect (100baseTX <full-duplex,flow-control>)
    status: active
fw0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 4078
    lladdr d8:a2:5e:ff:fe:f2:f9:d4 
    media: autoselect <full-duplex>
    status: inactive
en1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    ether d8:30:62:58:93:7d 
    inet6 fe80::da30:62ff:fe58:937d%en1 prefixlen 64 scopeid 0x6 
    inet 192.168.1.146 netmask 0xffffff00 broadcast 192.168.1.255
    media: autoselect
    status: active
p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304
    ether 0a:30:62:58:93:7d 
    media: autoselect
    status: active
vboxnet0: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    ether 0a:00:27:00:00:00 

@host : ~/tmp
$ sudo arp -ad 
192.168.1.1 (192.168.1.1) deleted
192.168.1.109 (192.168.1.109) deleted

@host : ~/tmp
$ arp -an
? (192.168.1.1) at 0:1d:7e:f8:1c:dd on en0 ifscope [ethernet]

@host : ~/tmp
$ ping -c3 192.168.1.109
PING 192.168.1.109 (192.168.1.109): 56 data bytes
64 bytes from 192.168.1.109: icmp_seq=0 ttl=64 time=0.418 ms
64 bytes from 192.168.1.109: icmp_seq=1 ttl=64 time=0.457 ms
64 bytes from 192.168.1.109: icmp_seq=2 ttl=64 time=0.468 ms

--- 192.168.1.109 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.418/0.448/0.468/0.021 ms

@host : ~/tmp
$ ping -c3 192.168.1.129
PING 192.168.1.129 (192.168.1.129): 56 data bytes
64 bytes from 192.168.1.129: icmp_seq=0 ttl=64 time=104.138 ms
64 bytes from 192.168.1.129: icmp_seq=1 ttl=64 time=22.004 ms
64 bytes from 192.168.1.129: icmp_seq=2 ttl=64 time=44.780 ms

--- 192.168.1.129 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 22.004/56.974/104.138/34.622 ms

@host : ~/tmp
$ arp -an
? (192.168.1.1) at 0:1d:7e:f8:1c:dd on en0 ifscope [ethernet]
? (192.168.1.109) at 8:0:27:16:4d:32 on en0 ifscope [ethernet]
? (192.168.1.129) at d8:30:62:58:93:7d on en0 ifscope [ethernet]

#############################
# GUEST em0 down here
#############################

@host : ~/tmp
$ arp -an
? (192.168.1.1) at 0:1d:7e:f8:1c:dd on en0 ifscope [ethernet]
? (192.168.1.109) at 8:0:27:16:4d:32 on en0 ifscope [ethernet]
? (192.168.1.129) at d8:30:62:58:93:7d on en0 ifscope [ethernet]

@host : ~/tmp
$ sudo arp -ad
192.168.1.1 (192.168.1.1) deleted
192.168.1.109 (192.168.1.109) deleted
192.168.1.129 (192.168.1.129) deleted

@host : ~/tmp
$ ping -c3 192.168.1.129
PING 192.168.1.129 (192.168.1.129): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1

--- 192.168.1.129 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

@host : ~/tmp
$ arp -an
? (192.168.1.1) at 0:1d:7e:f8:1c:dd on en0 ifscope [ethernet]
? (192.168.1.129) at d8:30:62:58:93:7d on en0 ifscope [ethernet]





###################
GUEST OS

@guest : ~/tmp
$ sudo arp -ad
192.168.1.129 (192.168.1.129) deleted

@guest : ~/tmp
$ arp -an
? (192.168.1.129) at 08:00:27:3f:a7:44 on em1 permanent [ethernet]

#############################
# HOST pings .109 and .129 here
#############################

@guest : ~/tmp
$ arp -an
? (192.168.1.129) at 08:00:27:3f:a7:44 on em1 permanent [ethernet]
? (192.168.1.130) at c4:2c:03:2b:73:d1 on em0 expires in 1188 seconds [ethernet]

@guest : ~/tmp
$ sudo ifconfig em0 down

@guest : ~/tmp
$ arp -an
? (192.168.1.129) at 08:00:27:3f:a7:44 on em1 permanent [ethernet]
? (192.168.1.130) at c4:2c:03:2b:73:d1 on em0 expires in 1170 seconds [ethernet]

@guest : ~/tmp
$ sudo arp -ad
192.168.1.129 (192.168.1.129) deleted
delete: cannot locate 192.168.1.130

#############################
# HOST pings .129 here
#############################

@guest : ~/tmp
$ arp -an
? (192.168.1.129) at 08:00:27:3f:a7:44 on em1 permanent [ethernet]
? (192.168.1.130) at c4:2c:03:2b:73:d1 on em0 expires in 1095 seconds [ethernet]

YwH over 12 years

What does ifconfig -a show on the guest before and after you bring em0 down? What does arp -an show on the host and on the guest when things are working and when they aren't?
tony19 over 12 years

I just updated the question to show the before/after configs.
YwH over 12 years

How about arp -an on the host OSX system before and after? If you do "arp -d 192.168.1.130" in FreeBSD, then ping, then look at arp -an again, what does it show? My guess is that the host/guest are getting confused about which interfaces they should communicate on.
tony19 over 12 years

The host's arp is the same before and after. Nothing interesting.
YwH over 12 years

The host arp may be the same, but is it right? My guess is that this works in your Ubuntu system because of a thing called arp flux (google it), but I can't tell without more information. I believe what is happening is that when em0 is up, the host thinks that both .109 and .129 are at its MAC (ending in 4d:32). When em0 is down, the host still has that information cached, so connectivity to .129 fails. I'll post the steps I'd take to troubleshoot this in the next comment (not enough space here).
YwH over 12 years

Here is the information I'd get to troubleshoot this (just paste it into the question, not the comment): To make sure you are starting with a clean slate: do the following on both the guest and on the host: 1) sudo arp -d 192.168.1.129, .109 and .130. 2) get arp -an from both (should be empty, except for the permanent entries which are owned by the system). 3) ping .109 and .129 from host. 4) Get arp -an from both. 5) down em0. 6) arp -an on both. 7) repeat step 1 (delete arp entries everywhere). 8) ping .129 from host. 9) get arp -an from both.
tony19 over 12 years

Ok, I pasted the command output into the question. Thx
YwH over 12 years

One more thing: how about an ifconfig -a on the host?
tony19 over 12 years

Ok, added above.

tony19 over 12 years

Thanks, I appreciate your answer. However, I'm not looking for link aggregation or failover (I'm indeed aware of lagg). I should be able to disable one network interface without affecting another without link aggregation (just like I can in OSX and Ubuntu).
SaveTheRbtz over 12 years

you can indeed disable an interface but then you must change 192.168.1.0/24 and default routes' interface.