Separate Network Traffic on Two Network Interfaces

Question

Could you lend your expertise in understanding how to go about configuring the separation of network traffic on two network interfaces?

As I understand thus far, static routes are used for network traffic that is not designed to use a default gateway. The default gateway is used for all traffic which is not destined for the local network and for which no preferred route has been specified in a routing table.

The scenario is as follows.

• Each computer in the network has two network cards.
• The production interface for each is eth0 (GW = 10.10.10.1).
• The management interface for each is eth1 (GW = 192.168.100.1).
• Production and Management traffic should be totally separated.

I have posted, below, what things I have tried with Debian Wheezy. And, my problem is that, although I have hosts set up in such a way that they do communicate on both interfaces, individual hosts seem to "hear" traffic on the wrong interface. For example:

Host 140

eth0      Link encap:Ethernet  HWaddr 08:00:27:d1:b6:8f
          inet addr:10.10.10.140  Bcast:10.10.10.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fed1:b68f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1341 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2530 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:641481 (626.4 KiB)  TX bytes:241124 (235.4 KiB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:ad:14:b6
          inet addr:192.168.100.140  Bcast:192.168.100.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fead:14b6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7220 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5257 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:602485 (588.3 KiB)  TX bytes:1022906 (998.9 KiB)

From host 140, I execute this command: tcpdump -i eth0. In a separate session on host 140, I execute ping 192.168.100.50.

19:17:29.301565 IP 192.168.100.140 > 192.168.100.50: ICMP echo request, id 1400, seq 10, length 64
19:17:30.301561 IP 192.168.100.140 > 192.168.100.50: ICMP echo request, id 1400, seq 11, length 64
19:17:31.301570 IP 192.168.100.140 > 192.168.100.50: ICMP echo request, id 1400, seq 12, length 64
19:17:32.301580 IP 192.168.100.140 > 192.168.100.50: ICMP echo request, id 1400, seq 13, length 64

Why do I see the above output on eth0? I think I should only see traffic for 10.10.10.140. I also see this on eth1, as expected:

19:18:47.805408 IP 192.168.100.50 > 192.168.100.140: ICMP echo request, id 1605, seq 247, length 64

If I ping from Host 50 (same ifconfig results - just a different last quad), then eth0 is silent, and I see the ICMP echos on eth1, as expected.

I would like to understand how to configure each interface to handle only the traffic for which it is responsible in two major Linux varieties. I think I am almost there, but I am missing something I just can't seem to find.

• Debian Wheezy (7.x) or Debian Jessie (8.x)
• Enterprise Linux (6.x) (RedHat/CentOS/Scientific/Oracle).

I know that a solution for Debian should be good for both Wheezy and Jessie, and that a solution for an EL should be the same for all the EL 6.x versions. I would like to avoid using an RC script to execute commands, opting instead for using the configuration files.

In Debian the relevant configuration files that I know about are:

• /etc/network/interfaces

In EL 6.x, the relevant configuration files that I know about are:

• /etc/sysconfig/network
• /etc/sysconfig/network-scripts/ifcfg-eth0
• /etc/sysconfig/network-scripts/ifcfg-eth1
• /etc/sysconfig/network-scripts/route-eth0
• /etc/sysconfig/network-scripts/route-eth1
• /etc/sysconfig/network-scripts/rule-eth0
• /etc/sysconfig/network-scripts/rule-eth1

My Debian 8 "Jessie" /etc/network/interfaces file:

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# Production interface
auto eth0
allow-hotplug eth0
iface eth0 inet static
  address 10.10.10.140
  netmask 255.255.255.0
  gateway 10.10.10.1

# Management interface
auto eth1
allow-hotplug eth1
iface eth1 inet static
  address 192.168.100.140
  netmask 255.255.255.0

I think netstat -anr might illustrate the problem:

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.10.10.1      0.0.0.0         UG        0 0          0 eth0
10.10.10.0      0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.100.0   0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.100.0   0.0.0.0         255.255.255.0   U         0 0          0 eth1

Solution 1

I'd love to know more about this topic to refine the configuration to be the best that it can be, but here's what I have so far. Even without enabling ARP filtering on all network interfaces (net.ipv4.conf.all.arp_filter = 0), as mentioned by @spuk, traffic seems to be completely separated in this configuration.

The file, /etc/iproute2/rt_tables, is the same in EL 6.x and DEB 7/8, at least. This is the file that creates a named routing table for static routes.

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
252 mgmt

Above, the number of the named, static route, 252, is essentially arbitrary; or, each static route gets its own unique number between 1 and 252.

The file, /etc/network/interfaces in DEB 7/8, at least:

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
  iface lo inet loopback

# The production network interface
# The 'gateway' directive is the default route.
# Were eth0 configured via DHCP, the default route would also be here.
auto eth0
allow-hotplug eth0
iface eth0 inet static
  address 10.10.10.140
  netmask 255.255.255.0
  gateway 10.10.10.1

# The management network interface
# The 'gateway' directive cannot be used again because there can be
# one, and only one, default route. Instead, the 'post-up' directives
# use the `mgmt` static route.
auto eth1
allow-hotplug eth1
iface eth1 inet static
  address 192.168.100.140
  netmask 255.255.255.0
  post-up ip route add 192.168.100.0/24 dev eth1 src 192.168.100.140 table mgmt
  post-up ip route add default via 192.168.100.1 dev eth1 table mgmt
  post-up ip rule add from 192.168.100.140/32 table mgmt
  post-up ip rule add to 192.168.100.140/32 table mgmt

The result of ip route show on Debian:

default via 10.10.10.1 dev eth0
10.10.10.0/24 dev eth0  proto kernel  scope link  src 10.10.10.140
192.168.100.0/24 dev eth1  proto kernel  scope link  src 192.168.100.140

The EL 6.x /etc/sysconfig/network file:

NETWORKING=yes
HOSTNAME=localhost.localdomain
GATEWAY=10.10.10.1

Above, GATEWAY is the default route. Below, were BOOTPROTOCOL set to DHCP, the default route would be acquired from DHCP.

THE EL 6.x /etc/sysconfig/network-scripts/ifcfg-eth0 file, without "HWADDR" and "UUID":

DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTOCOL=none
IPADDR=10.10.10.140
NETMASK=255.255.255.0
NETWORK=10.10.10.0
BROADCAST=10.10.10.255

THE EL 6.x /etc/sysconfig/network-scripts/ifcfg-eth1 file, without "HWADDR" and "UUID":

DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTOCOL=none
IPADDR=192.168.100.140
NETMASK=255.255.255.0
NETWORK=192.168.100.0
BROADCAST=192.168.100.255

The EL 6.x /etc/sysconfig/network-scripts/route-eth1 file:

192.168.100.0/24 dev eth1 table mgmt
default via 192.168.100.1 dev eth1 table mgmt

The EL 6.x /etc/sysconfig/network-scripts/rule-eth1 file:

from 192.168.100.0/24 lookup mgmt

The result of ip route show on EL 6.x:

192.168.100.0/24 dev eth1  proto kernel  scope link  src 192.168.100.160
10.10.10.0/24 dev eth0  proto kernel  scope link  src 10.10.10.160
default via 10.10.10.1 dev eth0

Update for RHEL8

This method described above works with RHEL 6 & RHEL 7 as well as the derivatives, but for RHEL 8 and derivatives, one must first install network-scripts to use the method described above.

dnf install network-scripts

The installation produces a warning that network-scripts will be removed in one of the next major releases of RHEL and that NetworkManager provides ifup/ifdown scripts as well.

Update for Ubuntu 20.04 LTS

Creating a named routing table is ok, but not required with netplan, which will not use the name anyway. Nonetheless the number of the named routing table from the rt_tables file can be used for netplan. Corresponding NICs are enps03 (eth0) and enp0s8 (eth1).

network:
  version: 2
  ethernets:
    enp0s3:
      addresses:
      - 10.10.10.140/24
      dhcp4: false
      dhcp6: false
      gateway4: 10.10.10.1
      nameservers:
        addresses:
        - 1.2.3.4
        - 1.2.3.5
        search:
        - your-search-domain-name.com
    enp0s8:
      dhcp4: false
      dhcp6: false
      addresses:
      - 192.168.100.140/24
      routes:
      - to: 192.168.100.0/24
        via: 192.168.100.1
        table: 252
      routing-policy:
      - from: 192.168.100.0/24
        table: 252

This results in the following routes from ip r s.

default via 10.10.10.1 dev enp0s3 proto static
10.10.10.0/24 dev enp0s3 proto kernel scope link src 10.10.10.140
192.168.100.0/24 dev enp0s8 proto kernel scope link src 192.168.100.140

Solution 2

I haven't read throughly all your post (sorry, can't really spend the time right now), but I believe it may be related to the way Linux implements the IP host model:

... The IPv4 implementation in Linux defaults to the weak host model. ...

From that same page:

... If the IP stack is implemented with a weak host model, it accepts any locally destined packet regardless of the network interface on which the packet was received. ...

That is, in Linux, by default, the IP addresses "belong to the host", not strictly "to the interface". You can change that behavior via the arp_filter, rp_filter, arp_announce, arp_ignore sysctls (got from LVS: The ARP Problem, seen here). Also, see ip-sysctl.txt.

Comments

[Admin]

check iptabels -L -t nat