Two NICs on the same subnet but with different networks?

Question

Description

This is similar to this question, but different:

Linux computer has two NIC. eth0 is statically assigned with an IP. eth1 is assgiend by personnel in the field which could be same IP or different IP. The plan is for eth0 and eth1 to be on different network, and never exchange or bridge any packets. And my Linux application only listens and accepts connection on eth0 and eth1, and start sending packets upon receiving, but never try to initiate any activity.

Question

Can two NIC have IP in same subnet: 192.168.1.4 and 192.168.1.5? As you can see eth0 and eth1 are assigned to be in same subnet (255.255.255.0) but they are not connected and are not supposed to be connected.

In this case, when the application listens on eth0 port 55555, accepts connection and returns packet back, does the underlying layer knows it should return to eth0? Or will it possibly try eth1 since the OS thinks it is on same subnet? Is there anything I need to do to the routing table to make sure it doesn’t think that these two NICs are actually on the same subnet?

Should I avoid not only similar IPs but also same subnet in this case?

Update

What if PC0 and PC1 have same IP?

PC0 (1.100) <-------> [eth0 (1.4)  My System eth1 (1.5)]<-------PC1(1.100)

My application needs to listen on port 55555 on eth0, and eth1, and a request comes in through eth1, can the OS knows to reply go through eth1? Will this configuration cause a problem?

The business case is I am building this embedded system, and I pre-define the IP for eth0 and PC0 (PC0 could be DHCP as well). But my customer already has a network on the right side. What if they have a device which conflicts with either PC0 or eht0? Even with DHCP server on the eth0, there is no way to exclude the IPs on the right side when assigning IP to PC0. If this creates problems, I have many solutions. But I would like to hear peer opinions whether it is a problem.

Two of my colleagues think it is not a problem. My opinion is that the IP layer doesn't know which interface to use to reply a packet even using a socket (assume bind to right side). It will just pick one no matter how we set up the routing table.

Solution

Yes, you can. Your computer will send return packets out whichever interface is associated with the route to the destination network.

In likelihood, the destination network is contained within the default route - 0.0.0.0/0, through the default gateway.

Don't believe me ?
Let's examine :

[root@localhost ~]# ip address show label eth* | grep -v 'link\|val'
  2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
      inet 192.168.1.4/24 brd 192.168.1.255 scope global eth0
  3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
      inet 192.168.1.5/24 brd 192.168.1.255 scope global eth1

Two interfaces, eth0 and eth1, with IP addresses & mask as in your question.

Full start-up config :

[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
  DEVICE="eth0"
  NM_CONTROLLED="no"
  ONBOOT="yes"
  TYPE=Ethernet
  DEFROUTE=yes
  IPV4_FAILURE_FATAL=yes
  IPV6INIT=no
  Name="eth0"
  BOOTPROTO=none
  IPADDR=192.168.1.4
  NETMASK=255.255.255.0
  GATEWAY=192.168.1.1

[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
  DEVICE="eth1"
  NM_CONTROLLED="no"
  ONBOOT="yes"
  TYPE=Ethernet
  DEFROUTE=yes
  IPV4_FAILURE_FATAL=yes
  IPV6INIT=no
  Name="eth1"
  BOOTPROTO=none
  IPADDR=192.168.1.5
  NETMASK=255.255.255.0
  GATEWAY=192.168.1.2

Note that different gateway addresses are set for each - .1 and .2.
Now let's look at the routing table :

[root@localhost ~]# ip route list
  default via 192.168.1.2 dev eth0 
  192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.4 
  192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.5 

Looks like only one default route has been chosen - out eth0 for whatever reason (I don't know, but I assume because it's the lower-numbered NIC), but using the gateway .2. Perhaps because it was the latter gateway statement ? Hell if I know.

Let's see what the kernel considers the appropriate route for a given public-address destination, sourced from either local IP address :

[root@localhost ~]# ip route get to 8.8.8.8 from 192.168.1.4
  8.8.8.8 from 192.168.1.4 via 192.168.1.2 dev eth0 
      cache 

[root@localhost ~]# ip route get to 8.8.8.8 from 192.168.1.5
  8.8.8.8 from 192.168.1.5 via 192.168.1.2 dev eth0 
      cache 

As we'd expect from "default ... dev eth0", packets destined to the public IP address will exit eth0.
Note it doesn't matter what the source IP address is.

Let's check to be sure though !
We'll sniff both eth0 and eth1 while we ping from either interface and either source address.
First, ping using eth0's IP address as the source (.4) :

[root@localhost ~]# tcpdump -ni eth0 'icmp' & tcpdump -ni eth1 'icmp' & ping -nc 3 -I 192.168.1.4 8.8.8.8 2>&1 > /dev/null ; sleep 4 ; pkill tcpdump
  [1] 2603
  [2] 2604
  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
  23:30:30.347429 IP 8.8.8.8 > 192.168.1.4: ICMP echo reply, id 2605, seq 1, length 64
  23:30:31.331631 IP 192.168.1.4 > 8.8.8.8: ICMP echo request, id 2605, seq 2, length 64
  23:30:31.350134 IP 8.8.8.8 > 192.168.1.4: ICMP echo reply, id 2605, seq 2, length 64
  23:30:32.333378 IP 192.168.1.4 > 8.8.8.8: ICMP echo request, id 2605, seq 3, length 64
  23:30:32.350145 IP 8.8.8.8 > 192.168.1.4: ICMP echo reply, id 2605, seq 3, length 64

  5 packets captured
  5 packets received by filter
  0 packets dropped by kernel

  0 packets captured
  0 packets received by filter
  0 packets dropped by kernel
  [1]-  Done                    tcpdump -ni eth0 'icmp'
  [2]+  Done                    tcpdump -ni eth1 'icmp'

Looks good - consistent with our routing table !
Nothing is seen on eth1 (the second summary) as we'd expect.
Now let's ping from source .5 (belonging to eth1) :

[root@localhost ~]# tcpdump -ni eth0 'icmp' & tcpdump -ni eth1 'icmp' & ping -nc 3 -I 192.168.1.5 8.8.8.8 2>&1 > /dev/null ; sleep 4 ; pkill tcpdump
  [1] 2609
  [2] 2610
  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
  23:32:31.284113 IP 8.8.8.8 > 192.168.1.5: ICMP echo reply, id 2611, seq 1, length 64
  23:32:32.269281 IP 192.168.1.5 > 8.8.8.8: ICMP echo request, id 2611, seq 2, length 64
  23:32:32.284493 IP 8.8.8.8 > 192.168.1.5: ICMP echo reply, id 2611, seq 2, length 64
  23:32:33.270735 IP 192.168.1.5 > 8.8.8.8: ICMP echo request, id 2611, seq 3, length 64
  23:32:33.286849 IP 8.8.8.8 > 192.168.1.5: ICMP echo reply, id 2611, seq 3, length 64

  5 packets captured
  5 packets received by filter
  0 packets dropped by kernel

  0 packets captured
  0 packets received by filter
  0 packets dropped by kernel
  [1]-  Done                    tcpdump -ni eth0 'icmp'
  [2]+  Done                    tcpdump -ni eth1 'icmp'

See how even though the source address of the pings was set to eth1's IP address, the packets left eth0 ?
But!, we can specify to ping from a source interface rather than a source address.
Specifying eth0 works as you'd expect (success) but something interesting happens if we set eth1 as the source :

[root@localhost ~]# tcpdump -ni eth0 'icmp' & tcpdump -ni eth1 'icmp' & ping -nc 3 -I eth1 8.8.8.8 2>&1 > /dev/null ; sleep 4 ; pkill tcpdump
  [1] 2751
  [2] 2752
  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

... What gives ?
WELL, we're only sniffing ICMP traffic. Because no default route (or more specific route to 8.8.8.8) exists off eth1, the destination is assumed to exist on the same broadcast domain.
That means it tries to get the destination's MAC address before it can even construct the packet to send. If it can't get a MAC address, it doesn't send the packet :

[root@localhost ~]# tcpdump -eni eth1 'icmp or arp' & ping -nc 3 -I eth1 8.8.8.8 2>&1 > /dev/null ; sleep 10 ; pkill tcpdump 
  [5] 2759
  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
  00:00:26.075685 08:00:27:48:e7:5d > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 8.8.8.8 tell 192.168.1.5, length 28
  00:00:26.075685 08:00:27:48:e7:5d > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 8.8.8.8 tell 192.168.1.5, length 28
  00:00:27.075945 08:00:27:48:e7:5d > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 8.8.8.8 tell 192.168.1.5, length 28
  00:00:27.075945 08:00:27:48:e7:5d > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 8.8.8.8 tell 192.168.1.5, length 28
  00:00:27.075945 08:00:27:48:e7:5d > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 8.8.8.8 tell 192.168.1.5, length 28
  00:00:28.077935 08:00:27:48:e7:5d > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 8.8.8.8 tell 192.168.1.5, length 28
  00:00:28.077935 08:00:27:48:e7:5d > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 8.8.8.8 tell 192.168.1.5, length 28
  00:00:28.077935 08:00:27:48:e7:5d > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 8.8.8.8 tell 192.168.1.5, length 28

(Google's DNS server probably isn't on your LAN, just as it's not on mine.)

Okay, okay, okay.
Now what if we replace the default route from out eth0 to out eth1 ?
We should expect our connectivity situation to be mirrored, right ? :

[root@localhost ~]# ip ro
  default via 192.168.1.2 dev eth0 
  192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.4 
  192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.5 

[root@localhost ~]# ping -nc 3 -I eth0 8.8.8.8
  PING 8.8.8.8 (8.8.8.8) from 192.168.1.4 eth0: 56(84) bytes of data.
  64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=17.5 ms
  64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=15.9 ms
  64 bytes from 8.8.8.8: icmp_seq=3 ttl=54 time=15.5 ms
  --- 8.8.8.8 ping statistics ---
  3 packets transmitted, 3 received, 0% packet loss, time 2002ms
  rtt min/avg/max/mdev = 15.547/16.331/17.526/0.864 ms

[root@localhost ~]# ping -nc 3 -I eth1 8.8.8.8
  PING 8.8.8.8 (8.8.8.8) from 192.168.1.5 eth1: 56(84) bytes of data.
  From 192.168.1.5 icmp_seq=1 Destination Host Unreachable
  From 192.168.1.5 icmp_seq=2 Destination Host Unreachable
  From 192.168.1.5 icmp_seq=3 Destination Host Unreachable
  --- 8.8.8.8 ping statistics ---
  3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 1999ms
  pipe 3

[root@localhost ~]# ip route replace default via 192.168.1.1 dev eth1 

[root@localhost ~]# ip ro
  default via 192.168.1.1 dev eth1 
  192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.4 
  192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.5 

[root@localhost ~]# ping -nc 3 -I eth0 8.8.8.8
  PING 8.8.8.8 (8.8.8.8) from 192.168.1.4 eth0: 56(84) bytes of data.
  From 192.168.1.4 icmp_seq=1 Destination Host Unreachable
  From 192.168.1.4 icmp_seq=2 Destination Host Unreachable
  From 192.168.1.4 icmp_seq=3 Destination Host Unreachable
  --- 8.8.8.8 ping statistics ---
  3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 1999ms
  pipe 3

[root@localhost ~]# ping -nc 3 -I eth1 8.8.8.8
  PING 8.8.8.8 (8.8.8.8) from 192.168.1.5 eth1: 56(84) bytes of data.
  64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=14.7 ms
  64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=18.8 ms
  64 bytes from 8.8.8.8: icmp_seq=3 ttl=54 time=21.0 ms
  --- 8.8.8.8 ping statistics ---
  3 packets transmitted, 3 received, 0% packet loss, time 2003ms
  rtt min/avg/max/mdev = 14.760/18.216/21.010/2.596 ms

And so it does ! Now eth0 can't reach anything but eth1 can.

Assuming you want both interfaces to "just work", let's go down the road of madness that is load-balancing ... :

[root@localhost ~]# ip route delete default

[root@localhost ~]# ip route add default scope global nexthop via 192.168.1.2 dev eth0 weight 1 nexthop via 192.168.1.1 dev eth1 weight 1

[root@localhost ~]# ip ro
  default 
          nexthop via 192.168.1.2  dev eth0 weight 1
          nexthop via 192.168.1.1  dev eth1 weight 1
  192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.4 
  192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.5 

But does it work ?

[root@localhost ~]# ping -nc 3 -I eth0 8.8.8.8
  PING 8.8.8.8 (8.8.8.8) from 192.168.1.4 eth0: 56(84) bytes of data.
  64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=17.9 ms
  64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=16.2 ms
  64 bytes from 8.8.8.8: icmp_seq=3 ttl=54 time=17.4 ms
  --- 8.8.8.8 ping statistics ---
  3 packets transmitted, 3 received, 0% packet loss, time 2003ms
  rtt min/avg/max/mdev = 16.281/17.238/17.986/0.727 ms

[root@localhost ~]# ping -nc 3 -I eth1 8.8.8.8
  PING 8.8.8.8 (8.8.8.8) from 192.168.1.5 eth1: 56(84) bytes of data.
  64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=16.6 ms
  64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=17.3 ms
  64 bytes from 8.8.8.8: icmp_seq=3 ttl=54 time=26.0 ms
  --- 8.8.8.8 ping statistics ---
  3 packets transmitted, 3 received, 0% packet loss, time 2003ms
  rtt min/avg/max/mdev = 16.612/20.025/26.091/4.300 ms

Huzzah !

... Though, even I'm a little surprised.
Your own milage may vary. I'd advise against this madness.

In my testing, things get wonky when trying to reach destinations on the local subnet.
This is because in normal system use (i.e., not specifying an exit interface explicitly), one interface is chosen.
For example, if I ping a non-existant host (.17) on the subnet, ARP requests are only sent out eth0.
So in theory if the host did exist off eth1, would ARP even work ?
Likely not :

[root@localhost ~]# ping 192.168.1.17
  PING 192.168.1.17 (192.168.1.17) 56(84) bytes of data.
  From 192.168.1.4 icmp_seq=1 Destination Host Unreachable
  From 192.168.1.4 icmp_seq=2 Destination Host Unreachable
  From 192.168.1.4 icmp_seq=3 Destination Host Unreachable
  ...

[root@localhost ~]# tcpdump -eni eth0 'arp'
  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
  00:25:29.167575 08:00:27:f6:8f:c2 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.17 tell 192.168.1.4, length 28
  00:25:30.168001 08:00:27:f6:8f:c2 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.17 tell 192.168.1.4, length 28
  00:25:31.169967 08:00:27:f6:8f:c2 > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.17 tell 192.168.1.4, length 28
  ...

[root@localhost ~]# tcpdump -eni eth1 'arp'
  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes



  ^C
  0 packets captured
  0 packets received by filter
  0 packets dropped by kernel

Cheers.

Comments

[Splash]

I agree if the destination IPs are different, then routing table can be configured to accomplish what I wanted. What if the destination IP are same on both side? I have updated my question. Thanks

[robut]

I've read your edits. Besides the ambiguity of what you mean to ask ("What if PC0 and PC1 have same IP?" is not a fully-formed question - "what if" what, exactly ?), why on earth do you want such a convoluted setup ?

[Splash]

scenario: PC0 side is assigned by me and I preassign the IP for eth0 before I ship the computer and the (embedded) system. PC1 side has many devices, and all belong to my customer and what if they assigned an IP that equal to PC0?

[Greg0ry]

I like this answer, it shows how to approach the problem. Thanks!