Linux router: ping responses going through the wrong interface

Question

I have a Debian server acting as a router for a small network. It has 3 network interfaces, one LAN and 2 WANs (two different ISPs, let's call them A and B, each of them with a public static IP address).

The LAN has 5 subnets, some access the internet using ISP A and some use ISP B (plus there's a very simple script-based failover mechanism, which redirects all traffic through one ISP if the other one fails and an OpenVPN server).

This has been working well for quite some time. But up until now we had only used ISP A's public IP address to access the server from the outside. Now we need to be able to access through both public IP addresses, and I'm seeing ISP B's does not work.

Using tcpdump I've noticed that ping requests arrive to ISP B's network interface, but the responses don't make it back. Instead they seem to be going back through ISP A.

I suppose there's something wrong in my routing table, but I'm not sure what. Could you help me figure it out?

Here's how the router looks like (I've replaced the actual public IP addresses for 11.11.11.11 for ISP A and 22.22.22.22 for ISP B):

# ifconfig
eth1      Link encap:Ethernet  HWaddr 94:0c:6d:82:0d:98  
          inet addr:10.1.1.1  Bcast:10.1.1.255  Mask:255.255.255.0
          inet6 addr: fe80::960c:6dff:fe82:d98/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:70084654 errors:0 dropped:713 overruns:0 frame:0
          TX packets:87266365 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2955150829 (2.7 GiB)  TX bytes:3255030277 (3.0 GiB)
          Interrupt:20 Base address:0x2000 

eth1:0    Link encap:Ethernet  HWaddr 94:0c:6d:82:0d:98  
          inet addr:10.1.2.1  Bcast:10.1.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:20 Base address:0x2000 

eth1:1    Link encap:Ethernet  HWaddr 94:0c:6d:82:0d:98  
          inet addr:10.1.3.1  Bcast:10.1.3.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:20 Base address:0x2000 

eth1:2    Link encap:Ethernet  HWaddr 94:0c:6d:82:0d:98  
          inet addr:10.1.4.1  Bcast:10.1.4.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:20 Base address:0x2000 

eth1:3    Link encap:Ethernet  HWaddr 94:0c:6d:82:0d:98  
          inet addr:10.1.5.1  Bcast:10.1.5.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:20 Base address:0x2000 

eth3      Link encap:Ethernet  HWaddr 94:0c:6d:82:c8:72  
          inet addr:22.22.22.22  Bcast:22.22.22.255  Mask:255.255.255.0
          inet6 addr: fe80::960c:6dff:fe82:c872/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2013773 errors:0 dropped:0 overruns:0 frame:0
          TX packets:52720 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:128125141 (122.1 MiB)  TX bytes:4658309 (4.4 MiB)
          Interrupt:19 Base address:0x6000 

eth4      Link encap:Ethernet  HWaddr 6c:f0:49:84:79:ca  
          inet addr:11.11.11.11  Bcast:255.255.255.255  Mask:255.255.255.0
          inet6 addr: fe80::6ef0:49ff:fe84:79ca/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:57255186 errors:0 dropped:0 overruns:0 frame:0
          TX packets:39862172 errors:0 dropped:0 overruns:0 carrier:1
          collisions:0 txqueuelen:1000 
          RX bytes:1933578821 (1.8 GiB)  TX bytes:328150009 (312.9 MiB)
          Interrupt:27 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:952741 errors:0 dropped:0 overruns:0 frame:0
          TX packets:952741 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:116644740 (111.2 MiB)  TX bytes:116644740 (111.2 MiB)

tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:10.8.0.1  P-t-P:10.8.0.2  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:7907032 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6371185 errors:0 dropped:5 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:497588957 (474.5 MiB)  TX bytes:3980021182 (3.7 GiB)

# ip rule list
0:      from all lookup local 
32763:  from 10.1.5.0/24 lookup adsl 
32764:  from 10.1.3.0/24 lookup adsl 
32765:  from 10.1.2.0/24 lookup adsl 
32766:  from all lookup main 
32767:  from all lookup default  

# ip route show table main
10.8.0.2 dev tun0  proto kernel  scope link  src 10.8.0.1 
10.8.0.0/24 via 10.8.0.2 dev tun0 
11.11.11.0/24 dev eth4  proto kernel  scope link  src 11.11.11.11 
22.22.22.0/24 dev eth3  proto kernel  scope link  src 22.22.22.22 
10.1.4.0/24 dev eth1  proto kernel  scope link  src 10.1.4.1 
10.1.5.0/24 dev eth1  proto kernel  scope link  src 10.1.5.1 
10.1.1.0/24 dev eth1  proto kernel  scope link  src 10.1.1.1 
10.1.2.0/24 dev eth1  proto kernel  scope link  src 10.1.2.1 
10.1.3.0/24 dev eth1  proto kernel  scope link  src 10.1.3.1 
default via 11.11.11.1 dev eth4 
default via 22.22.22.1 dev eth3  metric 100 

# ip route show table adsl
10.8.0.2 dev tun0  proto kernel  scope link  src 10.8.0.1 
22.22.22.0/24 dev eth3  proto kernel  scope link  src 22.22.22.22 
11.11.11.0/24 dev eth4  proto kernel  scope link  src 11.11.11.11 
10.8.0.0/24 via 10.8.0.2 dev tun0 
10.1.4.0/24 dev eth1  proto kernel  scope link  src 10.1.4.1 
10.1.5.0/24 dev eth1  proto kernel  scope link  src 10.1.5.1 
10.1.1.0/24 dev eth1  proto kernel  scope link  src 10.1.1.1 
10.1.2.0/24 dev eth1  proto kernel  scope link  src 10.1.2.1 
10.1.3.0/24 dev eth1  proto kernel  scope link  src 10.1.3.1 
default via 22.22.22.1 dev eth3 

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.8.0.2        0.0.0.0         255.255.255.255 UH    0      0        0 tun0
10.8.0.0        10.8.0.2        255.255.255.0   UG    0      0        0 tun0
11.11.11.0      0.0.0.0         255.255.255.0   U     0      0        0 eth4
22.22.22.0      0.0.0.0         255.255.255.0   U     0      0        0 eth3
10.1.4.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
10.1.5.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
10.1.1.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
10.1.2.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
10.1.3.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
0.0.0.0         11.11.11.1   0.0.0.0         UG    0      0        0 eth4
0.0.0.0         22.22.22.1    0.0.0.0         UG    100    0        0 eth3

And here's the tcpdump test (here 22.22.22.22 is ISP B's IP and 99.99.99.99 is the IP of the remote box from which I'm pinging):

# Here's ISP B's interface, eth3
# tcpdump -i eth3 -qtln icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth3, link-type EN10MB (Ethernet), capture size 65535 bytes
IP 99.99.99.99 > 22.22.22.22: ICMP echo request, id 8099, seq 1, length 64
IP 99.99.99.99 > 22.22.22.22: ICMP echo request, id 8099, seq 2, length 64
IP 99.99.99.99 > 22.22.22.22: ICMP echo request, id 8099, seq 3, length 64
IP 99.99.99.99 > 22.22.22.22: ICMP echo request, id 8099, seq 4, length 64

# Here's ISP A's interface, eth4
# tcpdump -i eth4 -qtln icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth4, link-type EN10MB (Ethernet), capture size 65535 bytes
IP 22.22.22.22 > 99.99.99.99: ICMP echo reply, id 8099, seq 9, length 64
IP 22.22.22.22 > 99.99.99.99: ICMP echo reply, id 8099, seq 10, length 64
IP 22.22.22.22 > 99.99.99.99: ICMP echo reply, id 8099, seq 11, length 64
IP 22.22.22.22 > 99.99.99.99: ICMP echo reply, id 8099, seq 12, length 64

Solution 1

You forgot the rule for each interface. And better create 2 tables instead of main and adsl just for the better readability, eg. ispa and ispb

For eth4 add (you can do that with post-up in /etc/network/interface):

ip route add 11.11.11.12/32 dev eth4 src 11.11.11.11 table ispa
ip route add default via 11.11.11.12 table ispa
ip rule add from 11.11.11.11 table ispa

(exchange the 11.11.11.12/32 with the IP of your gateway).

Same goes for eth3 with it's addresses.

I don't know if the static routing for the subnets is wanted (it perhaps is), you should consider balancing the traffic between both wan interfaces.

If you need any help on this, this is a great help to setup routing for multiple uplinks:

http://www.debian-administration.org/article/377/Routing_for_multiple_uplinks

Solution 2

The behaviour you are experiencing is expected. This blogpost is a bit old but explains pretty well how to make it work as you want to: configuring-multiple-default-routes-in-linux

EDITED: Using default routes, your outgoing packets will be routed through the default route with biggest preferece, no matter which way they came in.

You need to use policy based routing, so I think that in your case you could just add this:

ip rule add from 11.11.11.11 table main
ip rule add from 22.22.22.22 table adsl

Check your rules are placed properly with ip rule list. Anyway in the end this is the same as @Broco's answer.

Comments

[El Barto]

Thanks for your answer. At this point, since the router is in production, I'd like to get this working only with the minimum possible changes (so I'd rather not create additional tables unless I have to). If I understand correctly, with my current scheme, what I would need to do is: ip route add default via 22.22.22.1 table adsl. Is that right? (I cannot test it right now, I need to wait until the end of office hours here).

[Broco]

Yep, this should work. However, you can still backup your old configuration and try the completely new setup as you don't have that many rules to follow. So if one connection fails you have a backup via routing.

[El Barto]

Thanks for your answer. I haven't been able to test it yet, because this router is in production and this seconary IP issue is not critical, but I look forward to testing your suggestion.