Almost working: How to set up iptables and iproute2 rules on dual WAN linux server?

Question

I have a Raspberry Pi (running Raspbian) with three network interfaces (one ethernet and two USB wifi nics). What I want to achieve is this: I have two WAN connections to the internet (DSL + Cable), and I want to set up the RasPi in such a way that it offers two default gateways. I want to make specific clients in my local network use the one or the other gateway. Here's an overview of what I want to achieve:

Usually, you can't have two default gateways for obvious reasons, but I found out that with iproute2 you can do exactly that and use multiple routing tables (and thereby default gateways) which are applied by rules. (See details of my configuration below)

The ethernet interface (192.168.10.1) is connected to a switch that connects my local network (192.168.10.0/24).

Interface wlan0 (192.168.178.199) is connected to the first WLAN router (DSL) (192.168.178.1). Interface wlan1 (192.168.0.199) is connected to the second router (Cable) (192.168.0.1).

Here's my /etc/network/interfaces

auto lo
iface lo inet loopback

iface eth0 inet static
address 192.168.10.1
netmask 255.255.255.0

iface wlan0 inet static
address 192.168.178.199
netmask 255.255.255.0
wpa-ssid "ALICE"
wpa-psk "yyyyyyyy"

iface wlan1 inet static
address 192.168.0.199
netmask 255.255.255.0
wpa-ssid "BOB"
wpa-psk "xxxxxxxx"

For some reason setting auto for the two wlan interfaces didn't work so well, most of the time at least one of them wouldn't get "upped", so I put the ifup's for them into rc.local for now. When I boot up, both wlan interfaces are up, connected to their wlans and both can ping their routers (e.g. ping 192.168.178.1 -I wlan0).

Next step was configuring iproute2. Here's my /etc/iproute2/rt_tables

#
# reserved values
#
255 local
254 main
253 default
0   unspec
#
# local
#
#1  inr.ruhep
1 alice
2 bob

After defining these two tables, I created these ip settings which are applied on each boot after all NICs are up:

ip route add 192.168.178.0/24 dev wlan0 src 192.168.178.199 table alice
ip route add default via 192.168.178.1 table alice
ip route add 192.168.0.0/24 dev wlan1 src 192.168.0.199 table bob
ip route add default via 192.168.0.1 table bob
ip rule add from 192.168.178.199 table alice
ip rule add from 192.168.0.199 table bob
ip route add default scope global nexthop via 192.168.178.1 dev wlan0 weight 1 nexthop via 192.168.0.1 dev wlan1 weight 1

For testing purposes, I set Google's nameserver (8.8.8.8) as default for now in /etc/resolv.conf.

At this moment, I can succesfully ping hosts on the internet from the RasPi, the requests get are being sent evenly spreaded via the one or the other gateway (I checked that with ping -R 8.8.8.8).

Now my question(s):

1. What do I have to do to make the clients (with a static ip, no DHCP needed) in my local 192.168.10.0/24 network use the RasPi as default gw (192.168.10.1 - the eth0 interface) to access the internet? I think that's done with masquerading and iptables, but I have no clue on how to do that with this setup.

2. How would I have to change the ip rules to not only let the clients connect to the internet just as the RasPi itself can, but instead use the one or the other gateway? For example, if I would like to make the host 192.168.10.100 use the "ALICE" gateway and 192.168.10.101 use "BOB"?

I feel quite close to the finishing line, any help on this probably not so common issue is greatly appreciated. Thanks in advance!

Regards, Rob

Update:

I made a iittle progress: after adding these rules...

ip rule add fwmark 1 table alice prio 1024
ip rule add fwmark 2 table bob prio 1025

...I am able (from 192.168.10.100) to ping not only the RasPi ethernet interface at 192.168.10.1, but also the wlan interface on 192.168.178.199. I can't ping the router behind that interface (192.168.178.1) or any hosts on the internet yet. But I think we're getting closer... So the packets arrive at the right wlan interface, but now the wlan nics need to forward them. How is that done?

Solution

route add default gw 192.168.10.1 on each of the clients

and

EDIT:

iptables -t mangle -A PREROUTING -s 192.168.10.100/32 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -s 192.168.10.101/32 -j MARK --set-mark 2

See also:

http://lartc.org/howto/lartc.rpdb.multiple-links.html

http://linux-ip.net/html/adv-multi-internet.html

Comments

[dmourati]

Much clearer now with the diagram and the wan links added.

[Rob]

Thanks for your answer, but that didn't do the job. I already added the gw to the clients and also the rules from above, but I still can't reach out. From 192.168.10.100 I can ping 192.168.10.1, but not 192.168.178.199 (wlan0 ip on RasPi) or 192.168.178.1 (Router). I think the missing link is some iptables magic that "connects" eth0 and the wlan interfaces.

[dmourati]

I don't see any public IPs anywhere, so a NAT doesn't really make sense here. There is some NAT going on upstream from your Pi presumably. You also need echo 1 >/proc/sys/net/ipv4_forward ;-)

[Rob]

Ip forwarding is and was already set to 1, but it does not help :-(

[dmourati]

ip route add nat 192.168.10.100 via 192.168.178.199

[Rob]

This throws an error: # ip route add nat 192.168.10.123 via 192.168.178.199 RTNETLINK answers: Invalid argument

[Rob]

From your linked "Split Access" example in your answer: "It will work for all processes running on the router itself, and for the local network, if it is masqueraded." - So far I have the same setup as in this example, I seem to just miss the right masquerading.

[dmourati]

linux-ip.net/html/ch-nat.html