Source Based Policy Routing & NAT (DNAT/SNAT) aka Multi WANs on CentOS 5

Question

Originally posted at Unix and Linux but nobody was able to answer it, so m migrating the question here:

My question is regarding Source Based Policy Routing on CentOS 5 with 2 WANs plus a LAN (NAT) port with Load Balancing, but first before anything some remarks before starting to describe the problem...

I know this topic has been brought many times here at stack exchange and seems that the top 5 answers are (ordered from the most to the least):

1. Disable rp_filter
2. Mark/Connmark based policy routing
3. IP Based policy routing (add more IPs)
4. Install pfSense, Shorewall, Ubuntu?, Etc...
5. Buy expensive Cisco/3com/Juniper/Etc... Router

Most of the times, some of this answers are correct but for me solutions 1 & 2 haven't workout (i don't discard at least point 2 cause i may have some issues with my setup), point 3 is basically isolating a problem rather than solving it (also adds complexity to the routing tables) and solutions 4 & 5 are just out of the scope since i don't have resources to buy specialized hardware nor can take offline the server since it is on production so to summarize replacing the CentOS server with something "better" is off the table.

Ok now back in the problem, lets first describe the current setup...

Interfaces:

eth1: IP: 10.0.0.1, GW: 10.0.0.1, NM: 255.255.255.0 (LAN)
eth0: IP: 10.0.1.1, GW: 10.0.1.254, NM: 255.255.255.0 (ISP1 - ADSL Router)
eth2: IP: 10.0.2.1, GW: 10.0.2.254, NM: 255.255.255.0 (ISP2 - ADSL Router)

/etc/sysctl.conf:

# Controls IP packet forwarding
net.ipv4.ip_forward = 1

# Controls source route verification
net.ipv4.conf.default.rp_filter = 0

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_messages = 1

/etc/iproute2/rt_tables:

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
2 ISP1
3 ISP2

/etc/sysconfig/network-scripts/route-eth0:

10.0.1.0/24 dev eth0 src 10.0.1.1 table ISP1
default via 10.0.1.254 dev eth0 table ISP1

/etc/sysconfig/network-scripts/route-eth2:

10.0.2.0/24 dev eth2 src 10.0.2.1 table ISP2
default via 10.0.2.254 dev eth2 table ISP2

/etc/sysconfig/network-scripts/rule-eth0:

fwmark 2 table ISP1
from 10.0.1.1 table ISP1

/etc/sysconfig/network-scripts/rule-eth2:

fwmark 3 table ISP2
from 10.0.2.1 table ISP2

/etc/sysconfig/iptables:

*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# Basic Rules
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT

# SSH
-A INPUT -i eth0 -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -i eth2 -m tcp -p tcp --dport 22 -j ACCEPT

# OpenVPN
-A INPUT -i eth0 -m udp -p udp --dport 1194 -j ACCEPT
-A INPUT -i eth2 -m udp -p udp --dport 1194 -j ACCEPT

# Allow everything from LAN
-A INPUT -i eth1 -j ACCEPT

# Allow everything from the VPN
-A INPUT -i tun0 -j ACCEPT

# Default Drop on everything else
-A INPUT -j DROP

# Allow forwarding from LAN and VPN
-A FORWARD -i eth1 -j ACCEPT
-A FORWARD -i tun0 -j ACCEPT

# Allow all outbound traffic
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -o eth1 -j ACCEPT
COMMIT

*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# DNAT to Developer Box (SSH Server)
-A PREROUTING -i eth0 -p tcp -m tcp --dport 2222 -j DNAT --to-destination 10.0.0.200:2222
-A PREROUTING -i eth2 -p tcp -m tcp --dport 2222 -j DNAT --to-destination 10.0.0.200:2222

# SNAT
-A POSTROUTING -o eth0 -j SNAT --to-source 10.0.1.1
-A POSTROUTING -o eth2 -j SNAT --to-source 10.0.2.1
COMMIT

*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
# Mark Based Routing? (based on NerdBoys site)
-A PREROUTING -j CONNMARK --restore-mark
-A PREROUTING --match mark --mark 2 -j ACCEPT
-A PREROUTING -i eth0 -j MARK --set-mark 2
-A PREROUTING --match mark --mark 3 -j ACCEPT
-A PREROUTING -i eth2 -j MARK --set-mark 3
-A PREROUTING -j CONNMARK --save-mark
COMMIT

Load Balancing is being possible with the gwping bash script which basically monitors the 2 wans (eth0 and eth2) and set up the default routes and weights in the server like this (while in load balance or the 2 wans up and running):

ip route replace default scope global nexthop via 10.0.1.1 dev eth0 weight 1 nexthop via 10.0.2.1 dev eth1 weight 1

The problem i have is that even with this setup that lot of people seems to agree is the correct one, m still having issues with accessing services inside the network from the outside (specifically the ssh developer box and the OpenVPN one) even that the packets are being "marked" and routed accordingly the answer from the dev box goes always to the wrong path. I don't know if m missing something in the mangle or nat area or m misunderstanding source based routing at all, anyway if someone know how to make this work accordingly it will be kindly appreciated.

My sources for this setup are:

lartc.org/lartc.html#LARTC.RPDB.MULTIPLE-LINKS
fatalsite.net/?p=90
nerdboys.com/2006/05/05/conning-the-mark-multiwan-connections-using-iptables-mark-connmark-and-iproute2/
policyrouting.org/PolicyRoutingBook/ONLINE/CH08.web.html
unix.stackexchange.com/questions/58635/iptables-set-mark-route-diferent-ports-through-different-interfaces
unix.stackexchange.com/questions/22770/two-interfaces-two-addresses-two-gateways
bulma.net/body.phtml?nIdNoticia=2145

Kindest Regards

PS1: I found a website that its saying that the marks in the routing table should be + 1 different from the iptables marks (kim.attr.ee/2010/08/source-based-policy-routing-on-centos.html) is this true? or this website is super-incorrect.

---

Update 15/08/2013 22:15

After more researching and debugging, i found a website which says that i forgot to add the SNAT part at the post-routing table so i add this rules to the iptables config:

-A POSTROUTING --match mark --mark 2 -j SNAT --to-source 10.0.1.1
-A POSTROUTING --match mark --mark 3 -j SNAT --to-source 10.0.2.1

But m still unable to connect to the devbox from the outside the network. On the good side an a iptables -t nat -nvL POSTROUTING gives a hint about the workings of connmark based policy routing, so maybe its something related at the ISP1 and ISP2 router edge:

Chain POSTROUTING (policy ACCEPT 520 packets, 56738 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 SNAT       all  --  *      *       0.0.0.0/0            0.0.0.0/0           MARK match 0x2 to:10.0.1.1
    6   312 SNAT       all  --  *      *       0.0.0.0/0            0.0.0.0/0           MARK match 0x3 to:10.0.2.1
  903 70490 SNAT       all  --  *      eth0    0.0.0.0/0            0.0.0.0/0           to:10.0.1.1
  931 78070 SNAT       all  --  *      eth2    0.0.0.0/0            0.0.0.0/0           to:10.0.2.1

Also i add more info from my setup, please somebody throw me a life-saver since m starting to ran out of ideas... >.<

ip route show:

10.8.0.2 dev tun0  proto kernel  scope link  src 10.8.0.1
10.0.2.0/24 dev eth2  proto kernel  scope link  src 10.0.2.1
10.0.0.0/24 dev eth1  proto kernel  scope link  src 10.0.0.1
10.8.0.0/24 via 10.8.0.2 dev tun0
10.0.1.0/24 dev eth0  proto kernel  scope link  src 10.0.1.1
169.254.0.0/16 dev eth2  scope link
default
        nexthop via 10.0.1.254  dev eth0 weight 1
        nexthop via 10.0.2.254  dev eth2 weight 1

ip rule show:

0:      from all lookup 255
1024:   from all fwmark 0x2 lookup ISP1
1025:   from all fwmark 0x3 lookup ISP2
2024:   from 10.0.1.1 lookup ISP1
2025:   from 10.0.2.1 lookup ISP2
32766:  from all lookup main
32767:  from all lookup default

New Sources:

sarcasmasaservice.com/2013/04/linux-routing-capabilities-my-abuse-thereof/

Kindest Regards

Solution

Well...

After thousands of hours debugging, trying different setups and 72 hours of heavy testing in production, i was able to found the correct solution/setup, the problem was in the iptables rules (mangle section) the packets apparently got marked right when they come in but when they come out there wasn't any for dnat'ted packets, anyway here it is my the final working solution to my problem:

/etc/sysconfig/iptables:

*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# Basic Rules
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT

# SSH
-A INPUT -i eth0 -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -i eth2 -m tcp -p tcp --dport 22 -j ACCEPT

# OpenVPN
-A INPUT -i eth0 -m udp -p udp --dport 1194 -j ACCEPT
-A INPUT -i eth2 -m udp -p udp --dport 1194 -j ACCEPT

# Allow everything from LAN
-A INPUT -i eth1 -j ACCEPT

# Allow everything from the VPN
-A INPUT -i tun0 -j ACCEPT

# Default Drop on everything else
-A INPUT -j DROP

# Allow forwarding from LAN and VPN
-A FORWARD -i eth1 -j ACCEPT
-A FORWARD -i tun0 -j ACCEPT

# Allow all outbound traffic
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -o eth1 -j ACCEPT
COMMIT

*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# DNAT to Developer Box (SSH Server)
-A PREROUTING -i eth0 -p tcp -m tcp --dport 2222 -j DNAT --to-destination 10.0.0.200:2222
-A PREROUTING -i eth2 -p tcp -m tcp --dport 2222 -j DNAT --to-destination 10.0.0.200:2222

# SNAT
-A POSTROUTING -o eth0 -j SNAT --to-source 10.0.1.1
-A POSTROUTING -o eth2 -j SNAT --to-source 10.0.2.1
COMMIT

*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
# CONNMARK Source Based Routing
-A PREROUTING -i eth0 -m state --state NEW,RELATED,ESTABLISHED -d 10.0.1.1 -j CONNMARK --set-mark 0x2
-A PREROUTING -i eth2 -m state --state NEW,RELATED,ESTABLISHED -d 10.0.2.1 -j CONNMARK --set-mark 0x3
-A PREROUTING -i eth1 -m connmark --mark 0x2 -j CONNMARK --restore-mark
-A PREROUTING -i eth1 -m connmark --mark 0x3 -j CONNMARK --restore-mark
-A OUTPUT -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark
COMMIT

Obviously plus all the previous setup related to iproute and gwping (for link load balancing & fail-over), the solutions were made possible thanks to sources [1] & [2], both which pointed me to a different part (Luca Gibelli for the PREROUTING part and Karl Bowden for the OUTPUT part) of the solution, also m living here some more sources for other websites that pointed me to the right direction to go looking. Hope this help another sysadmin in the future.

Kindest Regards

Sources:

[1]www.nervous.it/2010/09/dnat-and-ip-source-routing-woes/
[2]blog.khax.net/2009/12/01/multi-gateway-balancing-with-iptables/
[3]home.regit.org/netfilter-en/links-load-balancing/
[4]mailman.ds9a.nl/pipermail/lartc/2006q2/018964.html
[5]web.archive.org/web/20120320115329/http://versa.net.au/index.php?option=com_content&task=view&id=21&Itemid=34

Update 10/10/2013

OpenVPN requires an extra configuration directive to work with a multiple wan setup (like the previous one), so just add the multihome option in your server.conf (OpenVPN >= 2.1, for lower versions just change the local directive to only listen in a particular ip) and you are good to go.

Comments

[Hvisage]

Thanks, the -t mangle -A OUTPUT ... -j CONNMARK --restore-mark was something I was missing in my setup :)