NAT box with multiple internal and external interfaces

Question

Similar questions have been asked before but my setup is a little different and the solutions to those questions are not working. A have a CentOS 6 server running iptables with 5 interfaces:

• eth0: Management 136.2.188.0/24
• eth1: Cluster1 internal 10.1.0.0/16
• eth2: Cluster1 external 136.2.217.96/27
• eth3: Cluster2 internal 10.6.0.0/20
• eth4: Cluster2 external 136.2.178.32/28

What I'm trying to do is to have traffic from eth1 to go out eth2 and be NATd, traffic from eth3 go out eth4 and be NATd, all other traffic (e.g. SSH to the box itself) use eth0.

To do that I configured route tables like so:

ip route add default via 136.2.178.33 src 136.2.178.37 table 1
ip route add default via 136.2.217.97 src 136.2.217.124 table 2

ip rule add fwmark 1 pref 1 table 1
ip rule add fwmark 2 pref 2 table 2

The source IPs are those of the NAT box. The regular default route the management interface will use is in table 0 as it usually is.

I then configured iptables to mark packets using the mangle table so that they use a specific route table (if I am understanding this correctly) and NAT particular source traffic to a particular interface:

iptables -A PREROUTING -t mangle -j CONNMARK --restore-mark
iptables -A PREROUTING -t mangle -m mark --mark 0x0 -s 10.6.0.0/20 -j MARK --set-mark 1
iptables -A PREROUTING -t mangle -m mark --mark 0x0 -s 10.1.0.0/16 -j MARK --set-mark 2
iptables -A POSTROUTING -t mangle -j CONNMARK --save-mark
iptables -A POSTROUTING -t nat -s 10.6.0.0/20 -o eth4 -j MASQUERADE
iptables -A POSTROUTING -t nat -s 10.1.0.0/16 -o eth2 -j MASQUERADE
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -j LOG --log-level debug
iptables -A FORWARD -m state --state NEW -s 10.6.0.0/20 -o eth4 -j ACCEPT
iptables -A FORWARD -m state --state NEW -s 10.1.0.0/16 -o eth2 -j ACCEPT
iptables -A FORWARD -j DROP

When I test this (a simple wget of google.com from a client machine) I can see traffic come in the internal interface (eth3 in the test), then go out the external interface (eth4) with the NAT box's external IP as the source IP. So, the NAT itself works. However, when the system receives the response packet it comes in eth4 as it should but then nothing happens, it never gets un-NAT'd and never shows up on eth3 to go back to the client machine.

Internal interface:

11:52:08.570462 IP 10.6.0.50.50783 > 74.125.198.103.80: Flags [S], seq 4030201376, win 14600, options [mss 1460,sackOK,TS val 34573 ecr 0,nop,wscale 7], length 0
11:52:09.572867 IP 10.6.0.50.50783 > 74.125.198.103.80: Flags [S], seq 4030201376, win 14600, options [mss 1460,sackOK,TS val 35576 ecr 0,nop,wscale 7], length 0
11:52:11.576943 IP 10.6.0.50.50783 > 74.125.198.103.80: Flags [S], seq 4030201376, win 14600, options [mss 1460,sackOK,TS val 37580 ecr 0,nop,wscale 7], length 0
11:52:15.580846 IP 10.6.0.50.50783 > 74.125.198.103.80: Flags [S], seq 4030201376, win 14600, options [mss 1460,sackOK,TS val 41584 ecr 0,nop,wscale 7], length 0
11:52:23.596897 IP 10.6.0.50.50783 > 74.125.198.103.80: Flags [S], seq 4030201376, win 14600, options [mss 1460,sackOK,TS val 49600 ecr 0,nop,wscale 7], length 0

External interfaces:

11:52:08.570524 IP 136.2.178.37.50783 > 74.125.198.103.80: Flags [S], seq 4030201376, win 14600, options [mss 1460,sackOK,TS val 34573 ecr 0,nop,wscale 7], length 0
11:52:08.609213 IP 74.125.198.103.80 > 136.2.178.37.50783: Flags [S.], seq 1197168065, ack 4030201377, win 42540, options [mss 1380,sackOK,TS val 1835608368 ecr 34573,nop,wscale 7], length 0
11:52:08.909188 IP 74.125.198.103.80 > 136.2.178.37.50783: Flags [S.], seq 1197168065, ack 4030201377, win 42540, options [mss 1380,sackOK,TS val 1835608668 ecr 34573,nop,wscale 7], length 0
11:52:09.572882 IP 136.2.178.37.50783 > 74.125.198.103.80: Flags [S], seq 4030201376, win 14600, options [mss 1460,sackOK,TS val 35576 ecr 0,nop,wscale 7], length 0
11:52:09.611414 IP 74.125.198.103.80 > 136.2.178.37.50783: Flags [S.], seq 1197168065, ack 4030201377, win 42540, options [mss 1380,sackOK,TS val 1835609370 ecr 34573,nop,wscale 7], length 0
11:52:11.576967 IP 136.2.178.37.50783 > 74.125.198.103.80: Flags [S], seq 4030201376, win 14600, options [mss 1460,sackOK,TS val 37580 ecr 0,nop,wscale 7], length 0

So, why is traffic getting out but iptables is not sending the return traffic back to the client? It seems that routing is correct since packets leave and arrive on the correct interfaces, so what is iptables doing with the return traffic?

Solution

OK, I figured it out. What I had to do was add the internal subnet route to each route table then set rules to control what interface traffic routes to/from. Then in iptables marking packets with the mangle table was not needed, just the typical forward and nat rules.

ip route add 136.2.178.32/28 dev eth4 table 1
ip route add 10.6.0.0/20 dev eth3 table 1
ip route add default via 136.2.178.33 src 136.2.178.37 table 1
ip rule add iif eth4 table 1
ip rule add from 10.6.0.0/20 table 1

ip route add 136.2.217.96/28 dev eth2 table 2
ip route add 10.1.0.0/16 dev eth1 table 2
ip route add default via 136.2.217.113 src 136.2.217.124 table 2
ip rule add iif eth2 table 2
ip rule add from 10.1.0.0/16 table 2

iptables -A FORWARD -i eth2 -o eth1 -m state --state RELATED,ESTABLISHED -j     ACCEPT
iptables -A FORWARD -i eth1 -o eth2 -m state --state NEW -j LOG --log-level debug
iptables -A FORWARD -i eth1 -o eth2 -j ACCEPT
iptables -A FORWARD -i eth4 -o eth3 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i eth3 -o eth4 -m state --state NEW -j LOG --log-level debug
iptables -A FORWARD -i eth3 -o eth4 -j ACCEPT
iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited
iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth4 -j MASQUERADE