iproute rt_table and mark not working on linux

Question

I have this configuration which comes from official and unofficial guides and questions readings here and a lot of failed tests. CentOS 7 and Ubuntu server 15 (LAMP and only eth0).

/etc/iproute2/rt_tables

    1   tunnel0

I PREPARE THE ROUTES AND MARKS

ip route add 0.0.0.0/0 dev tun0 table 1
ip rule add from all fwmark 1 table 1
ip route flush cache

(also tried to use table ID, to declare via "10.123.123.x" with the tun0 address and the tun0 gateway...)

IPTABLES

iptables -A PREROUTING -t mangle -p tcp --dport 80 -j MARK --set-mark 1
iptables-save

I see the eth0 ISP public IP, instead of tun0, which I can see forcing a "route add" static route. What am I missing? Thank you.

Solution

I solved and documented it here: http://aftermanict.blogspot.it/2015/11/bash-iptables-iproute2-and-multiple.html

This will make the kernel permanently route packets, enables multiple routes and even for networks not attested on the machine:

nano /etc/sysctl.conf

net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
net.ipv4.ip_forward = 1

for f in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 0 >| $f ; done

This will initialize iptables and in particular mangle and nat, which are needed for marking the traffic:

iptables -F
iptables -t nat -F
iptables -t mangle -F
iptables -X

add the alternative routes editing:

nano /etc/iproute2/rt_tables

Add (names are your references):

1 tunnel0
2 tunnel1

add routes and rules, we use tables IDs instead of names which are more immediate. As you can notice, the gateway is irrelevant, especially for tunnels which can have dynamic gateways:

ip route add 0.0.0.0/0 dev tun0 table 1
ip route add 0.0.0.0/0 dev tun1 table 2

add rules to mark traffic and bind to the corresponding table:

ip rule add from all fwmark 1 table 1
ip rule add from all fwmark 2 table 2
ip route flush cache

check if you like:

ip route show table 1
ip route show table 2
ip rule show

if you miss something, you can delete this way:

ip rule del table 1
ip route flush table 1

NOW THE MISSING PART: THIS WONT WORK:

iptables -A PREROUTING -t mangle -p tcp --dport 80 -j MARK --set-mark 1

THIS WILL:

iptables -A OUTPUT -t mangle -p tcp --dport 80 -j MARK --set-mark 1
iptables-save

Do you need to select traffic and push it simultaneously in a device / tunnel? No problem, I solved this too:

iptables -A OUTPUT -t mangle -p tcp --dport 10001 -j MARK --set-mark 1
iptables -A OUTPUT -t mangle -p tcp --dport 10002 -j MARK --set-mark 2
iptables -t nat -A OUTPUT -p tcp --dport 10001 -j DNAT --to :80
iptables -t nat -A OUTPUT -p tcp --dport 10002 -j DNAT --to :80

NAT mandatory for reply

iptables -t nat -A POSTROUTING -o $DEV1 -j MASQUERADE
iptables -t nat -A POSTROUTING -o $DEV2 -j MASQUERADE

iptables-save

Comments

[EEAA]

Link-only answers aren't useful nor appreciated here (even when answering your own question). Please provide content here and link elsewhere for more details if necessary.

[fab]

Posted the full solution for user convenience

[EEAA]

Thank you. FYI it's not only for convenience, but also to ensure that your answer remains helpful for future readers, even after your blog disappears.

[Enjoy87]

THIS WORKED! It was the natting that totally messed me around!

[fab]

@Enjoy87 you can be grateful by rating my solution with 1 click :-)

[pgr]

I got mine working with PREROUTING (which this answer says doesn't work)... I'm not sure why my case is different.