"ip rule to" works but "ip rule fwmark" fails - why?
SOLVED!
With detailed logging I found out that packets were dropped after mangle/PREROUTING and not entering nat/PREROUTING. This is because of Kernel default handling of "unwanted" packets.
An
# echo 0 >/proc/sys/net/ipv4/conf/$interface/rp_filter
disables this filter mechanism and everything works fine again.
Related videos on Youtube
Dmitry Donskih
Updated on September 18, 2022Comments
-
Dmitry Donskih almost 2 years
I have a CentOS 6 (kernel 2.6.32) router with working OpenVPN client on it, and I want to redirect some traffic via VPN server.
Client (
192.168.60.159
) sends request to router (192.168.60.6:1443
), and router redirects it via VPN connection (10.200.0.54
) to server (185.61.149.21:443
).I've created a specific routing table
tunde
and a rule to use this table on packets MARKed by iptables. This rule has default GW on VPN while main routing table has real GW. When I try to useip rule add fwmark 0x64 lookup tunde
, it fails. Request passes to server OK, server replies, router receives reply but doesn't pass it to client.But, if I add
ip rule add to 185.61.149.21 lookup tunde
- all works perfect. (But this rule doesn't satisfy my needs, i need per-port routing)Looks like iptables somehow cannot de-masquerade replies.
Any ideas? Thank you!
#ip rule ls 32765: from all fwmark 0x64 lookup tunde
This routing table has default gateway on VPN:
#ip route ls table tunde 10.200.0.53 dev tunde proto kernel scope link src 10.200.0.54 192.168.60.0/24 dev eth0 proto kernel scope link src 192.168.60.6 default via 10.200.0.53 dev tunde src 10.200.0.54
While main routing table has real default gateway:
# ip route ls 10.200.0.53 dev tunde proto kernel scope link src 10.200.0.54 192.168.60.0/24 dev eth0 proto kernel scope link src 192.168.60.6 default via 192.168.60.1 dev eth0
Tcpdump shows both requests and replies on TUN interface:
# tcpdump -i tunde -nn tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tunde, link-type RAW (Raw IP), capture size 65535 bytes 15:50:37.136708 IP 10.200.0.54.51409 > 185.61.149.21.443: Flags [S], seq 302961260, win 65280, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0 15:50:37.209278 IP 185.61.149.21.443 > 10.200.0.54.51409: Flags [S.], seq 390829204, ack 302961261, win 29200, options [mss 1357,nop,nop,sackOK,nop,wscale 9], length 0 15:50:38.219458 IP 185.61.149.21.443 > 10.200.0.54.51409: Flags [S.], seq 390829204, ack 302961261, win 29200, options [mss 1357,nop,nop,sackOK,nop,wscale 9], length 0 15:50:40.136933 IP 10.200.0.54.51409 > 185.61.149.21.443: Flags [S], seq 302961260, win 65280, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0 15:50:40.182989 IP 185.61.149.21.443 > 10.200.0.54.51409: Flags [S.], seq 390829204, ack 302961261, win 29200, options [mss 1357,nop,nop,sackOK,nop,wscale 9], length 0 15:50:42.191772 IP 185.61.149.21.443 > 10.200.0.54.51409: Flags [S.], seq 390829204, ack 302961261, win 29200, options [mss 1357,nop,nop,sackOK,nop,wscale 9], length 0 15:50:42.892051 IP 185.61.149.21.443 > 10.200.0.54.51391: Flags [S.], seq 528990609, ack 3345061728, win 29200, options [mss 1357,nop,nop,sackOK,nop,wscale 9], length 0
In iptables' log I also see replies. All packets are MARKed and use correct routing table:
# tail -f /var/log/messages Oct 10 15:50:37 toy2 kernel: INPUT from Cli: IN=eth0 OUT= MAC=00:15:5d:3c:bc:03:1c:39:47:f0:74:87:08:00 SRC=192.168.60.159 DST=192.168.60.6 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=19050 DF PROTO=TCP SPT=51409 DPT=1443 WINDOW=65280 RES=0x00 SYN URGP=0 MARK=0x64 Oct 10 15:50:37 toy2 kernel: Forward To TUN: IN=eth0 OUT=tunde SRC=192.168.60.159 DST=185.61.149.21 LEN=52 TOS=0x00 PREC=0x00 TTL=127 ID=19050 DF PROTO=TCP SPT=51409 DPT=443 WINDOW=65280 RES=0x00 SYN URGP=0 MARK=0x64 Oct 10 15:50:37 toy2 kernel: OUTPUT To TUN: IN= OUT=tunde SRC=192.168.60.159 DST=185.61.149.21 LEN=52 TOS=0x00 PREC=0x00 TTL=127 ID=19050 DF PROTO=TCP SPT=51409 DPT=443 WINDOW=65280 RES=0x00 SYN URGP=0 MARK=0x64 Oct 10 15:50:37 toy2 kernel: INPUT from TUN: IN=tunde OUT= MAC= SRC=185.61.149.21 DST=10.200.0.54 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=443 DPT=51409 WINDOW=29200 RES=0x00 ACK SYN URGP=0 MARK=0x64 Oct 10 15:50:38 toy2 kernel: INPUT from TUN: IN=tunde OUT= MAC= SRC=185.61.149.21 DST=10.200.0.54 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=443 DPT=51409 WINDOW=29200 RES=0x00 ACK SYN URGP=0 MARK=0x64 Oct 10 15:50:40 toy2 kernel: INPUT from Cli: IN=eth0 OUT= MAC=00:15:5d:3c:bc:03:1c:39:47:f0:74:87:08:00 SRC=192.168.60.159 DST=192.168.60.6 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=19063 DF PROTO=TCP SPT=51409 DPT=1443 WINDOW=65280 RES=0x00 SYN URGP=0 MARK=0x64 Oct 10 15:50:40 toy2 kernel: Forward To TUN: IN=eth0 OUT=tunde SRC=192.168.60.159 DST=185.61.149.21 LEN=52 TOS=0x00 PREC=0x00 TTL=127 ID=19063 DF PROTO=TCP SPT=51409 DPT=443 WINDOW=65280 RES=0x00 SYN URGP=0 MARK=0x64 Oct 10 15:50:40 toy2 kernel: INPUT from TUN: IN=tunde OUT= MAC= SRC=185.61.149.21 DST=10.200.0.54 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=443 DPT=51409 WINDOW=29200 RES=0x00 ACK SYN URGP=0 MARK=0x64 Oct 10 15:50:42 toy2 kernel: INPUT from TUN: IN=tunde OUT= MAC= SRC=185.61.149.21 DST=10.200.0.54 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=443 DPT=51409 WINDOW=29200 RES=0x00 ACK SYN URGP=0 MARK=0x64 Oct 10 15:50:42 toy2 kernel: INPUT from TUN: IN=tunde OUT= MAC= SRC=185.61.149.21 DST=10.200.0.54 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=443 DPT=51391 WINDOW=29200 RES=0x00 ACK SYN URGP=0 MARK=0x64
(see MARK=0x64 on outgoing and incoming packets)
Now, iptables rules:
# iptables-save *mangle :PREROUTING ACCEPT [6278:3182515] :INPUT ACCEPT [4169:3043489] :FORWARD ACCEPT [9:468] :OUTPUT ACCEPT [677:98865] :POSTROUTING ACCEPT [703:101438] -A PREROUTING -d 192.168.60.6/32 -i eth0 -p tcp -m tcp --dport 1443 -m state --state NEW -j CONNMARK --set-xmark 0x64/0xffffffff -A PREROUTING -m state --state RELATED,ESTABLISHED -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff -A PREROUTING -m connmark --mark 0x64 -j MARK --set-xmark 0x64/0xffffffff -A PREROUTING -m state --state NEW -m connmark ! --mark 0x0 -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff -A PREROUTING -i tunde -j LOG --log-prefix " INPUT from TUN: " -A PREROUTING -s 192.168.60.159/32 -i eth0 -p tcp -m tcp --dport 1443 -j LOG --log-prefix " INPUT from Cli: " COMMIT *nat :PREROUTING ACCEPT [3487:307264] :POSTROUTING ACCEPT [57:13668] :OUTPUT ACCEPT [57:13668] -A PREROUTING -d 192.168.60.6/32 -i eth0 -p tcp -m tcp --dport 1443 -j DNAT --to-destination 185.61.149.21:443 -A POSTROUTING -o tunde -j LOG --log-prefix " OUTPUT To TUN: " -A POSTROUTING -d 192.168.60.159/32 -o eth0 -p tcp -m tcp --sport 1443 -j LOG --log-prefix " OUTPUT To Cli: " -A POSTROUTING -o tun+ -j MASQUERADE COMMIT *filter :INPUT DROP [0:0] :FORWARD DROP [0:0] :OUTPUT ACCEPT [680:99605] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -p tcp -m state --state NEW -m multiport --dports 22,23 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 1443 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -i tunde -j LOG --log-prefix " Forward From TUN: " -A FORWARD -o tunde -j LOG --log-prefix " Forward To TUN: " -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -p icmp -j ACCEPT -A FORWARD -i eth0 -o tun+ -j ACCEPT -A FORWARD -i tun+ -o eth0 -j ACCEPT -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT
-
NiKiZe over 2 yearsYou also need to
echo 0> all/rp_filter
otherwise that overrides. More info at serverfault.com/questions/816393/…