iptables/ebtables/bridge-utils: PREROUTING/FORWARD to another server via single NIC

Question

We have a number of iptables rules for forwarding connections, which are solid and work well.

For example, port 80 forwards to port 8080 on the same machine (the webserver). When a given webserver is restarting, we forward requests to another IP on port 8080 which displays a Maintenance Page. In most cases, this other IP is on a separate server.

This all worked perfectly until we installed bridge-utils and changed to using a bridge br0 instead of eth0 as the interface.

The reason we have converted to using a bridge interface is to gain access to the MAC SNAT/DNAT capabilities of ebtables. We have no other reason to add a bridge interface on the servers, as they don't actually bridge connections over multiple interfaces.

I know this is a strange reason to add a bridge on the servers, but we are using the MAC SNAT/DNAT capabilities in a new project and ebtables seemed to be the safest, fastest and easiest way to go since we are already so familiar with iptables.

The problem is, since converting to a br0 interface, iptables PREROUTING forwarding to external servers is no longer working.

Internal PREROUTING forwarding works fine (eg: request comes in on port 80, it forwards to port 8080).

The OUTPUT chain also works (eg: we can connect outwards from the box via a local destination IP:8080, and the OUTPUT chain maps it to the Maintenance Server IP on a different server, port 8080, and returns a webpage).

However, any traffic coming into the box seems to die after the PREROUTING rule if the destination IP is external.

Here is an example of our setup:

Old Setup:

iptables -t nat -A PREROUTING -p tcp --dport 9080 -j DNAT --to-destination $MAINTIP:8080
iptables -a FORWARD --in-interface eth0 -j ACCEPT
iptables -t nat -A POSTROUTING --out-interface eth0 -j MASQUERADE
echo 1 > /proc/sys/net/ipv4/ip_forward

New Setup: (old setup in various formats tried as well..., trying to log eth0 and br0 packets)

iptables -t nat -A PREROUTING -p tcp --dport 9080 -j DNAT --to-destination $MAINTIP:8080
iptables -a FORWARD --in-interface br0 -j ACCEPT
iptables -t nat -A POSTROUTING --out-interface br0 -j MASQUERADE
echo 1 > /proc/sys/net/ipv4/ip_forward

Before changing to br0, the client request would go to server A at port 9080, and then be MASQUERADED off to a different server $MAINTIP.

As explained above, this works fine if $MAINTIP is on the same machine, but if it's on another server, the packet is never sent to $MAINTIP under the new br0 setup.

We want the packets to go out the same interface they came in on, MASQUERADED, as they did before we switched to using a single-NIC bridge (br0/bridge-utils).

I've tried adding logging at all stages in iptables. For some reason the iptables TRACE target doesn't work on this setup, so I can't get a TRACE log, but the packet shows up in the PREROUTING table, but seem to be silently dropped after that.

I've gone through this excellent document and have a better understanding of the flow between iptables and ebtables: http://ebtables.sourceforge.net/br_fw_ia/br_fw_ia.html

From my understanding, it seems that the bridge is not forwarding the packets out the same interface they came in, and is dropping them. If we had a second interface added, I imagine it would be forwarding them out on that interface (the other side of the bridge) - which is the way bridges are meant to work ;-)

Is it possible to make this work the way we want it to, and PREROUTE/FORWARD those packets out over the same interface they came in on like we used to?

I'm hoping there are some ebtables rules which can work in conjunction with the iptables PREROUTING/FORWARD/POSTROUTING rules to make iptables forwarding work the way it usually does, and to send the routed packets out br0 (eth0) instead of dropping them.

Comments, flames, any and all advice welcome!

Best Regards, Neale

Solution 1

I guess you did, but just to be sure, did you add eth0 to the bridge?

Although, I am not sure what the problem is, I will give some debugging tips which might assist you or other when debugging bridge/ebtables/iptables issues:

Make sure that "/proc/sys/net/bridge/bridge-nf-call-iptables" is enabled (1)
This cause bridge traffic to go through netfilter iptables code.

Note that this could affect performance.

Check for packet interception by the ebtabels/iptables rules,
Use the commands:

iptables -t nat -L -n -v   

ebtables -t nat -L –Lc  

This might help you to understand if traffic is matched and intercepted or not.
Check that IP NAT traffic appears in the conntrack table:

conntrack –L (if installed)  

Or

cat /proc/net/nf_conntrack  

Check MAC learning of the bridge

brctl showmacs br0  

Use tcpdump on the eth0 and on br0 to check if packets seen at both as expected
Use the –e option to see MAC address as well.

For debugging, try to put the bridge interface in promiscuous mode, maybe the bridge receives packets with different MAC address (in promiscuous mode it will accept different MAC as well)

Maybe set bridge forward delay to 0

brctl setfd br0 0  

And disable stp (spanning tree protocol)

brctl stp br0 off  

What is your routing table looks like?
Try adding specific or default route rule with "dev br0"

I hope it helps a bit…
Good luck

Solution 2

Well only 1.5 years old but could be useful for later lookups. Looking at your link just now, it says specifically there brouting will ignore the packet, if MAC is on same side of bridge (and not another port or the bridge itself (see fig 2.b in your link).

Adding to that, I simply quote from this link: http://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxBridge

"... ebtables DROP vs iptables DROP

In iptables which in most cases is being used to filter network traffic the DROP target means "packet disappear".

In ebtables a "-j redirect --redirect-target DROP" means "packet be gone from the bridge into the upper layers of the kernel such as routing\forwarding" (<-- relevant bit!)

Since the ebtables works in the link layer of the connection in order to intercept the connection we must "redirect" the traffic to the level which iptables will be able to intercept\tproxy.

ANd therein is your answer (bet added for future visitors of course, unless you are still at it ,p)