How to cleanly and silently reload iptables rules?

Question

I have very complicated and long iptables script. It is not possible to operate on the existing iptables configuration, by doing manual inserts/replaces or deletions. I have a script which simply flushes all rules and custom chains, then reloads everything from scratch. This approach works well, to some extent.

I have a lot of sensitive traffic, like E1 lines encapsulated into IP packets and many others. I can't afford to just drop all rules and reinsert them, because this is simply too slow. Lots of stuff breaks if there is no rule for more then 50ms. Aside from that, some high throughput traffic runs in into partially restored firewall, which ends up in very bad conntrack entries, which require manual intervention to restore functionality.

The solution would be to append new rules at the end of current, then to remove the old ones, which can theoretically result in continuous ruleset in place. The problem is, that a script with custom chains, ipset's and such is getting very complicated and error prone.

Question is - do you know any existing solution (extra layer on top of iptables), which handles the problems I mentioned here?

Thanks in Advance.

Solution 1

Have you tried loading your new rules with the iptables-restore command? This is in theory an atomic operation, which may take care of most of your issues. This does require that you write your rules in the format used by iptables-save.

Solution 2

This should be pretty easy if you use chains.

Create a chain or two and add all your rules to that. All you should have to do is flush, delete, and recreate the chain(s) when you need to reapply your rules.

So during an update, you insert a rule at the top that permits established connections, (maybe you want this to be a rule by itself always enabled and never touched), flush the chain, then adds the new rules to the chain. This assumes you are using stateful rules wherever possible.

Solution 3

The only way for this to be feasible is to either rework your existing script to use the iptables-restore format or, modify your existing script to dump commands to stdout instead of executing them and then produce a second script to convert it to iptables-restore format.

iptables-restore is guaranteed to be atomic and is thus the only reliable way to do seamless rule replacement.

my suggestion to you in terms of getting the iptables-restore format would be to run your script against a VM or in a separate network namespace on the live machine and then use iptables-save to grab it.

Comments

[Scott Pack]

How frequently are you having to change rules, and which distribution are you running on top of?

[Phil Hollenback]

Is linux really the right tool for this job? It sounds like you are getting into territory better covered by devices from Cisco and other networking vendors.

[mdrozdziel]

@packs The distribution doesn't really matter here. I am not using anything specific to any distribution. Set of rules in contained in a lsb-like custom start script.

[mdrozdziel]

@Phil Mentioned Boxes are running BGP/IGP, IPSEC, LVS, policy routing and more layer 3 stuffh. Filtering is very complicated, mostly based on markings. Because several people maintain the environment, we have unified diffs generated between commits. Thus there is 4+ year old history of changes available. Equivalent functionality from Cisco/Juniper would cost us hundred of thousands of bucks in hardware and migration costs. I am sure parts of the stuff wouldn't be even possible to recreate with networking vendor hardware.

[mdrozdziel]

@packs It does not matter. All start-up scripts do is wrapping iptables-save/restore in some custom manner. As I pointed out before, I am using custom shell script, so - again - it doesn't matter in this case. In case you are dying to posses this knowledge - we are using Debian.

[Spence]

+1 - I like this. With a box doing as much critical filtering you really should give a lot of thought to the architecture of your iptables rule set, designing for performance, correct function (obviously), and maintainability (often overlooked).

[mdrozdziel]

Well, it even makes sense to create a new chain and replace the main hook point (iptables -R). This however won't work with ipset, since you can't remove ipset until it is referenced by iptables rule. In the end it pretty much comes down to the original idea - Append the rules at the end, then remove the old rules one-by-one. This can work in theory, but requires a major overhaul in our approach. Perhaps we should gave up on stuph like many custom chains or ipsets, to reduce complexity. Before that decision I would like to know if someone can suggest something less invasive.

[mdrozdziel]

Well, I never thought that iptables-save and ipset -S are any different than just simple flush and rebuild of ruleset. I will look into this, thanks for the tip.

[mdrozdziel]

I don't find this case any extreme. Parsing "-L" output for entries isn't a trivial task. Aside from that, there there is no need to overcomplicate things like this. First of all there is "-R" for replace available. Second, it would be trivial just to add all rules at the end, then remove old rules from start. Unfortunetely this wont work due to problems referenced in other answers / comments.