fail2ban doesn't add IPs to ipset (firewalld)

Question

For some reason I cannot figure out, Fail2Ban refuses to add IP addresses to ipset/firewalld.

I'm being bruteforced by a chinese IP address, Fail2Ban does seem to actually attempt to ban it (at least it shows up in the fail2ban-client status sshd command), but it doesn't actually stop bruteforcing the server, and it doesn't show up in ipset/iptables/firewalld.

fail2ban-client status sshd:

[root@GITserver alex]# fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 2
|  |- Total failed: 93613
|  `- File list:    /var/log/secure
`- Actions
   |- Currently banned: 2
   |- Total banned: 1003
   `- Banned IP list:   61.177.172.17 95.103.88.106

ipset/firewalld:

[root@GITserver alex]# ipset list
[root@GITserver alex]# firewall-cmd --direct --get-all-rules
[root@GITserver alex]#

Any ideas? more logs can be added on request

Edit, still no solution, still being bruteforced and DDoS'd, here's the complete /var/log/fail2ban.log if it may help https://paste.fedoraproject.org/paste/Y5aPF8~WY~fpuhaZBDo7Ml5M1UNdIGYhyRLivL9gydE=/raw (warning: 11mb)

ANY SUGGESTION WOULD BE VERY MUCH APPRECIATED. Still under attack.

Edit2: Looking at the logs it still seemed to use iptables, even though /etc/fail2ban/jail.d/00-firewalld.conf defined banaction=firewallcmd-ipset. I edited the main jail.conf to fix this, but it changed nothing. You can see the output after the reload in the log above.

Solution 1

I have spent an absolute age trying to get this working today on a VPS (CentOS Linux 7.6.1810) with fail2ban installed through VirtualMin.

In my situation Fail2Ban was working correctly from its logs

tail -f /var/log/fail2ban.log

And correctly recording decisions to ban, however these were not being processed into firewalll rules. Having read through all of the comments in the answer posted by Moshe and trying three of the solutions there without success, I found that the solution by Geraden07 on a GitHub discussion linked from that forum post worked on Centos.

1. Create a new action rule (/etc/fail2ban/action.d/custom-firewalld.conf)

[INCLUDES]
before  =

[Definition]
actionstart =
actionstop =
actioncheck =

actionflush = sed -i '/<source address=/d' /etc/firewalld/zones/drop.xml
actionban = firewall-cmd --change-source=<ip> --zone=drop && firewall-cmd --change-source=<ip> --zone=drop --permanent
actionunban = firewall-cmd --remove-source=<ip> --zone=drop && firewall-cmd --remove-source=<ip> --zone=drop --permanent || echo 0

[Init]

2. Create/update jail config to use this new rule as the default (/etc/fail2ban/jail.local)

[DEFAULT]
banaction = custom-firewalld

3. Update any jails that override the default to use the new rule if needed

I have tested this for banning and unbanning and it is the only thing that has worked for me.

Solution 2

Check to make certain that firewallcmd-ipset actually exists on your system; it no longer exists on CentOS 7.3 and above.

This firewalld-based work-around recreates firewallcmd-ipset. I'm about to implement it on my system.

Solution 3

I had the same problem and I think there are several bugs in fail2ban that need to be fixed, for instance, the 00-firewalld.conf seems to be ignored completely.

To solve it set the desired action in your sshd filter section. My sshd jail looks like this (/etc/fail2ban/jail.d/sshd.local)

[sshd]
enabled = true
port = ssh
# set the action explicitly or the default iptables-allports will be used
# ==============================
action = firewallcmd-ipset
# ==============================
logpath = %(sshd_log)s
findtime = 600
maxretry = 3
bantime = 86400

Now the problem is that the bantime in your jail is ignored and the one defined in /etc/fail2ban/action.d/firewalld-ipset.conf takes precedence. So edit that action configuration file to include your new bantime

...
bantime=86400
...

And the last problem is that bantime=-1 for permanent bans does not seem to work. The developers fixed this in fail2ban +0.9 but it does not work for me although I am using fail2ban-0.9.7-1.el7

For your commodity what I exactly did was to copy firewallcmd-ipset.conf to firewalldcmd-ipset-custom.conf and edit the bantime there. Finally in your sshd jail definition set the action to action=firewallcmd-ipset-custom

Restart your fail2ban service after the changes.

Some output:

# firewall-cmd --direct --get-all-rules
ipv4 filter INPUT_direct 0 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-default src -j REJECT --reject-with icmp-port-unreachable

# iptables -L -n | grep fail2ban
REJECT tcp -- 0.0.0.0/0   0.0.0.0/0    multiport dports 22 match-set fail2ban-default src reject-with icmp-port-unreachable

# ipset list | head
Name: fail2ban-default
Type: hash:ip
Revision: 1
Header: family inet hashsize 1024 maxelem 65536 timeout 86400
Size in memory: 53520
References: 1
Members:
x.x.x.x timeout 78498
y.y.y.y timeout 78506
z.z.z.z timeout 78454

What I still don't get is why the ipset is called fail2ban-default instead of fail2ban-sshd, probably I have something misconfigured, but the steps I describe are working and now IPs are being blocked from the set and I don't have +1000 rules in iptables anymore.

Comments

[Alexander-WorkUltimum]

PS the word "fail2ban" does not show up in /var/log/audit/audit.log so I don't think it's selinux either ..

[Jakuje]

What is the output of iptables-save? I don't think fail2ban is using firewalld native rules.

[Alexander-WorkUltimum]

paste.fedoraproject.org/paste/… @Jakuje I uploaded it with fpaste. It should be using firewalld as I'm using the recommended centos config which uses banaction=firewallcmd-ipset

[Jakuje]

Is even firewalld running?

[Alexander-WorkUltimum]

@Jakuje Yep.. Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2017-05-26 08:10:23 CEST; 3 days ago

[Stuart Cardall]

don't run sshd on port 22 & stealth ssh completely with fwknop it-offshore.co.uk/security/… - see my site also for 4096bit RSA keys with fwknop

[Alexander-WorkUltimum]

@StuartCardall I have to run SSH on port 22 in this case as it's used for hosting git repositories, developers got upset when they had to manually alter repo links to use the right ports, using this port is far more convenient. That fwknop package looks interesting though, I'll keep it in mind for the future, thanks!

[dan]

Which workaround? Your link has a bunch of imcomplete answers and it could be dead by the time the next reader sees your answer.

[dan]

This is the only solution that actually works for me on CentOS 7 with the latest packaged fail2ban version (0.10.4 as time of writing). It does add (and effectively block) IPs, and I can unban IPs too using the fail2ban-client commandline. I still haven't tested performances, people have complained on the GitHub discussion that fail2ban tries to add duplicates (which are ignored by firewallcmd) and also that having an IP in 2 different jails could be a problem (the first unban will unban it completely from every jail), so this still needs a little tweaking probably.