Fail2Ban: already banned?

Question

I have Fail2Ban running on my Centos Server. (Config below)

In my var/log/messages I noticed something really weird:

Jun 19 12:09:32 localhost fail2ban.actions: INFO   [postfix] 114.43.245.205 already banned

I configured Fail2Ban to add the banned IP to iptables.

My jail.conf:

[postfix]

enabled  = true
filter   = postfix
action   = iptables
port     = smtp,ssmtp
filter   = postfix
logpath  = /var/log/maillog
bantime  = 43200
maxretry = 2

My postfix.conf:

[INCLUDES]

before = common.conf

[Definition]
failregex = reject: RCPT from (.*)\[<HOST>\]: 550 5.1.1
            reject: RCPT from (.*)\[<HOST>\]: 450 4.7.1
            reject: RCPT from (.*)\[<HOST>\]: 554 5.7.1
            reject: RCPT from (.*)\[<HOST>\]: (.*)@yahoo.com.tw
ignoreregex =

My question is how can anybody that has already been blocked in iptables still connect to the server?

Solution 1

The recidive jail recommended in the other answer here did not fix the issue for me. I eventually fixed this, however, so here's my method in case it helps others.

Fail2ban only blocks over TCP by default. At least with my setup, I noticed the "already banned" message was appearing when bots came back to try the blocked port over UDP instead.

To fix this issue, tell Fail2ban to block the port over all protocols instead of just TCP. You will need to make this change in /etc/fail2ban/jail.conf and in the [Init] section of every action you are using at /etc/fail2ban/action.d/.

Change this:

# Default protocol
protocol = tcp

To:

# Default protocol
protocol = all

Next, I disabled ICMP echo requests so blocked IPs had no way of hitting the server:

1. nano /etc/sysctl.conf

2. Add these two lines:

   net.ipv4.icmp_echo_ignore_all = 1  
   net.ipv4.icmp_echo_ignore_broadcasts = 1
   

3. Exit and save the file.
4. Run sysctl -p for the change to take effect.

After that, run fail2ban-client reload and you should not see these "already banned" messages any more unless you are spammed by an IP who gets a couple of access attempts before the block takes effect.

Also, it's important to block all ports for every offender rather than the port they were trying to access by using the action iptables-allports in each of the Jails. Otherwise, they may trigger another Jail and end up as "already banned" in the logs.

Solution 2

If you are using Docker containers to run your application but fail2ban on the host you may run into this issue: https://github.com/fail2ban/fail2ban/issues/2292

Borrowing from the workarounds there, this might be fixable by configuring in you jail.local the line:

[YOUR-JAIL-NAME]
chain = DOCKER-USER
...

Or for Kubernetes

chain = KUBE-FIREWALL

See for more info: https://github.com/fail2ban/fail2ban/issues/2292#issuecomment-593216779

Another option might be in some cases be to use host networking instead of Docker Networking, see: https://docs.docker.com/network/host/

Solution 3

If you look at the output of iptables-save, you will see that the fail2ban chains are setup so they evaluate packets according to the rules defined by the filters, for example:

:fail2ban-ssh - [0:0]
-A INPUT -p tcp -A INPUT -p tcp -A INPUT -p tcp -m multiport --dports 22 -j fail2ban-ssh 
-A fail2ban-ssh -j RETURN

Traffic still reaches the server before the other routing rules are applied and the traffic is rejected. fail2ban still sees this initial traffic, and that is why you see the "already banned" messages. Moreover, there's a special filter for recidivists (/etc/fail2ban/filter.d/recidive.conf):

# Fail2Ban filter for repeat bans
#
# This filter monitors the fail2ban log file, and enables you to add long
# time bans for ip addresses that get banned by fail2ban multiple times.
#
# Reasons to use this: block very persistent attackers for a longer time,
# stop receiving email notifications about the same attacker over and
# over again.
#
# This jail is only useful if you set the 'findtime' and 'bantime' parameters
# in jail.conf to a higher value than the other jails. Also, this jail has its
# drawbacks, namely in that it works only with iptables, or if you use a
# different blocking mechanism for this jail versus others (e.g. hostsdeny
# for most jails, and shorewall for this one).

[INCLUDES]

# Read common prefixes. If any customizations available -- read them from
# common.local
before = common.conf

[Definition]

_daemon = fail2ban\.server\.actions

# The name of the jail that this filter is used for. In jail.conf, name the
# jail using this filter 'recidive', or change this line!
_jailname = recidive

failregex = ^(%(__prefix_line)s| %(_daemon)s%(__pid_re)s?:\s+)WARNING\s+\[(?!%(_jailname)s\])(?:.*)\]\s+Ban\s+<HOST>\s*$

[Init]

journalmatch = _SYSTEMD_UNIT=fail2ban.service PRIORITY=4

# Author: Tom Hendrikx, modifications by Amir Caspi

Solution 4

This will happen if the IP address you're banning isn't actually the IP address of the client that's connecting to the server. E.g. if your server happens to be behind a load balancer or proxy.

It took me quite a while to figure this out recently. The red herring was that the logs were configured to capture the X-Forwarded-For IP address, instead of the real source address, which in my case was the load balancer.

In this case fail2ban isn't much help, since banning the offending IP would end up blocking all the traffic.

View more solutions

Comments

[NineCattoRules]

for me it's not very clear...in my /etc/fail2ban/jail.local some filters have action = iptables-multiport[name=apache-myadmin, port="http,https", protocol=tcp] and some not, should I change all of those? Should I change something in /etc/fail2ban/filter.d?

[Hassan Baig]

So what alternative action did you take?

[Dale Anderson]

@HassanBaig - none. Fail2ban can't do anything if it's operating behind a load balancer or reverse proxy.

[Hassan Baig]

Hmm then what measures would you take against Distributed DoS occuring at the application layer, say an HTTP GET flood?

[Dale Anderson]

@HassanBaig Talk to your hosting provider. Chances are you're not the only person that has encountered the same issue on their systems.

[Patrik Laszlo]

sorry, but protocol = all is not working, giving an error!

[Patrik Laszlo]

"iptables v1.6.2: multiport needs -p tcp', -p udp', -p udplite', -p sctp' or `-p dccp'"

[Patrik Laszlo]

ok, for me the problem was, that the ban was working, but the attacker was using persistent connections so the ban was not in effect right away, as it was still connected and there was no new connection, the only way to do when it was happening restart the mail server

[xekon]

What you basically need is a firewall in front of your load balancer or reverse proxy, and a way to get your ip list from fail2ban, upstream to the firewall. Atleast this is how I plan to solve the problem.

[xekon]

In addition if you are using nginx as your reverse proxy you can use fail2ban directly on the reverse proxy in conjunction with nginx limit_req_zone