How to block IPs that cause excessive 404 errors with Fail2ban?

Question

I have installed Fail2Ban v0.10.2 on Ubuntu 18.04 with Apache 2.4.29 and enabled the standard ssh and apache jails for basic protection with email notification warnings, when an IP is blocked.

Having a look at the documentation, I was not able to find a relevant filter that would help with the following situation:

I would like to ban IPs that hit the server and produce large numbers of 404 errors due to fake URL requests, which can be a typical spam bot behavior. So ideally, an IP is blocked that produces more than three 404 errors in a row with some exceptions for official search engine crawlers.

Is there a default regex for this situation?

I would appreciate your assistance on how to implement this.

Solution

I recommend you start by implementing the built-in apache-noscript filter for fail2ban. To do so, add the following lines to/etc/jail.local`

[apache-noscript] 
     enabled = true 
     port = http,https 
     filter = apache-noscript 
     logpath = /var/log/apache2/*error.log 
     maxretry = 3 
     bantime = 600 

tweak the bantime setting to your liking and consider implementing the recidiv filter/jail for repeat offenders.

Note: there is a possible bug with the filter regex

Comments

[danblack]

PS, don't take me for a fail2ban hater, I was a core maintainer for a number of years before the number of requests like this where too much and distracting from writing real security benefits so I quit.

[ChrisBint]

I run a number of websites and do not 'frequently' get 404s. The majority of 404s are people attempting to find exploits on the websites and believe it is perfectly reasonable to block multiple attempts that result in a 404

[Smooth Criminal]

That's actually very bad advice. If someone is trying to break your website, there should be traces and warnings about it, not silent ignores, because "real visitors". If your server has trouble logging errors, then you should buy a better server, not ignore this problem.

[danblack]

404s are harmless. Automaticly classing them as potential threat is a stretch considering most are just automated and even the concept they could do damage is remote. Deploying f2b for low end noise distracts from serious effort in hardening and vulnerability in the same way AV acts as a deficient placebo for managing malware risk. This is a really hard concept to grasp when you're invested in classifying 404 messages as threats and gain euphoria from seeing bans and less log messages. This relief is a distraction from doing stronger access controls and code review that will actually protect.

[Barry]

Saying 404's are harmless is ridiculous. If someone is port scanning your server and stumbles across an open port, is port scanning also harmless?

[danblack]

@Barry stick to the topic and don't try making a generality to make it true. If you have a 404 then you are running a server so a http port is exposed. A 404 processing on the web server takes very little time. fail2ban processing web log to generate iptables rules take more cpu time. You've taking way cputime from your real visitors to save you from making monsters out of noisy bots looking for non-existant exploits. And yes, port scanning is harmless because you shouldn't run services that don't need to be exposed. Use fail2ban to block brute force attempts could result in damage only.

[Barry]

@danblack Sorry but you could not be more wrong. Fail2ban has a rolling ban/unban system, so the cpu overhead you speak of only exists under ddos scenario. And in that scenario, you shouldn't be relying on ip rules, let alone analyzing http error codes via log files. On topic would be per the title, blocking erroneous error codes from someone scanning your server for exploits. Saying that's not effective is ridiculous at base level, let alone a deep level. That's fail2ban's precise job. Analyze logs, block ips. I am entirely on topic, you are not. Which is why i gave the port scan example

[Barry]

It's truly fascinating to see someone suggest not using protection in a security related topic. If we were talking about ddos, which we were not and are not, then I would agree there are far better alternative solutions. Like cloudflare @ dns level, or modsecurity w/ owasp. But to outright tell someone not to use fail2ban.... And to take that even a step further and say port scanning is harmless is beyond all recourse. If you can stop someone port scanning, they lose intel. If you can stop scripts testing for exploits, hacker loses. This is security 101, like bare minimum understanding...

[danblack]

risk assessment. It's what security is about. And read the answer's properly