Memory consumption Fail2Ban CentOS

Question

Fail2Ban is using a huge amount of memory on my system (1.2 GB). There are several articles which describe how to reduce it. Below is an example for Debian.

• append the 1ulimit1 command to /etc/default/fail2ban file.

• Add (to the file) on last line:

  ulimit -s 256
  

Unfortunately there is no such file or directory on CentOS 7. How to apply this on my system?

---

After some tips my Systemd file for Fail2Ban is

[Unit] 
Description=Fail2ban Service 

[Service] 
Type=forking 
ExecStart=/usr/bin/fail2ban-client -x start 
ExecStop=/usr/bin/fail2ban-client stop 
ExecReload=/usr/bin/fail2ban-client reload 
PIDFile=/var/run/fail2ban/fail2ban.pid 
Restart=always 
LimitSTACK=256`

Unfortunately the result is still 1251888 KB.

Solution 1

The solution was to edit /etc/init.d/fail2ban.

This is the start script:

start() {
echo -n $"Starting fail2ban: "
ulimit -s 256
${FAIL2BAN} -x start > /dev/null
RETVAL=$?
if [ $RETVAL = 0 ]; then
touch ${lockfile}
echo_success
else
echo_failure
fi
echo
return $RETVAL
}

Unfortunately it's only saving me 50 mb

Solution 2

/etc/default

The directory /etc/default is never used on any Red Hat based distros. That's a Debian/Ubuntu-ism. For Centos 7 you can take a look at the packages that were installed that relate to fail2ban like so:

$ rpm -aq|grep fail
fail2ban-server-0.9-9.el7.noarch
fail2ban-sendmail-0.9-9.el7.noarch
fail2ban-firewalld-0.9-9.el7.noarch
fail2ban-systemd-0.9-9.el7.noarch
fail2ban-0.9-9.el7.noarch

Contents of fail2ban-server

The fail2ban-server contains the service file for Systemd.

$ rpm -ql fail2ban-server-0.9-9.el7.noarch | grep systemd
/usr/lib/python2.7/site-packages/fail2ban/server/filtersystemd.py
/usr/lib/python2.7/site-packages/fail2ban/server/filtersystemd.pyc
/usr/lib/python2.7/site-packages/fail2ban/server/filtersystemd.pyo
/usr/lib/systemd/system/fail2ban.service

Systemd service file

The contents of the Systemd service file:

$ more /usr/lib/systemd/system/fail2ban.service
[Unit]
Description=Fail2ban Service
After=syslog.target network.target firewalld.service

[Service]
Type=forking
ExecStart=/usr/bin/fail2ban-client -x start
ExecStop=/usr/bin/fail2ban-client stop
ExecReload=/usr/bin/fail2ban-client reload
PIDFile=/var/run/fail2ban/fail2ban.pid
Restart=always

[Install]
WantedBy=multi-user.target

So one could add the extra options to this file, as a quick and dirty way to confirm if they're working.

Long term fixes

To make them permanent, I'd add the options in a more "official" way so that updates to the fail2ban package do not overwrite the modifications to this file. This can be accomplished by adding a customized version of the fail2ban.service file in this directory:

/etc/systemd/system/fail2ban.service

NOTE: A file in this directory, /etc/systemd/system always overrides the default .service file.

However doing it this way have caveats, one being that if a service file is present here when fail2ban were to be updated via yum it would cause the service to be disabled, until you manually reenabled it. So instead you can override fragments of the .service file by adding them to this directory under /etc instead.

Excerpt

To edit a unit file provided by a package, you can create a directory called /etc/systemd/system/unit.d/ for example /etc/systemd/system/httpd.service.d/ and place *.conf files in there to override or add new options. systemd will parse these *.conf files and apply them on top of the original unit. For example, if you simply want to add an additional dependency to a unit, you may create the following file: /etc/systemd/system/unit.d/customdependency.conf

   [Unit]
   Requires=new dependency
   After=new dependency

As another example, in order to replace the ExecStart directive for a unit that is not of type oneshot, create the following file: /etc/systemd/system/unit.d/customexec.conf

   [Service]
   ExecStart=
   ExecStart=new command

so you could create a directory, /etc/systemd/system/fail2ban.service.d and add *.conf files in it with contents like this:

[Service]
ExecStart=
ExecStart=new command

Adding your options there.

Ulimits & Systemd

If you're trying to set a ulimit option for a particular service, then have a look at the man page for systemd.exec.

Excerpt

LimitCPU=, LimitFSIZE=, LimitDATA=, LimitSTACK=, LimitCORE=, LimitRSS=, 
LimitNOFILE=, LimitAS=, LimitNPROC=, LimitMEMLOCK=, LimitLOCKS=,
LimitSIGPENDING=, LimitMSGQUEUE=, LimitNICE=, LimitRTPRIO=, LimitRTTIME=
These settings control various resource limits for executed processes. See 
setrlimit(2) for details. Use the string infinity to configure no limit 
on a specific resource.

So simply adding LimitSTACK=256 to the customized .conf file that I describe above should give you the same effect as setting ulimit -s 256.

Excerpt - setrlimit(2) man page

If you have a look through the setrlimit(2) man page you can see how the ulimit switches line up with the Systemd limits.

   RLIMIT_STACK
        The maximum size of the process stack, in bytes.  Upon reaching 
        this limit, a SIGSEGV signal is generated.  To handle this signal, 
        a process must employ an alternate signal stack (sigaltstack(2)).

        Since Linux 2.6.23, this limit also determines the amount of space 
        used for the process's  command-line  arguments  and  environment
        variables; for details, see execve(2).

References

• systemd - ArchLinux Wiki

Comments

[vinc17]

@user79801 If you used systemctl, then you're using systemd. So, you probably need the method via fail2ban.service. I'm not sure whether you'll see the effect immediately; this is really specific to fail2ban.

[Il Quadrifoglio]

So basically I should make the directory fail2ban.service.d and add a *.conf file in it? But it's not clear which code I should use for the file. Please note that I build Fail2Ban manually from the source code.

[slm]

@user79801 - Yes. Did you look at the referenced link I included? That shows full details, I tried to summarize them here. The last stanza that shows [service] would be the way to override the ExecStart= with your own version.

[Il Quadrifoglio]

Ok thanx! But what should I use for the ExecStart=?

[slm]

@user79801 - The method shown by vinc17 should now make more sense given what I've shown above 8-) ExecStart=/bin/sh -c 'ulimit -s 256; /usr/bin/fail2ban-client -x start'. Also can you please add a reference to the site where you found that ulimit fix?

[Il Quadrifoglio]

Where should I use the code in my init file? I've added 'ulimit -s 256' without any result.

[Il Quadrifoglio]

This is my source for the suggested fix. hacksnsnacks.com/snippets/reduce-fail2ban-memory-usage

[slm]

@user79801 - see updates.

[Il Quadrifoglio]

After adding the code the result is 1251888 KB. So it doesn't make any change. This is my systemd file [Unit] Description=Fail2ban Service [Service] Type=forking ExecStart=/usr/bin/fail2ban-client -x start ExecStop=/usr/bin/fail2ban-client stop ExecReload=/usr/bin/fail2ban-client reload PIDFile=/var/run/fail2ban/fail2ban.pid Restart=always LimitSTACK=256

[slm]

@user79801 - please add that to your A, I can't read it in the comments.

[mattdm]

I might investigate why your fail2ban is taking up so much memory, and possibly adjust the configuration. On my system, it's more like 9M!

[Brian Thomas]

What's the best way to investigate this? my Virtual memory usage for fail2ban-server is at 1054644 on an iRedMail installation with Resource mem now at ~33600 down from ~96000 after making recommended adjustments to a new file /etc/systemd/system/fail2ban.service.d/memlimit.conf by adding the following only LimitSTACK=265 to a new [Service] header.

[Brian Thomas]

So ill note a couple issues i had here, i was able to reduce it a bit, its still high, but i wanted to note for some reason the accent char at the end, found in the OP, e.g. LimitSTACK=128` or i was getting failures. Also, users need to systemctl daemon-reload each time they make these adjustments. Fail2ban is now no longer at the top of the list. Another question would be how do we get the virtual mem down? (digital ocean low mem machine).

[Florian Heigl]

Even if the OP didn't understand it at least this was a pragmatic solution that actually tried to find something working for him!