Increasing nproc for processes launched by systemd on CentOS 7

Question

I have successfully increased the nofile and nproc value for the local users, but I couldn't find a proper solution for the processes launched by systemd. Adding max_open_files to the MariaDB configuration doesn't help. su - mysql to change the limit manually doesn't work either (This account is currently not available).

/etc/security/limits.conf

* soft nofile 102400
* hard nofile 102400
* soft nproc 10240
* hard nproc 10240

/etc/security/limits.d/20-nproc.conf (no other files present in the directory)

* soft nofile 102400
* hard nofile 102400
* soft nproc 10240
* hard nproc 10240

/etc/sysctl.conf

fs.file-max = 2097152

/etc/pam.d/system-auth

#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 1000 quiet_success
auth        required      pam_deny.so

account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 1000 quiet
account     required      pam_permit.so

password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
-session     optional      pam_systemd.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so

/etc/pam.d/systemd-user

#%PAM-1.0

# Used by systemd when launching systemd user instances.

account include system-auth
session include system-auth
auth required pam_deny.so
password required pam_deny.so

/var/log/mariadb/mariadb.log

[Warning] Changed limits: max_open_files: 1024  max_connections: 32  table_cache: 491

/proc/mysql_pid/limits

Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             30216                30216                processes
Max open files            1024                 4096                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       30216                30216                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

It is interesting that different processes (users) have different Max open files number:

mysql - 1024 4096
apache - 1024 4096
postfix - 4096 4096

Solution 1

systemd completely ignores /etc/security/limits*. If you are using an RPM that auto-squashes its systemd service file on update, you'll want to file a PR to ask them to mark those files as 'noreplace'

You need to update the .service file /usr/lib/systemd/system/<servicename>.service

[Unit]
Description=Some Daemon
After=syslog.target network.target

[Service]
Type=notify
LimitNOFILE=49152
ExecStart=/usr/sbin/somedaemon

[Install]
WantedBy=multi-user.target

sickill pointed out that you can also override the package-installed values (found in the above file) by adding them to /etc/systemd/system/<servicename>.d/override.conf

[Service]
LimitNOFILE=49152

This provides the added bonus of system-specific settings that aren't in danger of being overwritten on package update.

Then issue the command: systemctl daemon-reload

Solution 2

Configuration files in /usr/lib/systemd/system/ should not be edited by hand and it is perfectly normal (if not expected) that an rpm will update files that it manages in this directory on update.

As @sickill and @Cherif KAOUA pointed out in comments [https://stackoverflow.com/questions/27849331/how-to-set-nginx-max-open-files/36423859#36423859] you should add custom configuration including limits into /etc/systemd/system/<servicename>.service.d/override.conf. eg:

[Service]
LimitNOFILE=65536

Then reload the systemctl daemon config:

systemctl daemon-reload

RHEL has a great section on systemd in their System Administrator Guide which among other things, lists where systemd unit files should be located and how to override defaults.

Solution 3

Edit the /etc/sysctl.conf file and Add the following line to increase the maximum number of open files permitted to 64000.

This is a good default starting value but please research and tweak this value as needed for your environment.

fs.file-max=64000

Comments

[amq]

I haven't mentioned it, but it is already set. cat /proc/sys/fs/file-max: 2097152

[Pablo]

Then you will need to add a line specifying the LimitNOFILE value in your mariadb.service file. You can do this by overriding it in /etc/systemd/system/mariadb.service or directly at /lib/systemd/system/mariadb.service. If your service has a different name other than mariadb please change the filename. Google for LimitNOFILE and you'll find some documentation on this.

[Kalle Richter]

Is Type=notify necessary or does the solution work for every Type?

[gladiatr72]

@KarlRichter From what I've gleaned from the systemd docs, the Limit* entries are Just Another Argument that belongs under [Service]. That being said, I have not noodled with other Type values to be able to say that it is one way or the other with certainty.

[sickill]

Instead of modifying the unit file in place (which may be overriden on package upgrade) you can override only specific settings as seen in this answer: stackoverflow.com/a/36423859/264409

[gladiatr72]

@KarlRichter I haven't read anything in the systemd docs indicating that the service type is affected by limit definitions.

[Cherif KAOUA]

/etc/systemd/system/<servicename>.service.d/override.conf , systemctl daemon-reload and 'systemctl cat <servicename>' to check if it's correctly loaded.

[Frederick Nord]

How to set the soft limit? It seems that this directive controls the hard limit.

[Luka]

In my situation I have upgraded to MariaDB via cPanel and cPanel added /etc/systemd/system/mariadb.service.d/migrated-from-my.cnf-s‌​ettings.conf With LimitNOFILE and LimitMEMLOCK And even i created new files it didn't worked until I changed/removed already set custom config in /etc/systemd/system/mariadb.service.d/migrated-from-my.cnf-s‌​ettings.conf

[Doug Donohoe]

You can also use sudo systemctl edit <servicename>.service to create/edit the override.conf file. It also executes the daemon-reload when you save.

[danblack]

or systemctl edit {service} which handles the file location and daemon-reload