Increasing open files limit for all processes: Do I need to set Soft/Hard limits?

Question

I'm trying to fix the following error that I get after a day or two of running my game server.

2017/12/13 12:08:35 http: Accept error: accept tcp [::]:8081: accept4: too many open files; retrying in 1s

I added "fs.file-max = 2000000" to /etc/sysctl.conf and executed

sysctl -w fs.file-max=2000000
sysctl -p

My global limits are now updated (do I need to reboot?) but the soft and hard limits are still 1024 and 4096 respectively.

Also when using the following command to check for "open files" for root user:

su - root -c 'ulimit -aHS' -s '/bin/bash'

I'm getting 1024 as well.

What does the soft and hard limit do and do I need to change them in order for the global limits to have any effect? And how about the user (root) limit?

Thanks!

Solution 1

The limit you set with sysctl is a system setting that applies to the whole system. It is not a limit that applies to individual processes.

Each process can have no more than N files open where N is the process's NOFILE soft limit, and it can change its own soft limit to no more than the hard limit. Only processes running as root can raise their hard limit. Processes inherit their parent's limits.

The way to change the limit for a single service (which is what you should do) depends on your init system.

• For SysVinit (CentOS ≤6): edit the init script for the service (normally located in /etc/rc.d/init.d) to call ulimit before running the daemon, then restart the service.

• For Systemd (CentOS ≥7): edit the service's unit file /etc/systemd/system/my_game_server.service and add a directive

  LimitNOFILE=16384
  

  Then run systemctl daemon-reload to reload the configuration, then restart your service.

Solution 2

You can change the limits in /etc/security/limits.conf. Log out and in to take effect.

Hardlimit: The boundary for a certain user - can not be increaed by that user during runtime, only decreased: ulimit -Hu 2000. List of Hardlimits: ulimit -Ha

Softlimit: A "soft" boundary within the hardlimit, can be changed by the user during runtime: ulimit -Su 10000.List of Softlimits: ulimit -Sa

Change ulimits for a running process: prlimit -p $$ --nproc=1200:. This will change the number of processes (softlimit) to 1200 for the current shell $$.

Comments

[ilkkachu]

the error you get "after a day or two"... Does the problem appear when there's a large number of simultaneous users, or at arbitrary times after the server has been running sufficiently long? I.e. do we know it's not just opening files all the time, but neglecting to close them?

[SJ19]

Yes it mostly happens when it's been populated for a while.

[SJ19]

@ilkkachu It mostly happens after 2 busy evenings, where people have been leaving and joining frequently.

[Atul Vekariya]

I am afraid so, as far as I know sysctl will change the parameter in kernel, but if you have limit in limits.conf they will limit you

[Ulrich Schwarz]

CentOS7 may or may not also need corresponding entries in the systemd service file. (LimitFiles if I remember correctly.)

[SJ19]

Thanks! I'm running my program in a screen on root user, CentOS7. Does that mean I don't have to change anything now?

[SJ19]

I should mention that I'm running my program in a screen on root user, does that make a difference?

[Atul Vekariya]

@SJ19, I do not think so. Change the limits for root user, logout, login again and run it

[Gilles 'SO- stop being evil']

@SJ19 You do need to change the service's unit file, unless you've changed the limits globally as in Romeo's answer.

[Gilles 'SO- stop being evil']

@ilkkachu It doesn't apply to daemons started by init at boot time. But under SysVinit it would indirectly apply to a daemon started by someone typing start-stop-daemon … or /etc/rc.d/… in a shell.

[SJ19]

@RomeoNinov Does the soft limit even matter when you're the root user? I might've misunderstood this, but I believe someone else said that the soft limit gets increased automatically when it's reached? And is there anything wrong with just setting both the soft and hard limit to like 2 million? Like what's the point of the soft limit.

[SJ19]

@ilkkachu What exactly does "pam" and "daemon" mean? And why would I need to change those rather than the user limits? It just keeps getting more confusing to me... I appreciate the help though!

[SJ19]

@Gilles I manually run the program after a restart (on root), so all of this doesn't really apply to me right?

[SJ19]

@RomeoNinov Rebooting is also okay right? Instead of logging out and in.

[Atul Vekariya]

@SJ19, correct. But this is very "heavy" way to do this :)