How to circumvent "Too many open files" in debian

Question

If I want to see all relevant log files of my apache2 server at once, I use

tail -f /var/kunden/logs/*log /var/kunden/logs/*log /var/log/apache2/*log |grep -v robots|grep -v favicon

But since those are too many files by now, I would like to encrease that limit.

How can I increase it for one ssh session? And How could I increase it globally systemwide?

I can see the open files limit is 1024 on my machine:

ulimit -n
1024

Solution

It is Important to know that there are two kinds of limits:

• A hard limit is configurable by root only. This is the highest possible value (limit) for the soft limit.
• A soft limit can be set by an ordinary user. This is the actual limit in effect.

Solution for a single session

In the shell set the soft limit:

ulimit -Sn 2048

This example will raise the actual limit to 2048 but the command will succeed only if the hard limit (check: ulimit -Hn) is the same or higher. If you need higher values, raise the hard limit using one of the methods below. The limits are set per process and they are inherited by newly spawned processes, so anything you run after this command in the same shell will have the new limits.

Changing hard limit in a single session

This is not easy because only root can change a hard limit and after switching to root you have to switch back to the original user. Here is the solution with sudo:

sudo sh -c "ulimit -Hn 9000 ; exec su \"$USER\""

System-wide solution

In Debian and many other systems using pam_limits you can set the system-wide limits in /etc/security/limits.conf and in files in /etc/security/limits.d. The conf file contains description. Example lines:

@webadmins       hard     nofile     16384
@webadmins       soft     nofile      8192

This will set the hard limit and default soft limit for users in group webadmins after login.

Other limits

The hard limit value is limited by global limit of open file descriptors value in /proc/sys/fs/file-max which is pretty high by default in modern Linux distributions. This value is limited by NR_OPEN value used during kernel compilation.

Is there not a better solution?

Maybe you could check if all the *log files you feed to tail -f are really active files which need to be monitored. It is possible that some of them are already closed for logging and you can just open a smaller number of files.

Comments

[BillThor]

Exceeding the file limit is often a sign of a problem. As already noted, you may have a problem with log rotation. (Rotated log may drop off your command.) Outside /var/log/apache2 there shouldn't be many (any) logs related to your Apache server. Based on your grep pattern, you likely want to restrict your access to access logs.

[rubo77]

thanks for the hint, but my logrotaion works fine. I have so many logs, cause all 1025 domains on my server have their own log file ;)

[BillThor]

Your patterns look like they will also pick up error logs as well as access logs. This should be a minor issue if you have a single error log. If you leave this running in a shell, you may want to use -F instead of -f so logs get reopened when rotated.

[rubo77]

ulimit -Sn 4096 -bash: ulimit: open files: Kann die Grenze nicht ändern: Das Argument ist ungültig -- means the argument is not valid

[pabouk - Ukraine stay strong]

@rubo77: I added information about the hard limit which is most probably the cause of the problem.

[pabouk - Ukraine stay strong]

@rubo77: It is also possible to set the hard limit in a shell session but is it tricky as you have to su to root and su back to the user: su -c "ulimit -Hn 6000 ; su $USER"

[rubo77]

OK, so for a temporary change of the limit I will use ulimit -Hn 6000; ulimit -Sn 6000 as root then

[rubo77]

since man ulimit doesent provide any help, this man page might be of interest: ss64.com/bash/ulimit.html

[pabouk - Ukraine stay strong]

@rubo77: Regarding the temporary switching: Since changing hard limits require root privileges I would recommend you raising the hard limit system-wide for users which need it in /etc/security/limits... otherwise you have to complicatedly switch to root and back which increases risk of doing mistakes under the root account. Regarding the man page: ulimit is a shell built-in command (like cd) so it is described in the man page of the shell. Alternatively in bash you can type help ulimit. man ulimit will probably show you C library call ulimit() from the man section 3.

[amenthes]

how do the values in /etc take effect? Do i need to restart something?

[pabouk - Ukraine stay strong]

@amenthes I guess you mean the configuration files in /etc/limits.*. These limits are applied whenever the PAM module pam_limits.so is called. Normally it is at start of a session. For example in Ubuntu 14.04 it is in these tools / commands: cron, login (text console login), lightdm (the GUI login), su. You can set where is pam_limits used in /etc/pam.*. See for example: faqs.org/docs/securing/chap5sec44.html