Socket accept - "Too many open files"

Question

I am working on a school project where I had to write a multi-threaded server, and now I am comparing it to apache by running some tests against it. I am using autobench to help with that, but after I run a few tests, or if I give it too high of a rate (around 600+) to make the connections, I get a "Too many open files" error.

After I am done with dealing with request, I always do a close() on the socket. I have tried to use the shutdown() function as well, but nothing seems to help. Any way around this?

Solution 1

There are multiple places where Linux can have limits on the number of file descriptors you are allowed to open.

You can check the following:

cat /proc/sys/fs/file-max

That will give you the system wide limits of file descriptors.

On the shell level, this will tell you your personal limit:

ulimit -n

This can be changed in /etc/security/limits.conf - it's the nofile param.

However, if you're closing your sockets correctly, you shouldn't receive this unless you're opening a lot of simulataneous connections. It sounds like something is preventing your sockets from being closed appropriately. I would verify that they are being handled properly.

Solution 2

I had similar problem. Quick solution is :

ulimit -n 4096

explanation is as follows - each server connection is a file descriptor. In CentOS, Redhat and Fedora, probably others, file user limit is 1024 - no idea why. It can be easily seen when you type: ulimit -n

Note this has no much relation to system max files (/proc/sys/fs/file-max).

In my case it was problem with Redis, so I did:

ulimit -n 4096
redis-server -c xxxx

in your case instead of redis, you need to start your server.

Solution 3

TCP has a feature called "TIME_WAIT" that ensures connections are closed cleanly. It requires one end of the connection to stay listening for a while after the socket has been closed.

In a high-performance server, it's important that it's the clients who go into TIME_WAIT, not the server. Clients can afford to have a port open, whereas a busy server can rapidly run out of ports or have too many open FDs.

To achieve this, the server should never close the connection first -- it should always wait for the client to close it.

Solution 4

Use lsof -u `whoami` | wc -l to find how many open files the user has

Solution 5

This means that the maximum number of simultaneously open files.

Solved:

At the end of the file /etc/security/limits.conf you need to add the following lines:

* soft nofile 16384
* hard nofile 16384

In the current console from root (sudo does not work) to do:

ulimit -n 16384

Although this is optional, if it is possible to restart the server.

In /etc/nginx/nginx.conf file to register the new value worker_connections equal to 16384 divide by value worker_processes.

If not did ulimit -n 16384, need to reboot, then the problem will recede.

PS:

If after the repair is visible in the logs error accept() failed (24: Too many open files):

In the nginx configuration, propevia (for example):

worker_processes 2;
worker_rlimit_nofile 16384;
events {
  worker_connections 8192;
}

View more solutions

Comments

[linjunhalida]

username hard nofile 20000

[Mat]

I'm sorry that is not plausible at all. If a very slight change in your code made the bug "go away", you most probably have a serious bug in your code that the change hid. Use valgrind or other such tools to track it down. A compiler optimizing away a close call would be catastrophic.

[Rafael Baptista]

And the answer to a memory leak is... buy more memory? No fix the file leak.

[Nick]

Seems you do not understand the problem (or you place the comment under wrong answer?. It has to do with file descriptor limit, and nothing to do with memory or memory leak.

[fluffy]

The file limit is 1024 because otherwise you run into a fundamental problem with select().

[Antwan van Houdt]

@RafaelBaptista High amount of concurrent connections is actually needed in some cases like for instance, a high performance chat server. This does not have to be about leaking FDs.

[Mikko Rantalainen]

@RafaelBaptista: if you have a server that can handle more than 512 paraller connections you need A LOT MORE open files. Modern servers can handle multiple million parallel connections so having a limit as low as 1024 really does not make any sense. It might be okay for default limit for casual users but not for server software handling parallel client connections.

[Mikko Rantalainen]

I agree. Checking the return value from any system call is important, though, because you could get EGAIN from many cases and if you ignore that, all bets are off.

[Nick]

However, most of the programs does not need more that 6-7 file descriptors. And if you have more than 1024, select() might have problem - suppose there is a web server with select(). then suppose it opens just 10 file descriptors, but one of them is with number after 1024. The server can not select() this descriptor in any way. Value of 1024 is hard coded and can not be changed easily. This is why they introduced poll() . Lated, epoll() and kqueue() were introduced for performance reasons, because poll() copy memory all the time.

[user786]

can u please tell can this limit be changed automatically without reboot. I am using Ubantu linux kernel 5.11.0-37-generic?