java.io.IOException: Too many open files on tomcat server

Question

I have web application which is based on JAVA (GWT Framework). I hosted my web application on Redhat Linux Server 6.0 using Tomcat 8.0.9 in front of apache http server connected using AJP connector and JDK 8. Whenever 20+ users access my website tomcat goes failed and that i checked the logs file i got this execption

19-Oct-2015 12:38:15.350 SEVERE [ajp-nio-8009-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed
 java.io.IOException: Too many open files
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
    at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:688)
    at java.lang.Thread.run(Thread.java:745)

Web Application uses :

Database : couchdb 1.5 (Running on ec-2 machine using 4gb RAM)

Search : elasticsearch 1.0 (Running on ec-2 machine using 4gb RAM)

Tomcat : Tomcat 8.0 in front of apache server connected using ajp connector(Running on ec-2 machine using 8gb RAM)

I searched alot & find out the solution to change the values of fs.file-max = 200000 and soft file and hard file limit to 65535 but still this error occur and tomcat stop responding. Then i need to restart the tomcat ec-2 machine then only application start running properly.

I could not understand why this problem occurs very frequently?

My clients get irritating because of this problem. Please help me out of this problem

Solution

Try increase open file limit as below (in /etc/security/limits.conf):

*         hard    nofile      65535
*         soft    nofile      200000

and restart your services.

You can check limits of a running process:

cat /proc/xxxx/limits

where xxxx is the pid of process

Comments

[Federico Sierra]

See serverfault.com/questions/122679/…

[Kushal Jain]

i already do that my ulimit is 4096 which is less than the file-max then whats the problem ?

[Kushal Jain]

why downvote my question ?

[Kushal Jain]

which process i need to check ? tomcat process or couchdb or elasticsearch ?

[Federico Sierra]

@KushalJain Try with lsof command (eg: lsof -p <pid> | wc -l) that list all open files used, and then check the limit for all process

[Kushal Jain]

pid is of tomcat process id ?