Tomcat memory management

Question

I'm running Tomcat7, the server is quite powerful, 8 GB RAM 8-core.

My problem is that the RES memory is geting higher and higher, until the server just doesn't respond anymore, not even calling OnOutOfMemoryError.

Tomcat configuration :

-Xms1024M
-Xmx2048M
-XX:PermSize=256m
-XX:MaxPermSize=512m
-XX:+UseConcMarkSweepGC
-XX:OnOutOfMemoryError='/var/tomcat/conf/restart_tomcat.sh'

Memory informations :

Memory:     Non heap memory = 106 Mb (Perm Gen, Code Cache),
Loaded classes = 14,055,
Garbage collection time = 47,608 ms,
Process cpu time = 4,296,860 ms,
Committed virtual memory = 6,910 Mb,
Free physical memory = 4,906 Mb,
Total physical memory = 8,192 Mb,
Free swap space = 26,079 Mb,
Total swap space = 26,079 Mb
Perm Gen memory:    88 Mb / 512 Mb    ++++++++++++
Free disk space:    89,341 Mb 

The memory used by Tomcat doesn't look that high compared to the top command.

I also had java.net.SocketException: No buffer space available when trying to connect to SMTP server or when trying to connect to facebook servers.

I use Hibernate, with c3p0 connection pool with this configuration :

        <property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
        <property name="hibernate.connection.url">jdbc:mysql://urldb/schema?autoReconnect=true</property>
        <property name="hibernate.connection.username">username</property>
        <property name="hibernate.dialect">org.hibernate.dialect.MySQL5InnoDBDialect</property>
        <property name="hibernate.connection.password"></property>
        <property name="connection.characterEncoding">UTF-8</property>

        <property name="hibernate.c3p0.acquire_increment">1</property>
        <property name="hibernate.c3p0.idle_test_period">300</property>
        <property name="hibernate.c3p0.timeout">5000</property>
        <property name="hibernate.c3p0.max_size">50</property>
        <property name="hibernate.c3p0.min_size">1</property>
        <property name="hibernate.c3p0.max_statement">0</property>
        <property name="hibernate.c3p0.preferredTestQuery">select 1;</property>
        <property name="hibernate.connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>

I couldn't find anything... does someone have an hint of where I should be looking for ?

Thanks!

[UPDATE 1]HEAP DUMP :

HEAP HISTOGRAM :

class [C                                    269780  34210054
class [B                                    5600    33836661
class java.util.HashMap$Entry               221872  6212416
class [Ljava.util.HashMap$Entry;            23797   6032056
class java.lang.String                      271170  5423400
class org.hibernate.hql.ast.tree.Node       103588  4972224
class net.bull.javamelody.CounterRequest    28809   2996136
class org.hibernate.hql.ast.tree.IdentNode  23461   2205334
class java.lang.Class                       14677   2113488
class org.hibernate.hql.ast.tree.DotNode    13045   1852390
class [Ljava.lang.String;                   48506   1335600
class [Ljava.lang.Object;                   12997   1317016 


Instance Counts for All Classes (excluding platform) :

103588 instances of class org.hibernate.hql.ast.tree.Node
33366 instances of class antlr.ANTLRHashString
28809 instances of class net.bull.javamelody.CounterRequest
24436 instances of class org.apache.tomcat.util.buf.ByteChunk
23461 instances of class org.hibernate.hql.ast.tree.IdentNode
22781 instances of class org.apache.tomcat.util.buf.CharChunk
22331 instances of class org.apache.tomcat.util.buf.MessageBytes
13045 instances of class org.hibernate.hql.ast.tree.DotNode
10024 instances of class net.bull.javamelody.JRobin
9084 instances of class org.apache.catalina.loader.ResourceEntry
7931 instances of class org.hibernate.hql.ast.tree.SqlNode 

[UPDATE 2] server.xml :

<Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443"
               URIEncoding="UTF-8"
               maxThreads="150"
               minSpareThreads="25"
               maxSpareThreads="75"
               enableLookups="false"
               acceptCount="1024"
               server="unknown"
               address="public_ip"
    />

****[UPDATE 3] Output from log files : ****

    2012-06-04 06:18:24,152 [http-bio-ip-8080-exec-3500] ERROR org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/api].[Jersey REST Service]- Servlet.ser
vice() for servlet [Jersey REST Service] in context with path [/socialapi] threw exception
java.net.SocketTimeoutException: Read timed out

    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:532)
    at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
    at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:563)
    at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:118)
    at org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:326)
    at org.apache.coyote.Request.doRead(Request.java:422)

[UPDATE 4] ServletContext

I use a ServletContextListenerin my application to instanciate controllers and keep a reference with event.getServletContext().setAttribute. Those controllers loads configurations and translations (the 88Mb in Perm).

Then to use the database i use :

SessionFactory sf = dbManager.getSessionFactory(DatabaseManager.DB_KEY_DEFAULT);
Session session = sf.openSession();
Transaction tx = null; 

try {
    tx = session.beginTransaction();

    //Do stuuf

    tx.commit();

} catch (Exception e){
    //Do something
} finally {
    session.close();
}

1. Could this be the source of a leak ?
2. Why not to use Manual transaction/session, and how would you do then ?

Solution 1

Try with this parameter:

+XX:+HeapDumpOnOutOfMemoryError -XX:+HeapDumpPath=dump.log

Also try with lower start memory parameters -Xms.

then you can inspect the dump to see if the problem was object allocation.

While running try

jps

That will output all java processes, lets say Tomcat is PID 4444:

jmap -dump:format=b,file=heapdump 4444

And

jhat heapdump

If you run out of memory while executing jhat just add more memory. From there you can inspect the heap of your application.

Another way to go is to enable Hibernate statistics to check that you are not retrieving more objects. Although it looks like a full garbage collection every hour should not be a problem (room for do it better there).

-verbose:gc -Xloggc:/opt/tomcat/logs/gc.out -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

And with GCViewer for example take a look at every space of memory (ternured, eden, survivors, perm).

Another handy tool:

jstack 4444 > stack.txt

That will retrieve a full stack trace of every thread running inside the java process with pid 4444.

Bear in mind that you need privileges if you started Tomcat as root or another user.

jps

won't output process which you have no privileges, therefore you cannot connect to it.

Since I don't know what your application is about (and therefore I don't know its requirements) 3 million instances looks like a lot.

With Hibernate statistics you can see which classes you instantiate the most.

Then tunning the proportions of your eden and ternured garbage recolection can be more efficient.

Newly instantiated objects goes to eden. When it fills up a minor gc triggers. What is not deleted goes to a survivor space. When this fills up it goes to ternured. Full gc will arise when ternured is full.

In this picture (which is inaccurate) I left aside String that become interned and Memory mapped files (that are not in heap). Take a look at which classes you instantiate most. Intensive use of String might lead to quickly fill up perm.

I guess you do so, but use a managed session factory, such as Spring (if in your stack) and avoid manually management of transactions and sessions.

Keep in mind that objects are deleted in the GC when no object refers to it. So as long as a object is reachable in your application the object remain.

If your ServletContextListener instantiate controllers and are stored in the event getServletContext. Make sure you completely remove the reference afterwards, if you keep a reference the objects won't be deleted, since they are still reachable.

If you manage your own transactions and session (which is fine if you cannot use a framework) then you must deal with code maintenance and bugs that Spring-tx for instance has solved and improved.

I personally would take advantage of FOSS. But of course sometime you cannot enlarge the stack.

If you are using Hibernate I would take a look at Spring-orm and Spring-tx to manage transactions and session. Also take a look at Hibernate patter Open Session In View.

Solution 2

I'd also recommend that you download Visual VM 1.3.3, install all the plugins, and attach it to the Tomcat PID so you can see what's happening in real time. Why wait for a thread dump? It'll also tell you CPU, threads, all heap generations, which objects consume the most memory, etc.