Weblogic intensive CPU usage

Question

I am using Weblogic 10.3.4 + Oracle DB both on Solaris machines (VM)

Unfortunately last week our app hosted on Weblogic became very very slow not being able to serve new users coming (everything lasted for 1h).

In order to find out the cause is we asked reports from the OS level , DB level.

The reports shown:

• a very intesive CPU usage that hour. (clearly was a peak from number of users using the system)
• Connection Pool reached the top limit on Weblogic (200 connections !)
• AWR stats from Db shown a degradation in response times.

NOW who was first the chicken or the egg ?

A very slow DB can cause WEBlogic to be slow and to reach the max numeber of connections from pool.

However if the CPU is very used , it might be that the threads holding the DB connections to be so busy that they do not return the connection to the pool (because the processor is busy)==> a new client comes a new connection is assgined and so on ...so the connection pool might be exhausted as well in this way ...?

Regards

Cris

Solution

High CPU usage is often caused by the java garbage collector that is trying to collect free memory in the JVM heap space, perm gen, etc. If this is the case, you need to add more memory, usually to JVM heap. See the -Xmx JVM option.

When the CPU usage is high in a JVM, try to find out which thread(s) is(are) using the CPU. During the problem create 2-3 thread dumps with pkill -3 java. The thread dumps will be located (for Solaris) in /proc/<JVM_PID>/path/1

If the thread pool would reach its limit, you should see errors in the logs. The CPU usage should not rise because of this. It is an event driven environment and no need to do pooling.

Slow DB could actually lower the CPU usage on the application server. Depends on how the application is programmed/configured.

Comments

[grassroot]

Please provide more details of the setup. By VM, do you mean Virtual Machines? What is the the virtualization technology used? Basically Solaris or VMWare? Are these servers running on same physical host(s)? What kind of capacity you have in the servers? Do you have Sar-statistics available of the situation? The AWR is able to provide you in detail what happened in the database at the time. The "Top 5"-list will give you pointers to start looking deeper from.

[Cris]

Thanks ...we can't add more memory...it might create even longer GC's we have 4GB / manages server (4 managed servers). It is a post mortem analysis...which i want to reproduce via stress and load testing. Don;t know know which threads used what...what I have are statistics from Domain health...connection pool was exhausted ...and new requests were coming waiting for a db connection...which never came ==> 400 threads waiting on each managed server might result in high CPU usage no ?

[Cris]

I did not understood what you meant by "It is an event driven environment and no need to do pooling." Weblogic has a pool for DB connections and as well a self tuning pool for its threads.

[Mircea Vutcovici]

It depends on the application and on the implementation of the connection pool. Threads waiting = 0% CPU usage.

[Mircea Vutcovici]

Usually the connection pools are implemented to wait for events, as opposed to check if the state has changed (pooling). Those threads will be either LOCKED or in a BLOCKED state, waiting for the kernel to wake them.

[Mircea Vutcovici]

Do you have garbage collector logs? Do you have the STDOUT logs of the JVMs?

[Mircea Vutcovici]

Are you running a 32 bit JVM or a 64 bit one? Is the java installed to be multi-lib? Run: java -version does it show: "mixed mode"?

[Mircea Vutcovici]

Check that the processes are running in 64 bit virtual memory space with pldd <JVM_PID> or pmap <JVM_PID>|grep .so

[Mircea Vutcovici]

At 4GB of heap space you can reach the limit of 32 bit virtual memory space. Check that the JVMs are started with -d64

[Mircea Vutcovici]

4GB is per java virtual machine (JVM) or is per Solaris non global zone?

[Cris]

it is 64 bit, started with -d64, 4gbfor jvm where weblogic is started not global zone. However now I asked datacenter to start JVM with heap dump on oome and with GC logs to have more info in case of this happens again....i try meanwhile to reproduce in a similar environment

[Cris]

will check about ...but i remember is mixed mode.You gave me a good idea...i will simulated a slow DB ...and perform some tests...to see how CPU behaves during this scenario