"java.lang.OutOfMemoryError : unable to create new native Thread"

Question

We are getting "java.lang.OutOfMemoryError : unable to create new native Thread" on 8GB RAM VM after 32k threads (ps -eLF| grep -c java)

However, "top" and "free -m" shows 50% free memory available. JDk is 64 bit and tried with both HotSpot and JRockit.Server has Linux 2.6.18

We also tried OS stack size (ulimit -s) tweaking and max process(ulimit -u) limits, limit.conf increase but all in vain.

Also we tried almost all possible of heap size combinations, keeping it low, high etc.

The script we use to run application is

/opt/jrockit-jdk1.6/bin/java -Xms512m -Xmx512m -Xss128k -jar JavaNatSimulator.jar /opt/tools/jnatclients/natSimulator.properties

Thanks for the reply.

We have tried editing /etc/security/limits.conf and ulimit but still that same

[root@jboss02 ~]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 72192
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 72192
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Solution 1

This is not a memory problem even though the exception name highly suggests so, but an operating system resource problem. You are running out of native threads, i.e. how many threads the operating system will allow your JVM to use.

This is an uncommon problem, because you rarely need that many. Do you have a lot of unconditional thread spawning where the threads should but doesn't finish?

You might consider rewriting into using Callable/Runnables under the control of an Executor if at all possible. There are plenty of standard executors with various behavior which your code can easily control.

(There are many reasons why the number of threads is limited, but they vary from operating system to operating system)

Solution 2

I encountered same issue during the load test, the reason is because of JVM is unable to create a new Java thread further. Below is the JVM source code

if (native_thread->osthread() == NULL) {    
// No one should hold a reference to the 'native_thread'.    
    delete native_thread;   
if (JvmtiExport::should_post_resource_exhausted()) {      
    JvmtiExport::post_resource_exhausted(        
        JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR | 
        JVMTI_RESOURCE_EXHAUSTED_THREADS, 
        "unable to create new native thread");    
    } THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(), "unable to create new native thread");  
} Thread::start(native_thread);`

Root cause : JVM throws this exception when JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR (resources exhausted (means memory exhausted) ) or JVMTI_RESOURCE_EXHAUSTED_THREADS (Threads exhausted).

In my case Jboss is creating too many threads , to serve the request, but all the threads are blocked . Because of this, JVM is exhausted with threads as well with memory (each thread holds memory , which is not released , because each thread is blocked).

Analyzed the java thread dumps observed nearly 61K threads are blocked by one of our method, which is causing this issue . Below is the portion of Thread dump

"SimpleAsyncTaskExecutor-16562" #38070 prio=5 os_prio=0 tid=0x00007f9985440000 nid=0x2ca6 waiting for monitor entry [0x00007f9d58c2d000]
   java.lang.Thread.State: BLOCKED (on object monitor)

Solution 3

If jvm is started via systemd, there might be a maxTasks per process limit (tasks actually mean threads) in some linux OS.

You can check this by running "service status" and check if there is a maxTasks limit. If there is, you can remove it by editing /etc/systemd/system.conf, adding a config: DefaultTasksMax=infinity

Solution 4

It's likely that your OS does not allow the number of threads you're trying to create, or you're hitting some limit in the JVM. Especially if it's such a round number as 32k, a limit of one kind or another is a very likely culprit.

Are you sure you truly need 32k threads? Most modern languages have some kind of support for pools of reusable threads - I'm sure Java has something in place too (like ExecutorService, as user Jesper mentioned). Perhaps you could request threads from such a pool, instead of manually creating new ones.

Solution 5

I would recommend to also look at the Thread Stack Size and see if you get more threads created. The default Thread Stack Size for JRockit 1.5/1.6 is 1 MB for 64-bit VM on Linux OS. 32K threads will require a significant amount of physical and virtual memory to honor this requirement.

Try to reduce the Stack Size to 512 KB as a starting point and see if it helps creating more threads for your application. I also recommend to explore horizontal scaling e.g. splitting your application processing across more physical or virtual machines.

When using a 64-bit VM, the true limit will depend on the OS physical and virtual memory availability and OS tuning parameters such as ulimitc. I also recommend the following article as a reference:

OutOfMemoryError: unable to create new native thread – Problem Demystified

View more solutions

Comments

[Jesper]

Operating systems have limits on the number of threads that you can create. Why are you creating more than 32k threads? Your system does most likely not have thousands of processor cores, creating so many threads is not useful. Use a thread pool (ExecutorService) instead.

[Deepak Tewani]

Thanks for the reply. We are using an open source library and trying to load test that. Any that open source library is creating so many threads. But what i dont understand, is when "top" is showing 50% free memory then why OutOfMemory Error.

[Deepak Tewani]

The open source library that we are using in ICE4j Library

[Andrzej Doyle]

OutOfMemoryError does not necessarily mean heap space, or "general" RAM, was exhausted. In this case it's clear that the failure was due to the OS not having the resources to allocate an extra thread. Having 50% free memory is irrelevant to this particular failure.

[Deepak Tewani]

What are the other resources required for creating new threads. We were under the impression that if we increase the RAM, then we may able to create more threads. Kindly guide us

[Deepak Tewani]

We tried using ExecutorService, then also number of threads are 32K, it is because all threads are forever live threads.

[Omer]

there is seems to be some open source lib which creating threads but not able to finish/stop the threads properly, may be all the threads were created and went into sleep mode.

[Harshad Chhaiya]

@DeepakTewani - I am facing a similar issue. Did you get the solution?

[Deepak Tewani]

Thanks for the reply We are using an open source library ICE4j and trying to load test that. Cant we increase the limit of threads in OS when we know that there is 50% memory left on the server.

[Deepak Tewani]

Thanks for the reply. We are using an open source library ICE4j and trying to load test that. Cant we increase the limit of threads in OS when we know that there is 50% memory left on the server.

[matbrgz]

Possibly, but I think it will not help you as such. If you run out of resources when load testing you need to be able to control what happens in your application. Why do you have 32000 threads active at once?

[Deepak Tewani]

We are creating 11K clients that uses 32 K threads for reading, writing data on UDP sockets. Out of these 32 K threads, 10K threads are keep alive threads that are used to keep the socket open

[Deepak Tewani]

We are creating 11K clients that uses 32 K threads for reading, writing data on UDP sockets. Out of these 32 K threads, 10K threads are keep alive threads that are used to keep the socket open

[matbrgz]

I believe this problem is solved in modern web servers. Also udp can loose packets - any reason you do not just use a web server?

[Deepak Tewani]

We have all already tried that change that is given on that link. But result is same :(

[Deepak Tewani]

We cant use web servers because, what are we creating are clients that would talk with other clients. Both the clients can be behind symmetric, asymmetric nats . For their connectivity , we are using ice4j library. What i am not able to understand is , if server has 50% memory left then what jvm is not able to create more threads

[matbrgz]

Because OutOfMemory exception should have been named OutOfResources. The operating system cannot provide the resource you need. (And it turned out I did not know ice4j)

[Deepak Tewani]

So , if we use 16GB RAM (double the memory that we are using now), then,are we able to create double number threads then now?

[Deepak Tewani]

let us continue this discussion in chat

[matbrgz]

You misunderstand. The reason Linux does not give you more threads is not because it is running out of memory but because you run into a limit set by Linux itself (this can be an arbitrary rule or because the data structure used isn't any larger). Note that regardless of how much you raise this limit your application must still be able to cope. I suggest you revisit your "a thread per connection" design.

[Deepak Tewani]

We also tried ExecutorService for thread and then also the application is creating 32K threads. ExecutorService is creating so many threads because all the threads created by application are forever live threads. I understand that, Linux have a limit of giving threads to an application, but cant we increase this limit?

[matbrgz]

Probably, but not something I've interested me for so I do not know how. Consider opening a new question asking that on the Linux SE site.

[matbrgz]

Also consider finding out how the ice4j community suggest you should solve this problem. You might have missed something in the documentation.

[T.E.D.]

@DeepakTewani - Perhaps running two or more copies of your program will do what you want? Still, I'd think if you are running the system low on threads and things are still working, most likely your program passed the load test. :-)

[ftrujillo]

This is not a solution to the question.

[matbrgz]

How was the method blocking? Never returning?

[sayap]

To expand on this, if you SSH into a server infected by systemd and then start a java process from CLI, you can also hit the silly tasks limit imposed on the sshd service.

[Ivan Herlambang]

Thanks! This is one of the solutions for thread problem. There's also an option called TasksMax (for more isolated settings) for services/systemd that are under [Service] TasksMax=(value). At SUSE, the default value is 512 (which is very limited for production use). More info suse.com/support/kb/doc/?id=000015901 systemctl daemon-reload, and to check, systemctl show --property DefaultTasksMax (for global) or systemctl status ${serviceName}|grep -e Tasks (for TasksMax) There's other things that cause "unable to create new native Thread” issues, such as the ulimit setup.

[MichaelRom]

Methods can block in many ways: waiting on some shared, synchronized object (in this case), waiting for a slow network call to timeout, or waiting just on some slow OS service.