Java Memory error: unable to create new native thread

Question

I get this error on my UNIX server, when running my java server:

Exception in thread "Thread-0" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at [... where ever I launch a new Thread ...]

It happens everytime I have about 600 threads running.

I have set up this variable on the server:

$> ulimit -s 128

What looks strange to me is the result of this command, which I ran when the bug occured the last time:

$> free -m
              total       used       free     shared    buffers     cached
Mem:          2048        338       1709          0          0          0
-/+ buffers/cache:        338       1709
Swap:            0          0          0

I launch my java server like this:

$> /usr/bin/java -server -Xss128k -Xmx500m -jar /path/to/myJar.jar

My debian version:

$> cat /etc/debian_version
5.0.8

My java version:

$> java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

My question: I have read on Internet that my program should handle something like 5000 threads or so. So what is going on, and how to fix please ?

---

Edit: this is the output of ulimit -a when I open a shell:

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 794624
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 794624
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

I run the script as a daemon from init.d, and this is what i run:

DAEMON=/usr/bin/java
DAEMON_ARGS="-server -Xss128k -Xmx1024m -jar /path/to/myJar.jar"
ulimit -s 128 && ulimit -n 10240 && start-stop-daemon -b --start --quiet --chuid $USER -m -p $PIDFILE --exec $DAEMON -- $DAEMON_ARGS \
    || return 2

---

Edit2: I have come across this stack overflow question with a java test for threads: how-many-threads-can-a-java-vm-support

    public class DieLikeADog { 
        private static Object s = new Object(); 
        private static int count = 0; 
        public static void main(String[] argv){ 
            for(;;){ 
                new Thread(new Runnable(){ 
                        public void run(){ 
                            synchronized(s){ 
                                count += 1; 
                                System.err.println("New thread #"+count); 
                            } 
                            for(;;){ 
                                try { 
                                    Thread.sleep(100); 
                                } catch (Exception e){ 
                                    System.err.println(e); 
                                } 
                            } 
                        } 
                    }).start(); 
            } 
        } 
    } 

On my server, the program crashes after 613 threads. Now i'm certain this is not normal, and only related to my server configuration. Can anyone help please ?

---

Edit 3: I have come across this article, and many others, explaining that linux can't create 1000 threads, but you guys are telling me that you can do it on your systems. I don't understand.

I have also ran this script on my server: threads_limits.c and the limit is around 620 threads.

My website is now offline and this is the worst thing that could have happened to my project. I don't know how to recompile glibc and this stuff. It's too much work imo.

I guess I should switch to windows server. Because none of the settings proposed on this page did make any change: The limit on my system is between 600 and 620 threads, no matter the program involved.

Solution 1

Just got the following information: This is a limitation imposed by my host provider. This has nothing to do with programming, or linux.

Solution 2

The underlying operating system (Debian Linux in this case) does not allow the process to create any more threads. See here how to raise the maximum amount: Maximum number of threads per process in Linux?

I have read on Internet that my program should handle something like 5000 threads or so.

This depends on the limits set to the OS, amount of running processes etc. With correct settings you can easily reach that many threads. I'm running Ubuntu on my own computer, and I can create around 32000 threads before hitting the limit on a single Java program with all my "normal stuff" running on the background (this was done with a test program that just created threads that went to sleep immediately in an infinite loop). Naturally, that high amount of threads actually doing something would probably screech consumer hardware to a halt pretty fast.

Solution 3

Can you try the same command with a smaller stack size "-Xss64k" and pass on the results ?

Solution 4

Your JVM fails to allocate stack or some other per-thread memory. Lowering the stack size with -Xss will help increase the number of threads you can create before OOM occurs (but JVM will not let you set arbitrarily small stack size).

You can confirm this is the problem by seeing how the number of threads created change as you tweak -Xss or by running strace on your JVM (you'll almost certainly see an mmap() returning ENOMEM right before an exception is thrown).

Check also your ulimit on virtual size, i.e. ulimit -v. Increasing this limit should let you create more threads with the same stack size. Note that resident set size limit (ulimit -m) is ineffective in current Linux kernel.

Also, lowering -Xmx can help by leaving more memory for thread stacks.

Solution 5

Related to the OPs self-answer, but I do not yet have the reputation to comment. I had the identical issue when hosting Tomcat on a V-Server.

All standard means of system checks (process amount/limit, available RAM, etc) indicated a healthy system, while Tomcat crashed with variants of "out of memory / resources / GCThread exceptions".

Turns out some V-Servers have an extra configuration file that limits the amount of allowed Threads per process. In my case (Ubuntu V -Server with Strato, Germany) this was even documented by the hoster, and the restriction can be lifted manually.

Original documentation by Strato (German) here: https://www.strato.de/faq/server/prozesse-vs-threads-bei-linux-v-servern/

tl;dr: How to fix:

-inspect thread limit per process:

systemctl show --property=DefaultTasksMax

-In my case the default was 60, which was insufficient for Tomcat. I changed it to 256:

vim /etc/systemd/system.conf

Change the value for:

DefaultTasksMax=60

to something higher, e.g. 256. (The HTTPS connector of tomcat has a default thread pool of 200, so it should be at least 200.)

Then reboot, to make the changes take effect.

View more solutions