Python: Can't start new thread. <100 active threads

Question

I'm getting the following error:

----- Match 93028: ------ Patch 5.11 ------78 Threads Active
----- Match 93029: ------ Patch 5.11 ------77 Threads Active
----- Match 93030: ------ Patch 5.11 ------76 Threads Active
----- Match 93031: ------ Patch 5.11 ------71 Threads Active
----- Match 93032: ------ Patch 5.11 ------55 Threads Active
----- Match 93033: ------ Patch 5.11 ------56 Threads Active
----- Match 93034: ------ Patch 5.11 ------57 Threads Active
----- Match 93035: ------ Patch 5.11 ------58 Threads Active
----- Match 93036: ------ Patch 5.11 ------59 Threads Active
Traceback (most recent call last):
  File "pulldata.py", line 91, in <module>
    getPatchData('5.11', '511')
  File "pulldata.py", line 64, in getPatchData
    matchThread.start()
  File "/usr/lib/python3.4/threading.py", line 850, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

Normally this is caused by having too many threads open but as you can see I'm also printing the number of threads active. There are <100 active threads so I'm not sure what the problem is. Here is the relevant code:

slot = threading.BoundedSemaphore(value=1000)
def getMatchData(index,match,patch):
    global requestsSent
    global logfile
    print("----- Match {0}: ------ Patch {1} ------{2} Threads Active".format(index,patch,threading.active_count()))
    logfile.write("Parsing Match {0} for patch {1}:\n".format(index,patch))

    #match is a class. get is a function that sends a request to the server and returns a request object from where I get the json response.
    data = match.get().json()

    #processdata

    slot.release()

def getPatchData(patch, name):
    global logfile
    threads = []
    matches = getAllMatches(patch)
    for index, match in enumerate(matches):
        slot.acquire()
        matchThread = threading.Thread(target=getMatchData, args=(index,match,patch))
        threads.append(matchThread)
        matchThread.start()
        for t in threads:
            if not t.isAlive():
                threads.remove(t)

    for t in threads:
        t.join()

The slots semaphore is supposed to limit the number of active threads but I don't think I ever reached 1000 threads anyway. Before I assumed this error was being caused due to threads being pointed to by my array of threads so I added code to remove them from the array when they were no longer active.

I can't understand why I get can't start a new thread when there are only 59 active threads.

Also, is there a better way to achieve what I am trying to do? Each thread sends a request to an API. I tried doing it without concurrency but I wasn't even coming close to my rate limit.

Solution 1

I ran into a similar issue and here is how I solved it.

Not sure what OS the OP is using, but on Linux there is usually a limit on the number of processes per user. You can see it with ulimit -u (or also ulimit -a). The definition is a bit of a misnomer, as the limit is actually on the number of OS threads (or LWP). (see the accepted answer at: https://superuser.com/questions/376532/does-gnu-linux-counts-processes-and-threads-together-when-i-limit-their-number)

On my system, the limit appears to be set to 400 (but it can be changed by the admin).

You can see a list of all your threads with the command:

ps -fLu <your_username>

In my case, my python application would raise the same exception as reported by the OP, but threading.active_count() would return 7.

It turned out I had lots of left-over processes from previous sessions (I had been a bit too keen with nohup...), with several threads each, hanging around in the system. Removing them got rid of the thread creation error.

Solution 2

I was running on a similar situation, but my process needed a lot of threads running.

I counted the number of threads with the command:

ps -fLu user | wc -l

It displayed 4098.

I switched to the user and looked to system limits:

sudo -u myuser -s /bin/bash

ulimit -u

Got 4096 as response.

So, I edited /etc/security/limits.d/30-myuser.conf and added the lines:

myuser hard nproc 8192

myuser soft nproc 8192

Restarted the service and now it's running with 7017 threads.

Ps. I have a 32 cores server and I'm handling 18k simultaneous connections with this configuration.

Comments

[Abhishek Jain]

How did you actually find the hanging threads? We are facing something similar.