Dump stacktraces of all active Threads

Question

I'm trying to dump a list of all active threads including the current stack of each. I can get a list of all threads using threading.enumerate(), but i can't figure out a way to get to the stack from there.

Background: A Zope/Plone app freaks out from time to time, consuming 100% of cpu and needs to be restarted. I have a feeling it's a loop which doesn't terminate properly, but i cannot reproduce it in the test-environemt for verification. I managed to register a signal handler which can be triggered from the outside, so i can trigger some code as soon as the situation occurs again. If I could dump the stacktrace for all active threads, that would give me a clue what goes wrong. The hole thing runs on python 2.4...

Any ideas on how to trace down situations like these are appreciated :)

Cheers, Chriss

Solution 1

When using Zope, you want to install Products.signalstack or mr.freeze; these were designed for just this purpose!

Send a USR1 signal to your Zope server and it'll immediately dump stack traces for all threads to the console. It'll do this even if all Zope threads are locked up.

Under the hood these packages indirectly use threadframes; for Python versions 2.5 and up, when not using Zope, you can build the same functionality using the sys._current_frames() function to access per-thread stack frames.

As of Zope 2.12.5 this functionality is integrated into Zope itself, and there is no need to install additional packages anymore.

Solution 2

As jitter points out in an earlier answer sys._current_frames() gives you what you need for v2.5+. For the lazy the following code snippet worked for me and may help you:

print >> sys.stderr, "\n*** STACKTRACE - START ***\n"
code = []
for threadId, stack in sys._current_frames().items():
    code.append("\n# ThreadID: %s" % threadId)
    for filename, lineno, name, line in traceback.extract_stack(stack):
        code.append('File: "%s", line %d, in %s' % (filename,
                                                    lineno, name))
        if line:
            code.append("  %s" % (line.strip()))

for line in code:
    print >> sys.stderr, line
print >> sys.stderr, "\n*** STACKTRACE - END ***\n"

Solution 3

For Python 3.3 and later, there is faulthandler.dump_traceback().

The code below produces similar output, but includes the thread name and could be enhanced to print more information.

for th in threading.enumerate():
    print(th)
    traceback.print_stack(sys._current_frames()[th.ident])
    print()

Solution 4

2.4. Too bad. From Python 2.5 on there is sys._current_frames().

But you could try threadframe. And if the makefile gives you trouble you could try this setup.py for threadframe

Sample output when using threadframe

Solution 5

Just for completeness sake, Products.LongRequestLogger is super helpful to identify bottlenecks, and to do so it dumps stacktraces at specific intervals.

View more solutions

Comments

[Danimal]

Nowadays in Plone "no special packages are necessary" stackoverflow.com/a/36633215/3046069

[Martijn Pieters]

@Danimal: thanks, added to the post. 2.12.5 was released a year after I wrote this answer.

[Imran Rashid]

The link to threadframes is dead now, but in any case, as of python 2.5 you can use sys._current_frames().

[Martijn Pieters]

@ImranRashid: yes, my answer already states you can. I've replaced the broken link.

[Anton Kraievyi]

beware: faulthandler code is segfaulting in python 3.9.4 (which is currently the latest in ubuntu 21.04), and this has been fixes roughly two months ago in 3.9.7 python release