debugging stuck apache/php thread on production server

Question

I have a linux system with apache httpd and PHP which is loaded using LoadModule php5_module /usr/lib/apache2/modules/libphp5.so.

I've enabled the mod_status module of apache and I see a particular thread which is stuck doing something since yesterday. I also confirm this by doing ps -axu | grep apache which among the many threads it gives me that particular stuck thread:

www-data  5636  0.0  0.1 423556 23560 ?      S    XXXXX   0:04 /usr/sbin/apache2 -k start

Note that XXXXX is something like Jan02 which is yesterday. Also, the pid (5636) matches the pid of the stuck thread I see in the mod_status page of apache.

My question is: how can I do a thread dump or something similar in order to see where exactly in the PHP code this thing is stuck? Maybe it's waiting for something (i/o, network, db) but I don't know what.

In the java world I'd do a kill -3 pid and get a nice readable thread dump which would clearlly show me where exactly that particular thread is stuck at. Is there a similar technique for the php land?

Solution 1

The following instructions are Linux-centric:

• Identify the faulty / stuck process

In your case, the process is in state S, meaning from man ps:

S interruptible sleep (waiting for an event to complete)

So yes, it is probably waiting for some network or filesystem operation to complete.

• Trace system calls and signals with strace

Attach the strace program to the hanging thread by running:

# strace -p 

This will show you, in real time, the actions or more precisely the syscalls ran by the program, for instance, you might see a loop with open() returning an error such as ENOENT meaning that a particular file is not there.

Your ps output indicates that the process is not consuming CPU (3rd column), so the problem here is probably not related to a loop but just a waiting operation such as a locked file, waiting for a socket or an external action.

• kill and coredumps

The kill program, which is used to send a particular signal to a running program is far from being java-related, it very well can be used to send the signal 3 (SIGQUIT) which will close the program and generate a core file. The generation of a core file is permitted only if the correct ulimit permissions are in place, check it with the ulimit -c command. If it says 0, then you should modify it, for instance, to unlimited:

ulimit -c unlimited

Only then should you restart the application and provoke a coredump by sending a kill -3.

Solution 2

You want to install PHP xdebug extension and enable tracing log. It will then create a file, which will have stats for every executed function - time it took to complete, amount of consumed memory, path to file that contains that function, etc.

This data will help you to identify which function needs fixing, but beware that full tracing log grows in size very quick and you might want to trace only part of your application (which also described in the guide above)...

Comments

[Halfgaar]

Have you found out more about this? I have a similar issue, exactly with mod php which is keeping a worker hostage. If I do a graceful reload, it will also be stuck. I have to kill -9 it.

[shodanshok]

Maybe the wchain field can be interesting. Can you paste the output of ps -eHo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:32,comm ?

[Anubioz]

strace is very unintuitive way to debug php applications - it will tell you which syscalls are executing, but it won't tell you anything about which PHP function does it and why. xdebug is much easier and more effective way to do the same job, since it will show you php functions and scripts instead of syscalls/system libraries...

[cherouvim]

My problem with strace is that it tells me what is going on from now onward. If the thread is already stuck waiting on something then I cannot get any useful info from strace.

[iMil]

So you could launch apache with strace instead of attaching it, for example strace -f -o trace.txt /etc/rc.d/init.d/httpd start will output what happened into a trace.txt file.

[Simon MC. Cheng]

Just curious, have you found anything special in Apache, PHP or system log?

[Anubioz]

@cherouvim xdebug on the other hand will tell you exactly what is wrong with your application, since the name and location of buggy call will be last entry in trace log. It will almost certainly contain some system/exec call, so all you'll need to do will be dumping to some text file the exact command right before it is executed. You'll easily see what's wrong with that external command by launching it in your shell under web-server user account