Locating memory leak in Apache httpd process, PHP/Doctrine-based application

Question

I have a PHP application using these components:

• Apache 2.2.3-31 on Centos 5.4
• PHP 5.2.10
• Xdebug 2.0.5 with Remote Debugging enabled
• APC 3.0.19
• Doctrine ORM for PHP 1.2.1 using Query Caching and Results Caching via APC
• MySQL 5.0.77 using Query Caching

I've noticed that when I start up Apache, I eventually end up 10 child processes. As time goes on, each process will grow in memory until each one approaches 10% of available memory, which begins to slow the server to a crawl since together they grow to take up 100% of memory.

Here is a snapshot of my top output:

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1471 apache    16   0  626m 201m  18m S  0.0 10.2   1:11.02 httpd                                                                                                                                                          
 1470 apache    16   0  622m 198m  18m S  0.0 10.1   1:14.49 httpd                                                                                                                                                          
 1469 apache    16   0  619m 197m  18m S  0.0 10.0   1:11.98 httpd                                                                                                                                                          
 1462 apache    18   0  622m 197m  18m S  0.0 10.0   1:11.27 httpd                                                                                                                                                          
 1460 apache    15   0  622m 195m  18m S  0.0 10.0   1:12.73 httpd                                                                                                                                                          
 1459 apache    16   0  618m 191m  18m S  0.0  9.7   1:13.00 httpd                                                                                                                                                          
 1461 apache    18   0  616m 190m  18m S  0.0  9.7   1:14.09 httpd                                                                                                                                                          
 1468 apache    18   0  613m 190m  18m S  0.0  9.7   1:12.67 httpd                                                                                                                                                          
 7919 apache    18   0  116m  75m  15m S  0.0  3.8   0:19.86 httpd                                                                                                                                                          
 9486 apache    16   0 97.7m  56m  14m S  0.0  2.9   0:13.51 httpd 

I have no long-running scripts (they all terminate eventually, the longest being maybe 2 minutes long), and I am working under the assumption that once each script terminates, the memory it uses gets deallocated. (Maybe someone can correct me on that).

My hunch is that it could be APC, since it stores data between requests, but at the same time, it seems weird that it would store data inside the httpd process.

How can I track down which part of my app is causing the memory leak?

What tools can I use to see how the memory usage is growing inside the httpd process and what is contributing to it?

Solution

There is a very good answer to a similar question here:

How can I determine the cause of an apparent memory leak in my Apache/PHP based web app?

Comments

[UndeadKernel]

I've also learned that Apache child processes do NOT release memory back to the OS, even if the memory inside the process has been freed: * php.net/manual/en/function.apache-child-terminate.php * doughboy.wordpress.com/2008/02/13/apache-worker-and-php/…