Where's my memory?! Nginx + PHP-FPM front end webserver slows to a crawl

Question

I'm not sure if I have a problem with a memory leak (as my hosting company suggests), or if we both need to read http://linuxatemyram.com. Maybe you clever people can help us out?

This is a front-end webserver VM running essentially only nginx & php-fpm on RHEL 5.5. This server is powering Magento, a PHP eCommerce thinggy. The server is running in a shared environment, but we're changing that soon.

Anyway.. after a reboot the server runs just fine, but within a day it will grind itself into nothingness. Pages will take literally 2 minutes to load, CPU spikes like crazy, etc.. The console is even sluggish when I SSH in. It's like my whole server is being brought to its knees.

I've also been monitoring the DB server via top and tcpdumping incoming traffic. The DB stays idle for a good portion of that "slow" load time. When i start seeing queries coming from the front-end server, the page loads soon afterward.

Here are some stats after me logging in during a slow-down, after restarting php-fpm:

[mike@front01 ~]$ free -m
             total       used       free     shared    buffers     cached
Mem:          5963       5217        745          0        192        314
-/+ buffers/cache:       4711       1252
Swap:         4047          4       4042


[mike@front01 ~]$ top
top - 11:38:55 up 2 days,  1:01,  3 users,  load average: 0.06, 0.17, 0.21
Tasks: 131 total,   1 running, 130 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.0%us,  0.3%sy,  0.0%ni, 99.3%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   6106800k total,  5361288k used,   745512k free,   199960k buffers
Swap:  4144728k total,     4976k used,  4139752k free,   328480k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
31806 apache    15   0  601m 120m  37m S  0.0  2.0   0:22.23 php-fpm
31805 apache    15   0  549m  66m  31m S  0.0  1.1   0:14.54 php-fpm
31809 apache    16   0  547m  65m  32m S  0.0  1.1   0:12.84 php-fpm
32285 apache    15   0  546m  63m  33m S  0.0  1.1   0:09.22 php-fpm
32373 apache    15   0  546m  62m  32m S  0.0  1.1   0:09.66 php-fpm
31808 apache    16   0  543m  60m  35m S  0.0  1.0   0:18.93 php-fpm
31807 apache    16   0  533m  49m  30m S  0.0  0.8   0:08.93 php-fpm
32092 apache    15   0  535m  48m  27m S  0.0  0.8   0:06.67 php-fpm
 4392 root      18   0  194m  10m 7184 S  0.0  0.2   0:06.96 cvd
 4064 root      15   0  154m 8304 4220 S  0.0  0.1   3:55.57 snmpd
 4394 root      15   0  119m 5660 2944 S  0.0  0.1   0:02.84 EvMgrC
31804 root      15   0  519m 5180  932 S  0.0  0.1   0:00.46 php-fpm
 4138 ntp       15   0 23396 5032 3904 S  0.0  0.1   0:02.38 ntpd
  643 nginx     15   0 95276 4408 1524 S  0.0  0.1   0:01.15 nginx
 5131 root      16   0 90128 3340 2600 S  0.0  0.1   0:01.41 sshd
28467 root      15   0 90128 3340 2600 S  0.0  0.1   0:00.35 sshd
32602 root      16   0 90128 3332 2600 S  0.0  0.1   0:00.36 sshd
 1614 root      16   0 90128 3308 2588 S  0.0  0.1   0:00.02 sshd
 2817 root       5 -10  7216 3140 1724 S  0.0  0.1   0:03.80 iscsid
 4161 root      15   0 66948 2340  800 S  0.0  0.0   0:10.35 sendmail
 1617 nicole    17   0 53876 2000 1516 S  0.0  0.0   0:00.02 sftp-server
 ...

Is there anything else I should be looking at, or any more information that might be useful? I'm just a developer, but the slowdowns on this system worry me and make it hard to do my work..

Help me out, ServerFault!

Solution 1

In case anyone else suffers this.

We have just experienced the same issue. A memory leak in php5-fpm. The RAM is used up with every page request and eventually maxed out. Then the CPU goes into overdrive with the KSWAP process running the swap disk.

the only thing that fixed, although not the ideal setup was to change our www.conf pool file

pm = dynamic

to

pm = ondemand

memory now seems stable.

Solution 2

Here's two things to check:

1. Most likely you have a memory leak. pm.max_requests will kill/restart the php-fpm child process after it's handled this many requests. Since it takes about a day for your box to slow to a crawl, try setting this to a number to causes each child process to be respawned every 20 minutes or so. So if you get 200 requests a minute, and you have 5 processes, set pm.max_requests to 800. If you don't want to do the math, a setting between 500-1000 might work. Play around with it to find a balance between wasting time respawning processes vs wasting RAM on the memory leak.

2. Php-FPM may be creating too many child processes when a traffic spike happens, causing it to run out of RAM and start swapping to disk. Decide how much total RAM to allocate to php-fpm, then divide that by how much memory each child process takes. For Wordpress and PHP-based forums, I often see each child process requiring 30-45MB. If you're only using one php-fpm pool, set pm.max_children to that number.

If you've only got a single php-fpm pool, you'll see a speed boost from setting pm.type = static.

If you've got multiple php-fpm pools, possibly because you're hosting multiple apps and wish to isolate them for security reasons, you'll want to set pm.type = dynamic and play around with start_servers, min_spare_servers, and max_spare_servers. Just be sure the cumulative max_children across all pools isn't more than your box can handle.

If you've got a high number of low-traffic apps, each with its own php-fpm pool, it's better to set pm.type=ondemand so that each app is only taking up resources when it's actually being used. Also set pm.max_children reasonably low so that no one app can completely overwhelm the box.

Solution 3

My guess is that php5 just got updated with the default memory limit of 128Mb and there are too many php5 instances running ( check pm.max_children in your php-fpm config ).

The above happened to mee too, and I've been pulling my hair trying to figure it out. I have logged a bugreport with ubuntu, but I assume it needs to be fixed upstreams with php-fpm -- the defaults I had would eat up 6Gb RAM if it could. And that sure as hell made my 1Gb server useless...

Solution 4

You probably don't have some opcode cache installed on the server which is causing the slowness.

Moreover, as Marco said, you might be running a large number of php-fpm instances on the system.

In case you suspect a memory leak trouble, install suhosin extension for php, which will prevent memory leaks.

Actually if it is a memory leak it should be logged in the php error log, php.ini has an option to report memleaks, report_memleaks = On

Solution 5

The next time the server slows down, run 'vmstat 1' and 'iostat 1', then report the results back to us.

View more solutions

Comments

[Mahn]

One year later, how did this work out? Did you end up keeping this configuration?

[Scott Buchanan]

Note that the report_memleaks feature likely won't help in production: "This parameter only has effect in a debug build, and if error_reporting includes E_WARNING in the allowed list."

[jgmjgm]

The problem with this is that when a PHP request finishes, it should free its memory and drop back down to nominal amounts, even on a single process. If PHP processes are still using the max mem even when idle then something might be up with the GC. /As far as I'm aware a process can handle a request once at a time. PHP instantiates itself to run the request, then destroys everything on script completion.

[tmarois]

I would also like to know how this worked out. Its been many years now. I'm seeing similar issues today.

[Michael Chourdakis]

Still valid after 8 years, still the same leak, still the same fix....