How should I tune apache when I see high CPU, but low memory usage?

Question

My CentOS web nodes (LAMP stack) are experiencing high CPU usage but low memory usage. Nearly all of the processes are apache. I know I could just add additional web nodes until my cpu usage becomes reasonable, but I imagine there are some quick tuning tips that could go a long way here.

CPU bounces between 90-100%. Mysql bounces between 5% and 45%. Apache makes up the rest.

Top:

top - 21:15:49 up 46 days, 20:39,  4 users,  load average: 34.17, 44.62, 51.03
Tasks: 133 total,  32 running, 101 sleeping,   0 stopped,   0 zombie
Cpu(s): 90.4%us,  9.6%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.1%st
Mem:   8388608k total,  6116448k used,  2272160k free,    84160k buffers
Swap: 16777208k total,       68k used, 16777140k free,  3488044k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15775 mysql     15   0  827m 153m 3712 S 43.7  1.9   0:55.04 mysqld
15911 apache    16   0  275m  10m 3872 S 24.1  0.1   0:03.15 httpd
15816 apache    16   0  288m  23m 4116 R 23.5  0.3   0:11.88 httpd
15807 apache    15   0  281m  17m 5028 S 21.5  0.2   0:11.71 httpd
15897 apache    16   0  277m  13m 4104 S 21.2  0.2   0:02.68 httpd
15834 apache    16   0  276m  11m 3916 S 20.6  0.1   0:05.22 httpd
15842 apache    16   0  275m  11m 4116 S 19.3  0.1   0:07.86 httpd
15870 apache    16   0  294m  30m 5044 R 17.4  0.4   0:06.36 httpd
15782 apache    15   0  278m  13m 4124 R 12.2  0.2   0:08.54 httpd
15819 apache    15   0  280m  17m 5016 S 11.6  0.2   0:07.01 httpd
15683 apache    16   0  288m  23m 4128 R 10.9  0.3   0:09.90 httpd
15876 apache    16   0  284m  19m 3880 R 10.6  0.2   0:04.35 httpd
15878 apache    15   0  276m  11m 4104 S 10.6  0.1   0:06.42 httpd
15913 apache    16   0  275m  10m 3920 R  9.6  0.1   0:07.98 httpd
15898 apache    16   0  280m  16m 4104 S  9.3  0.2   0:02.85 httpd
15817 apache    16   0  277m  13m 4108 S  9.0  0.2   0:05.55 httpd
15843 apache    16   0  280m  15m 4104 R  8.7  0.2   0:10.80 httpd
15812 apache    16   0  280m  16m 5012 R  8.0  0.2   0:05.18 httpd
15844 apache    16   0  281m  16m 4116 R  7.4  0.2   0:08.63 httpd
15833 apache    16   0  281m  18m 5036 R  7.1  0.2   0:04.76 httpd
15795 apache    16   0  280m  15m 3920 R  6.8  0.2   0:08.65 httpd
15704 apache    15   0  280m  16m 4096 S  6.4  0.2   0:09.06 httpd
15849 apache    16   0  280m  15m 4124 R  6.4  0.2   0:05.59 httpd
15806 apache    16   0  276m  11m 3876 S  6.1  0.1   0:10.33 httpd
15902 apache    16   0  280m  15m 3912 R  6.1  0.2   0:03.40 httpd
15688 apache    16   0  279m  14m 4116 R  5.8  0.2   0:06.07 httpd
15904 apache    16   0  280m  15m 3860 R  5.8  0.2   0:02.74 httpd
15703 apache    16   0  281m  17m 5048 R  5.1  0.2   0:03.86 httpd
15705 apache    16   0  281m  17m 5044 R  4.5  0.2   0:13.54 httpd
15821 apache    15   0  276m  11m 4072 S  4.5  0.1   0:04.24 httpd
15830 apache    16   0  278m  13m 4112 R  4.2  0.2   0:06.37 httpd
15850 apache    16   0  277m  12m 3872 R  3.9  0.2   0:04.64 httpd
15912 apache    15   0  276m  11m 3920 S  3.9  0.1   0:05.58 httpd
15804 apache    15   0  280m  15m 3932 R  3.5  0.2   0:09.13 httpd
15805 apache    16   0  281m  15m 3932 R  3.5  0.2   0:09.00 httpd
15899 apache    16   0  277m  12m 4104 R  2.6  0.2   0:03.26 httpd
15701 apache    16   0  277m  12m 3920 R  2.3  0.2   0:08.35 httpd
15781 apache    15   0  274m 9.8m 3924 S  1.9  0.1   0:10.48 httpd
15848 apache    15   0  276m  11m 3876 S  1.9  0.1   0:06.63 httpd
15901 apache    15   0  275m  10m 3916 S  1.9  0.1   0:02.18 httpd
15879 apache    16   0  280m  16m 4104 R  1.3  0.2   0:02.03 httpd
15820 apache    16   0  281m  17m 5260 R  1.0  0.2   0:05.24 httpd

relevant httpd.conf:

Timeout 120
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

<IfModule worker.c>
StartServers         2
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>

Solution

It is probably, that some PHP scripts running in mod_php consume too much CPU time. Overall recomendations are general and each of them will have some benefit:

• Install APC cache and check that is really running (APC has bundled apc.php file which shows APC status and cache hits and misses)
• Install nginx and install it as a reverse proxy in front of apache - it will reduce overhead of serving static files and leave much more CPU resources to server dynamic content
• Add CustomLog to apache with %t %T directives and examine the log - you will see wich requests are running the most time, thus consuming most CPU
• optimize/rewrite those scripts

Comments

[HTTP500]

What is your application in? PHP? Are you using an opcode cache like APC?

[Ryan]

Yes, PHP. I have memcache and apc but not utilizing either right now. I'm building a separate memcache server to avoid losing it when rebooting a node.

[Jack ilkgu]

It is possible that your apache processes are waiting for response from MySQL, and piling up. Do you mind posting your MySQL's my.conf? Also, do you have slow logs configured in MySQL? What do those tell you, if you have them?

[grosshat]

Do you work with prefork or worker? And do you face such problems after a new release, code changes, server updates or anything like that?

[Mikko Rantalainen]

The problem is definitely something that runs inside Apache process but is not the apache itself. The mod_php is indeed the most likely cause. You need to profile the PHP scripts you're running and figure out what those are doing to waste the CPU.It is not the database, though. If database is the bottle neck, you'll see lots of Apache processes IO blocked waiting for response from the database. The fact that Apache/PHP processes eat CPU means that they are not waiting for IO (unless you're running scripts that use busy waiting and non-blocking IO, in which case you have found the cause).

[mahyard]

can you guide me about your second recommendation? any HowTo guide?