Apache tuning with very high-load LAMP server

Question

I have a LAMP server that can be serving ~400 requests per second at times, the server configuration is:

CPU: Intel Quad Core Xeon X3430 (4 x 2.40 GHz, 8MB Cache)
RAM: 16 GB REG ECC DDR3
HD: 500 GB Enterprise Grade SATA II
OS: CentOS 64 Bit (Latest Stable)

Apache2 prefork config:

StartServers      128
MinSpareServers   16
MaxSpareServers   64
ServerLimit     1500
MaxClients      1500
MaxRequestsPerChild  10000

At peak times the server load is very high, cpu utilization around 90% and load average around 130

I'm not sure it's server hardware limit or my prefork config have something wrong - What could be the problem?

Solution 1

You are allowing Apache to spawn up to 1500 children to serve requests (ServerLimit / MaxClients) -- It's no wonder your server load (the number of processes waiting in the run queue) is getting enormous!

My first suggestions at 400 requests per second with the numbers you're quoting in your question would be "Move the MySQL server to its own box" or perhaps a better choice: "Add another web server and load balance your requests".
To help you figure out how to size the additional server follow the advice HDDP500 gave in their answer to figure the average size of an Apache process. Figure out how many apache processes you want to run on the new server and determine how much RAM you'll need to handle them. Remember to add a safety margin (a gig or two).

Figuring out CPU requirements (speed/number of cores) is a little more difficult - You will need to take into account how much "work" the server has to do to generate each page. A tool like XDebug can help you out here by telling you how long it takes to generate a page (you should perform this test on an unloaded server as well as one under load, but you should also not use a tool like XDebug on a production server in most cases.)

Solution 2

What is the average size of your httpd processes?

Run this command when the server is under load:

ps -ylC httpd --sort:rss | awk '{sum+=$8; ++n} END {print "Tot="sum"("n")";print "Avg="sum"/"n"="sum/n/1024"MB"}'

That will tell you approximately the average size of an Apache process.

Your MaxClients is probably way too high.

Comments

[ceejayoz]

Investigate a CDN, move the database onto a dedicated server, implement caching, etc. Lots of things you can do.

[Roy]

It's dedicated server, Caching & CDN have been used.

[Khaled]

Have you looked at server memory usage and IO load?

[ceejayoz]

@Roy If MySQL is already on its own dedicated server, you don't have a LAMP server, you have a LAP server.

[Roy]

Yes, my server us runing MySQL also, do you mean I should separate the database server? but the server do not have many MySQL query cos many page is cached.

[voretaq7]

I assume you're experiencing performance problems?

[Tom O'Connor]

+1 for this. Move MySQL. Get 2+ Apache nodes. Use HAProxy/Varnish to load balance.

[voretaq7]

you could probably even get away with DNS round-robin for a little while, but a real load balancing solution (I'll throw pf or pfsense into the ring) is definitely preferable