Too many httpd processes running and CentOS server freeze

Question

I am running a high traffic website on a Dedicated-Virtual 16GB RAM CentOS server hosted by Media Temple.
Very often the server dies because of high CPU usage (%1500)
When i check the running processes using "top" command i see a too many httpd processes running which i think are causing the high usage of CPU.
I have tried to tune mysql, tune apache, my.cnf, httpd.cnf, removed Apache modules that i do not use but that does not solve the problem.
Please let me know what values do you need to know in my server config in order to help me diagnosis the problem.
Thanks in advance.

Solution

what does tail -f /var/log/messages say when you experience high system loads?

Are your processes very I/O or very CPU intensive?

One observation:

Looks like you have 18G physical memory, but zero swap space..., e.g. your top command shows "Swap: 0k total" ... that means that you do not have any Swap Space configured.

In general, you should always have a sufficient swap space on a UNIX system! Swap-size = 1...2-times RAM size is a good idea. Using a fast partition is a good idea. Really bad things happen if your UNIX system runs out of RAM and doesn't have Swap .. processes just die inexplicably.. that is a very bad thing! especially in production. Disk is cheap! add a generous swap partition! :-) or worst case you can also create a swap-file later.

On any UNIX system, you do need swap space -- because that is where processes live while they are not scheduled on one of the CPUs. If you don't have swap space, really bad things will happen, because your system will run out of places (RAM) where to create new processes -- it can not put them in swap, because there is none, so you will see extremely high loads, the system freezing up, and processes die seemingly inexplicably.

To check your memory and swap space in use, you can run free -k. You can also check the swap configuration by running swapon -s .. this will not show any output if no swap is configured.

As a rule of thumb, configure Swap Space approximately 1..2 times the size of your physical memory. I'd say if you have 18G RAM, configure ~20..30G Swap -- don't be stingy with the Swap, because "disk is dirt-cheap"!

Disclaimer: there are a few exceptions when you can live without swap, but on the other hand, configuring and not needing it does not hurt! :)

Two choices for creating swap space:

• either create a swap partition, preferably on your fastest disk partition (best choice; or add a flash drive to your physical server)

• or create a swap file (if you don't have disk space left to partition; this is also a good option if you later find out that you need more swap than you anticipated)

...then format that swap partition or swap file to make it usable for swapping with the mkswap command.

Once you enable the swap space with sudo swapon -a, you should see it listed in top and with swapon -s , and your system should behave much more nicely..

Articles about Swap Space:

https://www.linux.com/news/software/applications/8208-all-about-linux-swap-space

https://www.linux.com/learn/tutorials/442430-increase-your-available-swap-space-with-a-swap-file

http://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/

http://www.thegeekstuff.com/2010/08/how-to-add-swap-space/

http://lissot.net/partition/partition-08.html

http://tldp.org/LDP/sag/html/swap-space.html

Some pointers:

man -k swap

man mkswap

man fstab , vim /etc/fstab

---

EDIT

as this question was migrated to ServerFault .. I'll add a comment here, because ppl are complaining that I quote the old rule: " 1..2-times RAM size == Swap size "

A 2TB SATA drive costs around $70 these days, that means 20GB costs you about 70 cents -- it's dirt cheap! I know 20GB sounds a lot, but think of it as a very inexpensive "worst case insurance"!

Somebody said: "if your system filled up 18GB and is swapping onto 20GB, then you have bigger problems"

Certainly true, but having generous Swap Space if unexpected things go wrong, will save your server and the processes on it from crashing completely! You might still be able to log in, examine the system while it's still running (slowly), and repair things (albeit slowly), and not have to reboot.. On a production server, there is really no reason not to use generous swap.

If I had a server with 18GB RAM and it's running MongoDB for example (which is very RAM intensive) .. I would certainly configure very generous Swap Space, perhaps even 40..60GB ... just in case something unexpected goes wrong..

Comments

[Admin]

you can't have 1500% CPU without at least 15 processors - how are you determining this?

[Admin]

Here is the top 15 lines of 'top' command pastebin.com/5c16JN4G

[Admin]

yeah, what does top say? If you hit 'I' it switches between IRIX mode which shows 100% per core, so it can be higher than 100%, but you're probably seeing load average not CPU.

[Admin]

Hrm, so yes, all your httpds are running hot and using a full core each.

[MarkR]

Can you access the Apache server-status page to see what all those httpds are doing? Is anything showing up in your error log?

[craphunter]

Here is my top pastebin.com/GSzDhZ22

[Janne Pikkarainen]

"As a rule of thumb, configure Swap Space approximately 1..2 times the size of your physical memory." .. this was true about ten years ago. Today, if you have 20 GB of swap in active use, your server probably consumes more time in transferring stuff from and to swap, and has less time for actual work.

[Deb]

Except the top information shows zero swap-space even configured. I'm not sure where you're getting 20GB of it.

[sandroid]

Great answer, and generally true, but it should be clear that this remains a generalization. There are some environments where you can safely, through various controls, deploy a unix-based server and not have any swap. I've done this many times, though admittedly never in an environment where a daemon like apache httpd was running, as it spawns itself to take on new requests which is more challenging to manage.

[Virulence]

"As a rule of thumb, configure Swap Space approximately 1..2 times the size of your physical memory. I'd say if you have 18G RAM, configure ~20..30G Swap "

[Tilo]

@sandroid : yes, certainly true! I just wanted to point out to user sunn that disk space is ridiculously cheap, and there's really no good reason to not configure swap in a regular server or desktop system -- even if it's 20GB (which sounds a lot, but it's dirt cheap).

[Tilo]

@JannePikkarainen : yes, you're absolutely right! The 1..2 times rule is old .. but think about how dirt cheap disk space is -- a 2TB disk costs ~$70 these days .. .that's 70 cents for the 20GB! There's no good reason to not add swap -- even if it's 20GB -- it's sort of a "worst-case insurance" when unexpected things happen..

[Tilo]

20GB of swap costs you about 70 cents these days.. there is really no reason to be stingy with it

[Mircea Vutcovici]

I prefer OOM-Killer to do its job and kill the apache, than having a server trashing that I can not connect remotely.

[BTR Naidu]

Were you able to solve your problem?

[Damodar Bashyal]

Please see meminfo - shrib.com/#BskRvetZWCeOcgQXV0MA Last few days memory issue is giving hard time. I can only see lots of httpd processes running.