Very high CPU and low RAM usage - is it possible to place some of swap some of the CPU usage to the RAM (with CloudLinux LVE Manager installed)?

Question

I had to install CloudLinux so that I could somewhat controle the CPU ussage and more importantly the Concurrent-Connections the Websites use. But as you can see the Server load is way to high and thats why some sites take up to 10 sec. to load!

• Server load 22.46 (8 CPUs) (!)
• Memory Used 36.32% (2,959,188 of 8,146,632) (ok)
• Swap Used 0.01% (132 of 2,104,504) (ok)

---

Server:

• 8 x Intel(R) Xeon(R) CPU E31230 @ 3.20GHz
• Memory: 8143680k/9437184k available (2621k kernel code, 234872k reserved, 1403k data, 244k init)
• Linux

Yesterday: Total of 214,514 Page-views (Awstat)

Now my question: Can I shift some of the CPU usage to the RAM?

Or what else could I do to make the sites run faster (websites are dynamic - so SQL heavy)

Thanks

top - 06:10:14 up 29 days, 20:37,  1 user,  load average: 11.16, 13.19, 12.81
Tasks: 526 total,   1 running, 524 sleeping,   0 stopped,   1 zombie
Cpu(s): 42.9%us, 21.4%sy,  0.0%ni, 33.7%id,  1.9%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8146632k total,  7427632k used,   719000k free,   131020k buffers
Swap:  2104504k total,      132k used,  2104372k free,  4506644k cached

    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
 318421 mysql     15   0 1315m 754m 4964 S 474.9  9.5  95300:17 mysqld          
   6928 root      10  -5     0    0    0 S  2.0  0.0  90:42.85 kondemand/3      
 476047 headus    17   0  172m  19m  10m S  1.7  0.2   0:00.05 php              
 476055 headus    18   0  172m  18m 9.9m S  1.7  0.2   0:00.05 php              
 476056 headus    15   0  172m  19m  10m S  1.7  0.2   0:00.05 php              
 476061 headus    18   0  172m  19m  10m S  1.7  0.2   0:00.05 php              
   6930 root      10  -5     0    0    0 S  1.3  0.0 161:48.12 kondemand/5      
   6931 root      10  -5     0    0    0 S  1.3  0.0 193:11.74 kondemand/6      
 476049 headus    17   0  172m  19m  10m S  1.3  0.2   0:00.04 php              
 476050 headus    15   0  172m  18m 9.9m S  1.3  0.2   0:00.04 php              
 476057 headus    17   0  172m  18m 9.9m S  1.3  0.2   0:00.04 php              
   6926 root      10  -5     0    0    0 S  1.0  0.0  90:13.88 kondemand/1      
   6932 root      10  -5     0    0    0 S  1.0  0.0 247:47.50 kondemand/7      
 476064 worldof   18   0  172m  19m  10m S  1.0  0.2   0:00.03 php              
   6927 root      10  -5     0    0    0 S  0.7  0.0  93:52.80 kondemand/2      
   6929 root      10  -5     0    0    0 S  0.3  0.0 161:54.38 kondemand/4      
   8459 root      15   0  103m 5576 1268 S  0.3  0.1  54:45.39 lvest

Solution

The problem is obviously the web app you're running. From the top output it seems that you're running some PHP code. You need to figure out which part of the PHP code causes the problem (either directly or via database access).

If the example top output describes the usual situation, I'd guess that some part of your processes block each other on the application level (some kind of lock contention).

I infer this from following facts: low IO wait time (the wa data in top output), 33% system idle and high load. This means that you're not running all the CPUs and you're not waiting for the IO. In this case, the only way to make system "too slow" is to make processes serial (one process waiting on CPU#2 until another process is complete on CPU#1). This happens only if there's some more or less explicit locking between different processes. If you truly cannot remove the inter-process locking, then the only option is to invest on faster CPU cores instead of multiple slow cores. Your CPU is already pretty close top-of-the-line, so I'd consider investigating the code first.

You asked if it was possible to use some RAM to reduce CPU usage. It might be possible with aggressive caching but only the application code (PHP) you're running can make use of this tradeoff. Again, you need to profile and modify the PHP code. There's no magical switch to say use more memory, less CPU, please.

Comments

[Chriswede]

top - 06:09:14 up 29 days, 20:36, 1 user, load average: 19.03, 14.54, 13.19 Tasks: 576 total, 1 running, 574 sleeping, 0 stopped, 1 zombie Cpu(s): 32.0%us, 6.1%sy, 0.0%ni, 59.1%id, 2.8%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 8146632k total, 7775292k used, 371340k free, 129836k buffers Swap: 2104504k total, 132k used, 2104372k free, 4566388k cached

[Chriswede]

you can see wa is totally ok

[Chriswede]

please see updated question

[Mikko Rantalainen]

Looking at your top ouput suggests that mysqld is the bottleneck because it's using close to 5 cores in the same time all your php processses use only a little CPU time. With very high probability your PHP code executes some SQL queries that cause locking between MySQL processes. MySQL is known to be poor performer for concurrent write access from multiple clients so try to avoid any writes to the database if possible at all. If your PHP code supports other backends, you should try using some recent PostgreSQL version instead because it probably performs better with concurrent writes.