Apache CentOS Server very slow! Wrong configuration?

Question

I have a problem with server overload on a Apache configuration 8gb of RAM 1and1.com Virtual Server. Server gets extremely slow once it reaches 100+ active users (according to google analytics) during pick time. I've done siege & loadimpact tests and made some configuration changes to the httpd.conf file but I don't know if it's enough. Maybe I just need more ram like 16GB or am I missing something? I know I can optimize programming but I don't see how it will improve performance drastically. Interesting fact is that parallels memory usage only displays 10-20% during this overload, but the website is so slow! Help!

Avg. site file size is 1.5MB (We have lots of pictures)
Avg. load time: 1.8s (during normal days) - 1,600 page views, 270 highest sessions p/h

This are the analytics stats of a very slow server day:

Highest sessions per hour: 420 - 700
Total sessions: 4,300 - 37,000
Page views: 25,300 - 361,500
Avg. Session duration: 9min

Server data provided by 1and1:

Webspace: 200GB
RAM: 8GB
Unlimited Traffic: yes
Operating System: CentOS 6 minimal system (64-bit)
cpu cores: 8
Processors: 4   
Processor: AMD Opteron(tm) Processor 6378 

Http.conf

Timeout 75
KeepAlive On
MaxKeepAliveRequests 500
KeepAliveTimeout 3

<IfModule prefork.c>
 StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      220
MaxClients       220     
MaxRequestsPerChild  5000
</IfModule>

my.conf

max_allowed_packet             = 1G
max_connections                = 500                              
max_user_connections           = 500                                 
thread_cache_size              = 500                                 
query_cache_type               = 1                                   
query_cache_size               = 128M                                 
sort_buffer_size               = 2M                                 
tmp_table_size                 = 32M                                
read_buffer_size               = 128k                               
read_rnd_buffer_size           = 256k                              
join_buffer_size               = 128k                                
table_definition_cache         = 400                                
table_open_cache               = 400                                 
key_buffer_size                = 500M                                 
max_heap_table_size            = 64M                              
innodb_buffer_pool_size        = 500M                                
innodb_additional_mem_pool_size = 20M
innodb_file_per_table          = 1                                  
innodb_flush_log_at_trx_commit = 2                                  
innodb_log_buffer_size         = 12M                                                           
innodb_log_file_size           = 256M 

Am I doing something wrong? I appreciate all the help!

Solution

The first thing you need to look at is to determine where your bottleneck is.

• Is the system's CPU pegged, or is there CPU available but the web application's processes aren't using it effectively (ie, not enough worker threads)?
• Is the system out of RAM, or doing a lot of swapping?
• Are you bottlenecking on your database? Do you have slow queries, need indexes, or are you running into locking problems?
• Watch the time-to-first-byte for an application request, and compare it to that of a request for a static resource from the web server that doesn't touch your application code - that'll show you how quickly your app is processing the request.

Generally speaking, these kinds of performance problems are going to be in the application as opposed to the web server configuration.

Once you understand your bottleneck, you may be able to alleviate it with the help of your web server software (exact methods depend on what application code you're running and how you're attaching it to your web server, but there's a lot of info out there on getting the most performance out of FastCGI or mod_passenger), but there's only going to be so much that you can do there.

In summary: Don't start with the web server, start with tracking down your bottleneck and fixing it - if you find that there's anything you can do in your web server to help, then try that, but it's not likely to be able to fix a performance problem with your application.