Apache tuning recommendations for Windows Server 2008

Question

We have implemented Apache 2.4 64bit on Windows Server 2008 R2 as a reverse proxy to take advantage of a wildcard cert for our domain. We have been disappointed in the performance with even a light load so I have probably misconfigured something. The web sites served by the proxy (Glassfish 3.0.1) become increasingly slow to the point of being unresponsive. Restarting Apache temporarily fixes the problem but it quickly becomes unresponsive. Bypassing the proxy by going to the IP address of the Glassfish server directly we can see that GF isn't the issue. The only thing we are doing is the reverse proxy of the SSL frontend on Apache and passing traffic on the internal network with http to the Glassfish server.

When the websites are unresponsive, a look at the Windows Server processes indicates VERY little activity. CPU might be running at 10% (single CPU). There are less than 30 TCP connections. Memory utilization is less that a GB (it is a 4GB machine). Network traffic barely breaks the 100kbps threshold. All indications are that the server is barely breaking a sweat, yet the site is unresponsive.

I have set the following properties:

SSLSessionCache  "shmcb:C:/Apache_direct/logs/ssl_scache(1024000)"
SSLSessionCacheTimeout  300

ThreadsPerChild 3000
MaxRequestsPerChild 0 
KeepAlive On 
KeepAliveTimeout 15 

ProxyRequests Off
SSLStrictSNIVHostCheck off
ProxyBadHeader Ignore

I have virtual host settings for each http and https port for each server. VHost settings are similar to:

<VirtualHost *:443>
ServerName test.domain.org
#
ProxyPreserveHost On
SSLEngine on
ProxyPass / http://192.168.80.196:8080/
ProxyPassReverse / https://192.168.80.196:8080/

</VirtualHost>

Have I missed an important setting here or misconfigured a setting? Based on what I have read, Apache should be able to easily handle the traffic we have which is as follows:

800 visits an hour.  13k visits daily.
100 uploads an hour of roughly 60mb MP3 files.
400 downloads an hour of those same MP3 files.
800 downloads of small (100k) pdf files.

Glassfish is serving all the file uploads/downloads and we allow those sessions to persist up to an hour. Since most of the files are unique, we are a little uncertain of what cache settings we should use (if any). Perhaps I should omit the SSL cache settings?

There are many, many hits on Apache performance tuning on the web but few that deal specifically with reverse proxy, SSL and give recommendations based on type/amount of traffic. Some suggestions would be very helpful.

Solution

I am going to reply to my own query so that if someone comes across this they will at least see some kind of answers….

I never got confirmation that this particular configuration was correct, incorrect or could be improved. I do know that we decided to go with it one more time with the following modifications (pure guesswork on my part):

Threadlimit 3000
ThreadsPerChild 3000
MaxRequestsPerChild 0 
KeepAlive On 
KeepAliveTimeout 15 

AcceptFilter https none
AcceptFilter http none
EnableSendfile Off
EnableMMAP off

ProxyRequests Off
SSLStrictSNIVHostCheck off
ProxyBadHeader Ignore
HostnameLookups off
ExtendedStatus off

We experienced no performance problems with this config so we are running in production with it. The main question, which was how to have good performance in a reverse proxy environment has been answered by trial and error.

Comments

[voretaq7]

What have you tried? Where has your research lead you? As it stands this question is overly broad: you need to do a little legwork on your own in order for us to be able to help you effectively. See here for some tips on what we're looking for in a question.

[Pete Helgren]

Well, I have added the directives above based on research over a couple of weeks. I have done quite a bit of 'legwork', mostly reading articles that give vague, overly broad, not specific information when it comes to number of users, types of information served and traffic for reverse proxies. Those unsatisfactory answers lead to this vague, over broad question. I implemented what I thought were the best performance entries in the above directives (This answers your "What have I tried, where has my research led me" question.) I did try Apache 2.2 32bit with the same results.

[Pete Helgren]

I continue to find other snippets of suggestions (sorting out the Linux tweaks vs Windows is a hassle) Current directives are: Threadlimit 3000 ThreadsPerChild 3000 MaxRequestsPerChild 0 KeepAlive On KeepAliveTimeout 15 AcceptFilter https none AcceptFilter http none EnableSendfile Off EnableMMAP off ProxyRequests Off SSLStrictSNIVHostCheck off ProxyBadHeader Ignore HostnameLookups off ExtendedStatus off Seems to be some discussion relating to the USER heap as well as concurrent connections in Windows as relates to performance but I haven't found anything definite. Still need help...

[Philip]

You'll not find much information on running Apache on Windows as it's not really recommended for production environments. Apache will run on Windows, but was really meant to run in a *nix environment.

[Abbas Akhundov]

Can you provide a source for "apache on windows is not recommended for production environments? I found this in the apache docs for 2.0 on windows 95, but nothing newer. Also further reading here: apachelounge.com/viewtopic.php?p=22542

[Abbas Akhundov]

I use very similar settings as well with good performance.