Basic Apache httpd server setup becomes slow to serve images requiring frequent restart

Question

I have an Apache httpd server running on RHEL 5.3. I'm using this server to serve static content (images) to for a website running on another httpd instance. In the public web directory are several images, and that is all. Some time after startup, the server becomes very slow to serve images to the browser, and can often hang for several minutes whilst trying to serve a 16x16 png.

What is the best way to go about debugging this problem?

I've run top on the box to verify that there is enough free memory, and CPU available. The machine isn't going into swap, and the httpd process only takes up CPU and memory for a couple of seconds when the image is requested through the browser. However the image is only returned often after several minutes.

A restart of the httpd process resolves the issue, and the images load quickly.

This seems like the simplest setup of a web server possible. What could the issue be?

Thanks

UPDATE: httpd.conf settings:

Timeout 10
KeepAlive On
MaxKeepAliveRequests 150
KeepAliveTimeout 15
ServerLimit 3

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

Solution 1

Use mod_status to check and monitor server status. I guess most of the connections are waiting in status KeepAlive. You can try to turn it off or down to a low value like KeepAliveTimeout 4.

Solution 2

Diagnostic suggestions:

1. You've checked CPU and memory, but it would be a good idea to watch disk bandwidth and other statistics also (I suggest dstat -af). You're looking for things like a great deal of disk reads or writes, lots of context switches or interrupts, other measurements that vary widely between the start of the server and the server's behavior when it is in trouble.

2. Another thing to check is to figure out if the behavior is a function of time passing, or the number of requests, or both?

   • Start the server, don't request any images for a few hours, and then request an image, is the request fast, or ?
   • Start the server, request images as fast as possible (use loopback to avoid network latency) and see how long it takes to reach a certain pre-determined level of slowness (e.g. a 10 second response)

3. Test to see if it is a networking issue. When the server is having trouble (e.g. 10 second responses), request an image via the loopback interface. If it returns fast, look at the network. If slow, you can probably ignore networking issues outside your host.

Solution 3

A couple questions:

Have you done any tuning to the Apache.conf file? Which Apache MPM are you using?
What parameters are currently set in the Apache.conf file for:

Timeout
KeepAlive
MaxKeepAliveRequests
KeepAliveTimeout
MinSpareServers
MaxSpareServers
StartServers
MaxClients
MaxRequestsPerChild

Comments

[LazyOne]

Use nginx -- it was specifically designed to serve static content .. and serve fast.

[Alex]

Thanks. Unfortunately I'm restricted in terms of the tools I can use (corp environment), nginx isn't an option for me.

[Toby Mao]

I second what @LazyOne says, you should really be using NGINX/Lighttpd etc. as they were built to do just this. If all you need it to do is serve static images I see no reason why being in a corporate environment stops you. You're already using a dedicated apache instance, so just replace it with an NGINX instance. If you give one server two IPs you can even run both side by side

[Alex]

Re corp env: there are software restrictions meaning I cannot just download and use anything - all software has to be approved for use in the company. This often takes weeks/months of admin. Not ideal for development, but that's the situation I'm in.

[Stefan Lasiewski]

You should strongly consider an update to this RHEL box. RHEL 5.3 has reached it's End-of-Life, and is no longer supported. The latest version of RHEL has many bug, security and performance fixes to Apache. If this server is only running standard Apache and serving static content, then the update should be very straightforward and bug free. The whole update process can take less then 1 hour, or as little as 20 minutes if you download the files beforehand.

[Martin Tournoij]

^ What he said. Plus you may want to consider looking at something like Varnish. Setting up Varnish to cache images is dead easy and can make a (very) large difference.

[George P]

@Carpetsmoker Yeah for sure. I used to be all crazy about building lean static servers, since then I've caved and just CNAME a subdomain to Amazon S3

[Alex]

Thanks - have added conf settings in original question above. Unfortunately I'm restricted in terms of the tools I can use (corp environment), so Varnish, or nginx aren't an option for me.