Apache has a long lag before responding

Question

I have an Apache server on CentOS that occasionally takes 1-5 seconds to serve a file, including a text file that is 2 characters. I have a JMeter test that gets a few different types of files, and the delay often happens on whatever file it grabs first, but not always. This is true whether there are several request in parallel or sequentially. The pages it actually needs to serve are PHP/Oracle, but I'm assuming the problem isn't in the code because PHP isn't parsing the text file (and I did confirm that).

• HostnameLookups is off. I grepped /etc/httpd to confirm I rooted them all out.
• People inside the network and outside the network experience the lag.
• If I run wget from within the server itself is also subject to the lag, which leads me to think it's not a DNS issue.
• When I see it with Apache Bench, it shows a lag both during connect and processing.
• I've never seen the lag on a ping.
• We temporarily added an exception to the firewall, but restored it when it made no difference.
• I set MaxClients according the the formula given [here] (https://servercheck.in/blog/3-small-tweaks-make-apache-fly)
• The server supports SSL, but the tests I'm running generally test over HTTP. Every once in a while I run tests over HTTPS, and its speed is generally representative of its less secure counterpart.
• I am using AllowOverride to parse .htaccess files, but when I disabled it and tested, I still saw the same lag.
• There are very few no allow from all/deny from all settings lurking in httpd.conf
• I'm still looking into Apache MPM and valgrind.

Some relevant Apache settings:

StartServers            8
MinSpareServers         5
MaxSpareServers        20
ServerLimit           256
MaxClients            136
MaxRequestsPerChild  4000

Some of the resources I've consulted:

• Apache / PHP server delaying for a minute before responding
• Centos/Windows apache network lag
• Random Lag in Apache before serving files
• Slow http responses
• Why is the response on localhost so slow?
• PHP + Apache long wait time

Any help would be much appreciated. Thank you.

Solution

A colleague pointed me at this page which recommended adding the following to /etc/resolv.conf:

options single-request-reopen

I'm not sure if that's what fixed it, or if it was some combination of several things I tried, but other people accessing the resource indicated a significant speed up after I added this. I hope it helps someone else.

Comments

[SteAp]

Does this happen if you serve an empty PHP script? What do you mean be 'including a text file'. Does it include() a file as source?

[kjc26ster]

Thanks for replying so fast! I put a text file in the htdocs and had Apache serve it to me by putting the URL in my browser. There's no PHP involved.

[Twitch]

My hunch would be to look at the routing. Does tcptraceroute to the webserver reveal anything?

[kjc26ster]

Good thought. tcptraceoute said: Selected device p4p1, address 10.100.4.43, port 38802 for outgoing packets Tracing the path to [server] (10.200.1.55) on TCP port 80 (http), 30 hops max 1 10.100.4.1 0.380 ms 0.338 ms 0.313 ms 2 172.16.25.1 24.922 ms 38.065 ms 25.000 ms 3 10.200.1.55 [open] 9.024 ms 3.999 ms 13.089 ms

[Meroje]

There may be some merit to expressly disabling IPv6 as well. Here's some notes for doing that: thegeekdiary.com/centos-rhel-7-how-to-disable-ipv6