Apache2 reverse proxy seems to intermittently freeze?

apache-2.2 tomcat reverse-proxy web-applications

13,620

Solution 1

I would try to use ajp to connect to tomcat from apache.
Please test the configuration using mod_proxy_ajp with Tomcat.
From logs, it looks like the tomcat did not responded to apache's request. It refused the connection.

If you have high traffic, search deeper in your application's logs. It could be a deadlock or maxthreads reached.

HTH

Solution 2

Check that your apache file cache is disabled.

# LoadModule file_cache_module modules/mod_file_cache.so

If that doesn't solve the problem (shouldn't, this doesn't seem a cache problem) try debugging using mod_jk. Download appropiate mod_jk version for your Apache, setup it and set the log level to DEBUG

# Set the jk log level [debug/error/info] 
JkLogLevel          debug

Also, include tons of info in your apache logs

LogFormat "%h %l %u %t \" %m \"%V\" \"%r \" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D CustomLogFormat
. . .
CustomLog "|C:/Apache2/bin/rotatelogs.exe C:/Apache2/logs/access_%Y%m%d.log 86400" CustomLogFormat

You should now have an Apache log with microseconds taken to serve the page, and a debug-level log of your AJP 1.3 connection to tomcat (with mod_jk).

With that done, restart apache and try forcing that freezing error. When error happens, check time, and browse logs to see what happened in that instant, how much time did apache spend processing, and so.

It won't solve anything, but most possibly will give you some insight on what is happening.

13,620

tinny

Modern software development & Linux. Groovy & Grails, Rails, Django etc... Oh, and I love fishing

Updated on September 18, 2022

Comments

tinny over 1 year
Apache2 reverse proxy seems to intermittently freeze when proxying a tomcat web app.

The problem is that every so often (once or twice a day usually after inactivity) when I try to log into this proxied web application it just freezes. The application just freezes indefinitely, no 500 response code... the browser just waits and waits.

If I keep trying to hint the web address from my browser it will eventually just start working again.

Ive tried to reproduce this issue by navigating directly to the web app on tomcat port 8080, but ive never been able to reproduce this without going through the reverse proxy.

Here is my reverse proxy config... any fault finding ideas? thanks

/etc/apache2/sites-available/default
```
ProxyPass /manage/ http://localhost:8080/manage/
ProxyPassReverse /manage/ http://localhost:8080/manage/

ProxyPass /manage http://localhost:8080/manage
ProxyPassReverse /manage http://localhost:8080/manage
```
/etc/apache2/mods-available/proxy.conf
```
  <IfModule mod_proxy.c>
    #turning ProxyRequests on and allowing proxying from all may allow
        #spammers to use your proxy to send email.

        ProxyRequests Off

        <Proxy *>
                AddDefaultCharset off
                Order deny,allow
                Deny from all
                Allow from all
        </Proxy>

        # Enable/disable the handling of HTTP/1.1 "Via:" headers.
        # ("Full" adds the server version; "Block" removes all outgoing Via: headers)
        # Set to one of: Off | On | Full | Block

        ProxyVia On
    </IfModule>
```
More info: (not sure if relevant)
- Each web page is averaging 2 - 3 Ajax calls
- Haven't applied 14 Rules for Faster-Loading Web Sites. Looking to apply expires header for caching static content soon.
Enabled apache modules:

core_module (static) log_config_module (static) logio_module (static) mpm_worker_module (static) http_module (static) so_module (static) alias_module (shared) auth_basic_module (shared) authn_file_module (shared) authz_default_module (shared) authz_groupfile_module (shared) authz_host_module (shared) authz_user_module (shared) autoindex_module (shared) cgid_module (shared) deflate_module (shared) dir_module (shared) env_module (shared) mime_module (shared) negotiation_module (shared) proxy_module (shared) proxy_ajp_module (shared) proxy_connect_module (shared) proxy_http_module (shared) reqtimeout_module (shared) rewrite_module (shared) setenvif_module (shared) status_module (shared)

Update: I am now using the ajp protocol for the proxy. An have set some addition ProxyPass config e.g.
```
ProxyPass /manage ajp://localhost:8009/manage max=20 ttl=120 acquire=10000 retry=0
ProxyPassReverse /manage ajp://localhost:8009/manage
```
- Yury Kisliak about 13 years
  
  Any errors in error.log? Or in catalina.out.
- tinny about 13 years
  
  No errors in catalina.out but there is an interesting entry the appears every now and then in error.log [Sat May 07 07:27:59 2011] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed [Sat May 07 07:27:59 2011] [error] ap_proxy_connect_backend disabling worker for (localhost)
tinny about 13 years

Ive setup proxying using the ajp protocol. I did a bit of reading, sounds like a much better protocol for between apache and tomcat. Not sure if this will solve my problem. Time will tell. Thanks
tinny about 13 years

Nope, didn't solve the problem. Still get intermittent freeze when using reverse proxy. Connecting directly to Tomcat (port 8080) is working well
Yury Kisliak about 13 years

Any errors in apache's logs (error.log or vhost's error logs)?
Yury Kisliak about 13 years

How many httpd processes do you have running when is not working? Count them with: ps ax | grep httpd | wc -l
tinny about 13 years

maxClient in apache 150, maxThread in tomcat not set but defaults to 200 (I have now explicitly set it to 200 in my ajp connector). I will try retry=0
tinny about 13 years

No vhost error logging. I do have a few "Connection refused: proxy:" in error.log but I now think this was only from times when I restarted tomcat
tinny about 13 years

BTW I am the only user of this server right now. Very very low traffic.
Yury Kisliak about 13 years

OK, it happens :). It could be slow at startup.
TimS about 13 years

Is it always after a period of inactivity? If so then you may be running into a situation where you are exceeding the tcp connection timeout setting for your OS. You may want to try setting a ttl timeout on your ProxyPass statements. For instance ttl=120 will cause inactive connections to not be used again after being inactive for 120 seconds.
tinny about 13 years

Not sure about the period of inactivity anymore. Seems to sometimes happen after a short period of use. Although it does seem to become more stable during longer periods of use (if that makes sense???). Either way I think an explicitly defined ttl is a good idea. Thanks
tinny about 13 years

Ran "ps ax | grep httpd | wc -l" the returned result was 1
Yury Kisliak about 13 years

Replace httpd with your apache's process name. I guess is "apache".
tinny about 13 years

Ok, just ran "ps ax | grep apache | wc -l" result was 6. (While the application was frozen)
tinny about 13 years

No file cache module. Ive added a list of enable modules to my original post.
Yury Kisliak about 13 years

Please check apache's logs and tomcat's logs after you have modified to mod_proxy_ajp. If the application is running (tomcat running and the app is deployed) there can be only a networking problem. But, being on localhost makes this odd. Can we have details about the application is running inside tomcat?
tinny about 13 years

Its a web app built with the Grails framework. About 1 - 3 Ajax calls per page. Secured using Grails spring security plugin. Using JQuery. Im also wondering if it might be a memory issue (I should have mentioned this earlier). This is a 512MB ubuntu VM running Apache, Tomcat and postgresql. "free -m" show 10MB ram free and 1MB swap in use. I wonder if I am tipping over into swap memory every now and then? Tomcat is currently setup for 512MB heap which I should lower. Also I do have iptables enabled so this might effect localhost networking....?