Proxy Error 502 "Reason: Error reading from remote server" with Apache 2.2.3 (Debian) mod_proxy and Jetty 6.1.18

Question

Apache is receiving requests at port :80 and proxying them to Jetty at port :8080

The proxy server received an invalid response from an upstream server
The proxy server could not handle the request GET /.

My dilemma: Everything works fine normally (fast requests, few seconds or few tens of seconds long requests are processed ok). Problems occur when request processing takes long (few minutes?).

If I issue request instead directly to Jetty at port :8080 the request is processed OK. So problem is likely to sit somewhere between Apache and Jetty where I am using mod_proxy. How to solve this?

I have already tried some "tricks" related to KeepAlive settings, without luck. Here is my current configuration, any suggestions?

#keepalive Off                     ## I have tried this, does not help
#SetEnv force-proxy-request-1.0 1  ## I have tried this, does not help
#SetEnv proxy-nokeepalive 1        ## I have tried this, does not help
#SetEnv proxy-initial-not-pooled 1 ## I have tried this, does not help
KeepAlive 20                       ## I have tried this, does not help
KeepAliveTimeout 600               ## I have tried this, does not help
ProxyTimeout 600                   ## I have tried this, does not help

NameVirtualHost *:80
<VirtualHost _default_:80>
    ServerAdmin [email protected]

    ServerName www.mydomain.fi

    ServerAlias mydomain.fi mydomain.com mydomain www.mydomain.com

    ProxyRequests On
    ProxyVia On
    <Proxy *>
            Order deny,allow
            Allow from all
    </Proxy>

    ProxyRequests Off
    ProxyPass / http://www.mydomain.fi:8080/ retry=1 acquire=3000 timeout=600
    ProxyPassReverse / http://www.mydomain.fi:8080/

    RewriteEngine On
    RewriteCond %{SERVER_NAME} !^www\.mydomain\.fi
    RewriteRule /(.*) http://www.mydomain.fi/$1 [redirect=301L]

    ErrorLog /var/log/apache2/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn

    CustomLog /var/log/apache2/access.log combined
    ServerSignature On

</VirtualHost>

Here is also the debug log from a failing request:

74.125.43.99 - - [29/Sep/2010:20:15:40 +0300] "GET /?wicket:bookmarkablePage=newWindow:com.mydomain.view.application.reports.SaveReportPage HTTP/1.1" 502 355 "https://www.mydomain.fi/?wicket:interface=:0:2:::" "Mozilla/5.0 (Windows; U; Windows NT 6.1; fi; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10"
[Wed Sep 29 20:20:40 2010] [error] [client 74.125.43.99] proxy: error reading status line from remote server www.mydomain.fi, referer: https://www.mydomain.fi/?wicket:interface=:0:2:::
[Wed Sep 29 20:20:40 2010] [error] [client 74.125.43.99] proxy: Error reading from remote server returned by /, referer: https://www.mydomain.fi/?wicket:interface=:0:2:::

Solution 1

I have solved the problem. The Keepalive=On should be inserted into ProxyPass config line:

ProxyPass / http://www.dom.fi:8080/ retry=1 acquire=3000 timeout=600 Keepalive=On

See that

Keepalive=On

there? It is critical ;)

Solution 2

This error can also occur if you don't end your proxy url with a /. Either both paths should end with a / or neither.

Solution 3

Have you tried setting setenv proxy-initial-not-pooled 1?

Reference here

Solution 4

Looking at the log, there's something that times out at 5 minutes (=300 seconds). That's a pretty long time to wait for a response. When you access the Jetty server directly, does this resource really take that long to produce a response?

If the five minutes really is within possible response times, you might try tweaking the ProxyTimeout configuration directive.

Depending on your network set-up, it might well be that there's no reason to even try to use any keepalive system (is there a firewall between the app server and proxy which might be configured to drop sessions that are idle for too long?), but the ProxyTimeout would affect the behaviour of the proxy itself.

If the same proxy also serves other backends, it would be better to keep the current ProxyTimeout, and configure the timeout in the ProxyPass directive (see mod_proxy documentation).

If, however, the responses without the proxy are consistently something much less than the five minutes see here as the cut-off limit, then there might really be some odd interference between the proxy and app server, but you're not providing anything of value for identifying what it might be.

View more solutions

Comments

[Martin]

Hi.. I am still stuck with this one. All above settings tried and also increasing jetty maxIdleTime did not help. Any pointers what to try next?

[Admin]

No, this didn't help. Then it's not about race condition, it's the long delay and something happens in between (some kind of misunderstanding between Jetty and mod_proxy).

[ravenmeister]

"When you access the Jetty server directly, does this resource really take that long to produce a response?" -- YES. I also tried setting this ProxyTimeout to 600. Does not help. There is no firewall between proxy and jetty. Timeout is configured also in ProxyPass.

[ravenmeister]

"you're not providing anything of value for identifying what it might be": I don't know what that could be. All i am getting is an error message from the server: Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET /. Reason: Error reading from remote server

[ravenmeister]

As for increasing the proxy timeout, did this also change the time your browser did spin before getting the 502 error?

[ravenmeister]

Then to ways how you could find out what is happening in the application server: you could take a couple of thread dumps, and look from them what is executing when the browser is waiting. Alternatively, add enough debug logging statements to be able to trace your code to pinpoint the cause of slowness.

[ravenmeister]

I know why it is slow. I don't know why apache proxy refuses the response when it is finally ready.

[Deb]

I believe you can mark your own answers as Accepted. It marks the question as resolved in the system for other people to find.

[Arnold Roa]

each library bring a different feature, not necesarily related to proxy

[AlxVallejo]

Where exactly do you put this?

[Steven]

@AlxVallejo You should find your config files here /etc/apache2/sites-enabled/[sitename].conf

[dokaspar]

We have exactly the same proxy errors. They happen very rarely (one in a thousand requests). Why exactly is that Keepalive=On critical?

[MattBianco]

Could it not be the timeout=600 or retry=1 that fixes this instead? (or the combo)

[HomeIsWhereThePcIs]

@dokaspar I know it's been a long time and this is a long shot but have you ever figured out what the problem was? I am also getting 502 very rarely (about 3 in 2000 requests). Did the KeepAlive=on option fix this for you?

[vijay]

Thanks, this solved my issue. Just a simple / at the end. :)

[Jamie Hutber]

setenv: command not found