How to debug/log 502 Bad Gateway from reverse proxy?

Question

We're running a web service and hitting 502 Bad Gateway randomly. The 502 is reported by reverse proxy (Apache 2.4.x) while talking to backend server (Apache 2.4.x). The problem randomly re-appears around once per 100,000 served requests so it's really hard to reproduce. I see that Apache allows configuring ProxyBadHeader directive where possible options are IsError|Ignore|StartBody. I have problems finding the exact cause of the error. I know that if I set ProxyBadHeader to StartBody the partial output would end up to the browser. I really would not want to emit the data to unknown end user so I need something better. Is it possible to log all the partially received data from the bad gateway?

The problem seems similar to Apache bug 37770 but it should be fixed in the version we're running.

Solution

In my experience, you are correct to point to Apache bug 37770, which in the Apache's reverse proxy functionality. If you examine the patches cited there from 2008 until 2016, you'll see yourself that these are really workarounds. If the root cause disappeared in httpd-2.4, then this is not apparent neither from the conversation there or from the observed httpd behavior.

Either

SetEnv proxy-initial-not-pooled 1     # prevent a small percentage of 502 errors, comes with a performance penalty

or accept that a little percentage of requests get 502'd. Or migrate to haproxy.

I guess that if you did gather the backend's conversation, you would see that backend doesn't see the HTTP GET. This was result of my little investigation and I concluded that the connection gets somehow lost inside reverse proxy code (perhaps where backend connection is taken from pool or returned there).

Comments

[Fredi]

Usually this is due to the backend server taking too much time to return the reply. You can add the duration on your log files at both servers (%D) and check it

[Mikko Rantalainen]

We've since migrated to haproxy and all the random problems have disappeared. The haproxy configuration syntax is weird and I still don't get why they spell let as acl in the config but otherwise, haproxy is a really nice piece of software. Avoid versions 2.2.1 ... 2.2.4 because those suffer from bug that causes all uploads to randomly fail. See github.com/haproxy/haproxy/issues/790 for details.

[Naveed Abbas]

/@MikkoRantalainen Good to see the final picture. I'm impressed to see you respond after 4 years; if more people followed up on such bugs to completion we would collectively have so much more nice things!

[Mikko Rantalainen]

We tried using Citrix Netscaler as reverse proxy after Apache but that seemed buggy, too. The bugs in Citrix Netscaler are different from Apache reverse proxy bugs but some random bugs occur about at the same rate. Citrix Netscaler also had major problems with HTTP/2 so we had to use HTTP/1.1 only. This far haproxy has worked perfectly except the versions with upload bug I referenced above.