Apache multipart POST "pass request body failed"

Question

We are having problems with our web server (which is configured ssl -> apache -> jetty) randomly rejecting multipart upload POST requests with a 400 Bad Request error code. The apache error log (on info level) shows the following two errors:

[info] [client x1.y1.z1.w1] (70007)The timeout specified has expired: SSL input filter read failed.
[error] proxy: pass request body failed to x.y.z.w:8087 from x1.y1.z1.w1
[info] [client x1.y1.z1.w1] Connection closed to child 74 with standard shutdown

or

[info] [client x2.y2.z2.w2] (70014)End of file found: SSL input filter read failed.
[error] proxy: pass request body failed to x.y.z.w:8087 from x2.y2.z2.w2
[info] [client x2.y2.z2.w2] Connection closed to child 209 with standard shutdown

both cases result from the client side in a 400 Bad Request. Sometimes our jetty server doesn't even see the request meaning that it gets rejected on apaches side, sometimes it starts processing it only to be rejected (this manifests itself as a MultipartException in our UploadFilter)

We have mod_proxy setup to use a fallback load balancing scheme but the logs show that a fallback has not yet been triggered causing me to believe this is not the cause of the problem.

I tried setting SetEnv proxy-sendcl 1 but that didn't change anything.

The upload requests are arount 1mb. Only these multipart file POST requests fail, we have multiple GET requests comming in at the same time and they always work as expected.

If anyone can share any advice or suggestions I would be very grateful! Thank you

Solution 1

If you are using some ajp-enabled backend server, like Tomcat, you may try using mod_proxy_ajp instead of mod_proxy_http. I had a similar problem on a heavy upload app and I fixed it by changing

ProxyPass /myapp http://localhost:8080/myapp
ProxyPassReverse /myapp http://localhost:8080/myapp

by

ProxyPass /myapp ajp://localhost:8009/myapp
ProxyPassReverse /myapp ajp://localhost:8009/myapp

It's also required to enable the ajp connector on tomcat side, of course.

Hope it helps!

Solution 2

Please check this one: https://issues.apache.org/bugzilla/show_bug.cgi?id=44592

The problem could be caused by the HTTP keepalive (KeepAliveTimeout directive) that is killing an established connection with a slow client (tcp latency, slow request body creation, etc).

Try to raise the KeepAliveTimeout in your apache conf, but don't keep it too high to avoid killin' your server (DOS).

Solution 3

You may be seeing this due to timeouts resulting from Apache trying to buffer the entire POST body before passing it through.

Enabling proxy-sendcl may exacerbate this, since this can force Apache to spool a large POST to disk just to calculate the Content-Length when "the original body was sent with chunked encoding (and is large)".

To avoid this, set the environment variable proxy-sendchunked.

Comments

[B Medeiros]

It's my other answer to this question

[Jitendra Chandani]

That works fine. Thanks. But I am not sure why http proxy not working on one particular server. Httpd configuration are same.