Apache Restart cause [proxy_http:error] [pid 120502] (70008)Partial results are valid but processing is incomplete

Question

I'm facing issue when ever we restart/reload apache process on Redhat 7.3 OS. After restart we get below error in apache log.

[Thu Aug 24 00:00:36.869582 2017] [proxy_http:error] [pid 120502] (70008)Partial results are valid but processing is incomplete: [client 139.59.39.38:56682] AH01110: error reading response. 

During that, we can see below message on the browser while if we browse URL during this issue.

ERR_CONTENT_LENGTH_MISMATCH

To fix the issue we have to restart Apache multiple time which fixes after 2 - 3 tries. The number of tries varies.

We are using Apache to only proxy request to other internal URL with one special Apache module installed. Module is web agent by OpenAM (Forge Rock OpenAm).

We are using latest OpenDJ and OpenAM for authentication and webagent for OpenAM is installed in Apache which proxy request to our application.

Every time we restart apache we start facing issue ERR_CONTENT_LENGTH_MISMATCH.

I want to know how we can debug the issue why during reload or restart apache fails.

httpd -v
Server version: Apache/2.4.26 (Red Hat)
Server built:   Jun 20 2017 15:37:14

OS : cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.3 (Maipo)

Please help me debug the issue and find a possible solution. Let me know if you need more information to debug the issue.

Solution

OK I've tracked this down

https://bugster.forgerock.org/jira/browse/AMAGENTS-349 is the upstream bug caused by this change in Apache HTTPD commit 1485d64698ef816f1586e2d772c58cb9b78c788d in github (Merge r1753257 from trunk) I can't post more than 2 links at the moment

Forgerock hasn't released WebAgents 4.1.2 yet or the source code for the fix

So you can either go back to Apache 2.4.23 or use my fix here:

https://github.com/FireBurn/web-agents-public/commit/67b27d8a77ce28ad5bee95842136050bb52577ae

The above patch seems to fix the issue for me, but I give no guarantees that its correct or wont cause other issues

Do let me know if you have any issues compiling it (I use a script here)

Comments

[Mike Lothian]

I've raised bz.apache.org/bugzilla/show_bug.cgi?id=61501 for this issue

[Mike Lothian]

and the forgerock forums forum.forgerock.com/topic/…

[Arun Singh]

Hi @mike did you got anything to fix? Because at our end we are not able to fix. I install Apache from yum repo. Due this unable to go production.Yes we are using the binary from forge rock.

[Mike Lothian]

I've updated my answer above with the real cause and some solutions

[Mike Lothian]

Here's my build script it assumes you have everything needed to build and compile code gist.github.com/FireBurn/00bff4931a53e4380575707ac0f20fc8

[Arun Singh]

Found one more bug report matching our issue on apache bug tracker. bz.apache.org/bugzilla/show_bug.cgi?id=60706 We can vote or link your bug and this bug? So that apache can fix in another release?

[Mike Lothian]

Based on that bug report being raised in February and the offending commit not being removed in 2 releases, I'm thinking they have no intention of changing the behaviour. Forgerock have worked around it

[Arun Singh]

That's right. Thank you for updating your answer. That cleared our long running issue.

[Mike Lothian]

Great (it worked for me too). Are you happy accepting it as the answer?