how to fix apache2 "proxy_http:error" AH01102: error reading status line from remote server localhost:4382

Question

how to fix apache2 "proxy_http:error" AH01102: error reading status line from remote server localhost:4382.

I have an apache2 webserver which is being used to serve few nodejs applications in the backend.

Apache2 webserver version: Server version: Apache/2.4.41 (Ubuntu) Server built: 2019-08-21T20:43:05

Nodejs version: v10.18.1

Mongo db version: v4.0.15

We have been plague by this below error/errors(see logs) on all our backend applications "proxy_http:error" AH01102: error reading status line from remote server localhost:4382.

[Mon Apr 27 20:09:05.697271 2020] [proxy_http:error] [pid 26792:tid 140063099688704] (70007)The timeout specified has expired: [client 178.153.198.97:52385] AH01095: prefetch request body failed to 127.0.0.1:4381 (localhost) from 178.153.198.97 (), referer: https://www.example.com/sub-admin/menus/add-menu
[Mon Apr 27 20:21:51.617095 2020] [proxy_http:error] [pid 26764:tid 140062901626624] (104)Connection reset by peer: [client 178.153.195.206:61268] AH01102: error reading status line from remote server localhost:4382, referer: http://www.example.com/restaurant/dashboard
[Mon Apr 27 20:21:51.617117 2020] [proxy:error] [pid 26764:tid 140062901626624] [client 178.153.195.206:61268] AH00898: Error reading from remote server returned by /restaurant/assets/img/avatars/5.jpg, referer: http://www.example.com/restaurant/dashboard
[Tue Apr 28 03:51:28.498423 2020] [proxy_http:error] [pid 26793:tid 140062868055808] (104)Connection reset by peer: [client 89.211.117.185:57622] AH01102: error reading status line from remote server localhost:4382, referer: http://www.example.com/restaurant/dashboard
[Tue Apr 28 03:51:28.498455 2020] [proxy:error] [pid 26793:tid 140062868055808] [client 89.211.117.185:57622] AH00898: Error reading from remote server returned by /restaurant/static/css/main.f8d32764.chunk.css, referer: http://www.example.com/restaurant/dashboard

So far i have tried these things to fix this error.

1. ProxyPreserveHost On - It didn't workout.

2. disablereuse=on. It didn't work.

3. Timeout=600, then Timeout=900. Didn't work.

4. retry=1 acquire=3000 timeout=600 Keepalive=On. Didn't work.

5. SetEnv force-proxy-request-1.0 1
   SetEnv proxy-nokeepalive 1

Worked for small duration(12 hours). Again same error got reproduce with pretty much same frequency as earlier.

6. SetEnv proxy-initial-not-pooled 1
   SetEnv proxy-nokeepalive 1

This one worked for 24 hours. Then after 24 hours, this error got reproduce. Though this time after 24 hours, error is occurring only once in an hour, sometimes only once in two hours, sometime once in two-five hours.

1Q. First of all why this error happening.

2Q. Is this error occurring only because of apache or nodejs application can also be culprit
(But there's hardly any error logs of the application, when this apache error occurs).

3Q. How/What is the best way to identify, diagnosis and resolve this error, coz after searching on various forums, no one seems to have resolve this error it perfectly or have a right answer for it and this has been the case since 2006, looking at some of the forums.

Here is my apache config file

<VirtualHost *:443>
  ServerAdmin [email protected]
  ServerName  www.example.com
  ServerAlias example.com

  SSLEngine On
  SSLProxyEngine On
  SSLCertificateFile "/etc/ssl/private/server.crt"
  SSLCertificateKeyFile "/etc/ssl/private/server.key"


  ProxyRequests Off

    #Admin
    ProxyPass /admin http://localhost:4380/
    ProxyPassReverse /admin http://localhost:4380/
    SetEnv proxy-nokeepalive 1
    SetEnv proxy-initial-not-pooled 1


    ProxyPass /restaurant http://localhost:4382/
    ProxyPassReverse /restaurant http://localhost:4382/
    SetEnv proxy-nokeepalive 1
    SetEnv proxy-initial-not-pooled 1


    ProxyPass /sub-admin http://localhost:4385/
    ProxyPassReverse /sub-admin http://localhost:4385/
    SetEnv proxy-nokeepalive 1
    SetEnv proxy-initial-not-pooled 1


    #API
    ProxyPass /admin-api/ http://localhost:4381/
    ProxyPassReverse /admin-api/ http://localhost:4381/
    SetEnv proxy-nokeepalive 1
    SetEnv proxy-initial-not-pooled 1


    ProxyPass /restaurant-api/ http://localhost:4379/
    ProxyPassReverse /restaurant-api/ http://localhost:4379/
    SetEnv proxy-nokeepalive 1
    SetEnv proxy-initial-not-pooled 1


</VirtualHost>

Solution

If you look at your log, first appear (70007)The timeout specified has expired: [client 178.153.198.97:52385] AH01095 the rest of the log is a kind of consequence of that.

What is happening here is that your localhost implementation is taking much time to return an answer. Maybe you can verify at your implementation why is that and try to optimize it. But in the apache you can define which will be the connection time and timeout for wait to localhost implementation

ProxyPass /restaurant http://localhost:4382/ connectiontimeout=[time in seconds] timeout=[time in seconds[

for example 60 seconds (1 min)

ProxyPass /restaurant http://localhost:4382/ connectiontimeout=60 timeout=60