Apache server reverse-proxying another apache server, getting "AH01102: error reading status line from remote server"

Question

I have two apache servers setup on two separate physical machines. My current setup is:

                      Apache 1 (Reverse Proxy) <===> Apache 2

Both apache server versions are Apache/2.4.29 (Ubuntu) running on Ubuntu 18.04.4 LTS and their /etc/apache2/apache.conf files are identical.

Apache 1 sites-enabled config:

<VirtualHost *:80>
        ServerName subdomain.domain.tld
        ServerAlias www.subdomain.domain.tld

        ServerAdmin [email protected]
        DocumentRoot /var/www/html

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        <Proxy *>
                Order deny,allow
                Allow from all
        </Proxy>

        ProxyRequests off
        ProxyPreserveHost On
        ProxyPass /maintenance_page !
        ProxyPass / http://[apache2-ip-address]:27300/ 
        ProxyPassReverse / http://[apache2-ip-address]:27300/
</VirtualHost>

Apache 2 sites-enabled config:

<VirtualHost *:27300>
        ServerName subdomain.domain.tld
        ServerAlias www.subdomain.domain.tld

        ServerAdmin [email protected]
        DocumentRoot /var/www/html

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        ErrorDocument 400 /notfound.html

        ProxyRequests off
        ProxyPreserveHost on
</VirtualHost>

If I directly hit http://[apache2-ip-address]:27300/ from the web browser the apache server landing page comes up fine. If I enter http://subdomain.domain.tld into the browser I get a proxy error:

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request

I logged a trace on both apache servers. Apache server 2 is receiving the proxied request from apache server 1 and is returning a 200 status response perfectly fine to apache server 1. The flow breaks at apache server 1 where I am seeing the following logs:

[Sat Jul 11 20:34:08.671267 2020] [proxy:debug] [pid 32275:tid 140388069250816] proxy_util.c(3075): AH00962: HTTP: connection complete to [apache2-ip-address]:27300 ([apache2-ip-address])
[Sat Jul 11 20:34:08.671333 2020] [core:trace6] [pid 32275:tid 140388069250816] core_filters.c(525): [remote [apache2-ip-address]:27300] core_output_filter: flushing because of FLUSH bucket
[Sat Jul 11 20:34:08.677508 2020] [proxy_http:error] [pid 32275:tid 140388069250816] (104)Connection reset by peer: [client xx.xxx.xxx.xx:39014] AH01102: error reading status line from remote server [apache2-ip-address]:27300
[Sat Jul 11 20:34:08.677575 2020] [proxy_http:debug] [pid 32275:tid 140388069250816] mod_proxy_http.c(1324): [client xx.xxx.xxx.xx:39014] AH01105: NOT Closing connection to client although reading from backend server [apache2-ip-address]:27300 failed.
[Sat Jul 11 20:34:08.677624 2020] [proxy:error] [pid 32275:tid 140388069250816] [client xx.xxx.xxx.xx:39014] AH00898: Error reading from remote server returned by /
[Sat Jul 11 20:34:08.677681 2020] [proxy:debug] [pid 32275:tid 140388069250816] proxy_util.c(2192): AH00943: HTTP: has released connection for ([apache2-ip-address])
[Sat Jul 11 20:34:08.677724 2020] [http:trace3] [pid 32275:tid 140388069250816] http_filters.c(1128): [client xx.xxx.xxx.xx:39014] Response sent with status 502, headers:

Things I've tried, from few other discussions I could find online, are the following changes to apache server 1 sites-enabled config :

1. SetEnv proxy-initial-not-pooled 1
2. SetEnv force-proxy-request-1.0 1
3. SetEnv proxy-nokeepalive 1
4. ProxyTimeout 600
5. ProxyPass / http://[apache2-ip-address]:27300/ timeout=600
6. ProxyPass / http://[apache2-ip-address]:27300/ nocanon

I've pretty much bruteforced the situation with several combinations of the above settings, but nothing seems to work. Any help is appreciated.

An additional check I ran is, if I run a nodejs application or python flask service on the same machine as either apache servers and proxy the service using ProxyPass / http://localhost:[port]/, the setup works properly. So both apache servers are running fine and are able to proxy services on their respective localhosts. Whatever is breaking has to do with the communication between the two apache servers.

UPDATE : Upon further triaging using curl with a networking person, the issue seems to be that the org firewall is only allowing inbound traffic to apache server 2 and blocking outbound traffic which may be causing 502 errors on apache server 1. This didn't seem like the issue up until I realized that my laptop was VPN'ed into the org network all along while testing and apache server 1 is sitting outside the org network. If this turns out to be the issue it's going to be a real bummer.

Solution

Adding following parameter in the http.conf file solves my issue of "proxy: error reading status line from remote server":

SetEnv proxy-initial-not-pooled 1

I go the reference from Apache URL https://httpd.apache.org/docs/2.4/mod/mod_proxy_http.html

Note: restart http server and try again.

Comments

[Dagmar]

http.conf is apache2.conf on ubuntu ... took me a while to figure this out

[Dagmar]

I think I had a similar situation ... there was a slight issue with the reverse proxy apache config which caused the timeouts. Once I fixed those the issue went away