mod_proxy returns 503 errors even after proxied service is back up

Question

I have a setup with Apache2 as a front-end server for multiple python apps served by gunicorn. My Apache2 setup using mod_proxy looks like this:

<VirtualHost *:80>
    ServerName example.com
    UseCanonicalName On
    ServerAdmin webmaster@localhost

    LogLevel warn
    CustomLog /var/log/apache2/example.com/access.log combined
    ErrorLog /var/log/apache2/example.com/error.log
    ServerSignature On

    Alias /media/ /home/example/example.com/pysrc/project/media/

    ProxyPass /media/ !
    ProxyPass / http://127.0.0.1:4711/
    ProxyPassReverse / http://127.0.0.1:4711/
    ProxyPreserveHost On
    ProxyErrorOverride Off
</VirtualHost>

Generally, this setup works pretty well. I have one problem though: When I restart the gunicorn process (takes 2-5 seconds) and there is a request from Apache, that request will fail with a 503 error. So far so good. But Apache keeps returning 503 errors, even after the gunicorn process is back up. It resumes serving content from the proxied server only after a full restart of Apache.

Is there a way around this behaviour?

Solution

Add retry=0 to your ProxyPass lines:

ProxyPass / http://127.0.0.1:4711/ retry=0

From the mod_proxy documentation:

Connection pool worker retry timeout in seconds. If the connection pool worker to the backend server is in the error state, Apache will not forward any requests to that server until the timeout expires. This enables to shut down the backend server for maintenance, and bring it back online later. A value of 0 means always retry workers in an error state with no timeout.

Comments

[Benjamin Wohlwend]

Thanks for your answer! I use supervisord to control my gunicorn processes, my supervisor.conf looks basically like the example in the gunicorn repo. Unfortunately, I can't move away from Apache as a front-end server for organizational reasons.

[Benjamin Wohlwend]

Thank you so much! Works perfectly :) You'll get the bounty in a couple of hours when I'm allowed to award it.

[paul-schultz]

I believe you now want this link instead - the quoted paragraph seems to be missing from the version linked in the answer, and no mention of the retry option is made. For others wondering the same thing: the default value for this parameter is 60 seconds.