How do I add no-cache headers to all 404 pages served by apache and nginx?

Question

I have recently run into an issue after switching to Cloudflare, and the solution is to basically stop Cloudflare from caching 404 responses.

In our load-balanced multi-server setup, occasional 404s happen, but they're quickly fixed by rsync (via lsyncd). Before Cloudflare, a re-request to the 404ed file would very quickly become 200 as rsync does its job.

However, since Cloudflare caches all data based on the caching header, and neither apache nor nginx send a no-cache header for 404s, Cloudflare ends up caching the 404 response for a while.

I've been searching for a solution to globally add such a header for 404s in both apache and nginx (globally, for all hosted domains), but so far have come up blank.

Can anyone help?

Thank you.

Solution 1

Can't you get by with using an error_page directive, and then handle the location separately with the added header?

e.g. in Nginx:

    server {
      ...
      error_page 404 /404.html;
      location = /404.html {
        root   /usr/share/nginx/html;
        add_header Cache-Control "no-cache" always;
      }
    }

Solution 2

In apache 2.4, you could try something like:

FileETag None
<IfModule mod_headers.c>
    Header always unset ETag "expr=%{REQUEST_STATUS} == 404"
    Header always set Cache-Control "max-age=0, no-cache, no-store, must-revalidate" "expr=%{REQUEST_STATUS} == 404"
    Header always set Pragma "no-cache" "expr=%{REQUEST_STATUS} == 404"
    Header always set Expires "Wed, 11 Jan 1984 05:00:00 GMT" "expr=%{REQUEST_STATUS} == 404"
</IfModule>

The always is important because this is a:

You're adding a header to a locally generated non-success (non-2xx) response, such as a redirect, in which case only the table corresponding to always is used in the ultimate response.

You said all 404s, but for full reference of course it might make sense to wrap that in a <FilesMatch> or <LocationMatch> to limit the scope.

I believe this is a new capability in apache 2.4 as using expr conditionals is not the in the 2.2 version of the mod_headers documentation.

curl -I [foo] test without this config:

HTTP/1.1 404 Not Found
Date: Thu, 24 May 2018 17:44:29 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Type: text/html; charset=iso-8859-1

curl -I [foo] test with this config:

HTTP/1.1 404 Not Found
Date: Thu, 24 May 2018 17:44:42 GMT
Server: Apache/2.4.18 (Ubuntu)
Cache-Control: max-age=0, no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Content-Type: text/html; charset=iso-8859-1

Sources:

http://httpd.apache.org/docs/current/mod/mod_headers.html

Solution 3

You can do it this way too :

map $status $cache_header {
    default <for_other_codes>;
    404     "no-cache";
}


server {

    [ ... ]

    add_header "Cache-Control" $cache_header always;

}

Comments

[Admin]

Similar question, though only for apache (still unanswered too): serverfault.com/questions/331544/….

[Admin]

So far, I am pretty sure that you can't override headers returned by default 404 handlers in both apache and nginx (please prove me wrong!). I was able to override the 404 handler and point it to a PHP file that sends such headers in apache, but since nginx doesn't have PHP support on my setup, and setting "expires -1;" on the 404 location didn't seem to actually do anything, I'm still at a loss as to how to do this in nginx.

[Artem Russakovskii]

There are two issues here, but that's the direction I went with. 1. I didn't want to override the default 404 page and have to create rules in every listen since location is not supported inside http directly. 2. More importantly, your snippet actually doesn't work because add_header only applies to 20X and 30X (nginx.org/en/docs/http/ngx_http_headers_module.html). However, we're in luck, since as of the recently released 1.7.5, you can now add an always modifer which will apply it to all response codes. I had to upgrade nginx but it was a good kick in the butt. It works.

[jimmiw]

I updated my answer, you should perhaps add an answer showing what you did to solve the question?

[Artem Russakovskii]

Looks like omitting root works as well. If that's removed, it is pretty much what I ended up doing for nginx.

[Artem Russakovskii]

Clever. I am probably going to go with the other solution, since it already works and doesn't require a default Cache-Control header to be specified every time, but this was some thinking outside the box.

[Artem Russakovskii]

For apache, I decided that I actually wanted to use a custom page, so I did this: ErrorDocument 404 /path/to/my/404.php. And then inside the 404.php, I did <?php // do not cache 404 header("Cache-Control: no-cache, must-revalidate"); // HTTP/1.1 header("Expires: Sat, 26 Jul 1997 05:00:00 GMT"); // Date in the past ?>

[Artem Russakovskii]

I'm marking your answer as accepted. If someone wants to provide a more generic apache answer later, or an nginx one that works globally rather than per server, they can get upvoted.

[nh2]

There is a semicolon missing after 404 "no-cache" but stackexchange's stupid minimum-6-character-edit restriction prevents me from fixing it. Clearly not a good restriction for a site that's all about coding and configs...