How to clear complete cache in Varnish?

Question

I'm looking for a way to clear the cache for all domains and all URLs in Varnish.

Currently, I would need to issue individual commands for each URLs, for example:

curl -X PURGE http://example.com/url1
curl -X PURGE http://example.com/url1
curl -X PURGE http://subdomain.example.com/
curl -X PURGE http://subdomain.example.com/url1
// etc.

While I'm looking for a way to do something like

curl -X PURGE http://example.com/*

And that would clear all URLs under example.com, but also all URLs in sub-domains of example.com, basically all the URLs managed by Varnish.

Any idea how to achieve this?

This is my current VCL file:

vcl 4.0;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

sub vcl_recv {
    # Command to clear the cache
    # curl -X PURGE http://example.com
    if (req.method == "PURGE") {
        return (purge);
    }
}

Solution 1

With Varnish 4.0 I ended up implementing it with the ban command:

sub vcl_recv {
    # ...

    # Command to clear complete cache for all URLs and all sub-domains
    # curl -X XCGFULLBAN http://example.com
    if (req.method == "XCGFULLBAN") {
        ban("req.http.host ~ .*");
        return (synth(200, "Full cache cleared"));
    }

    # ...
}

Solution 2

Well, I suggest just restart varnish. It will purge all of the files because varnish keeps cache into memory.

Run: sudo /etc/init.d/varnish restart

Solution 3

Assuming no change of URL or internal cache key, for a full flush the simplest approach would be to restart Varnish, as it maintains its cache in memory.

If doing a quick restart is not acceptable, the BAN suggested by Rastislav is a great approach. It will need to stay active as long your longest TTL, so if you frequently need a full flush, the BAN list will be pretty much permanent as the ban lurker (which sweeps for BANs which no longer are relevant) may always think that your BAN is useful

So in your case, your VCL would be:

# Highly recommend that you set up an ACL for IPs that are allowed
# to make the BAN call
acl acl_ban {
    "localhost";
    "1.2.3.4"/32;
}

sub vcl_recv {
   if (client.ip ~ acl_ban && req.method == "BAN") {
      ban("req.http.host == " + req.http.host);
      # Throw a synthetic page so the request won't go to the backend.
      return(synth(200, "Ban added"));
   }
}

However as noted by Carlos in the comments, this will actually create a lazy invalidation (and so only removed at request time). If you want to have the objects actually get purged by the background ban lurker every so often, you can instead do:

# Highly recommend that you set up an ACL for IPs that are allowed
# to make the BAN call
acl acl_ban {
    "localhost";
    "1.2.3.4"/32;
}

sub vcl_recv {
   if (client.ip ~ acl_ban && req.method == "BAN") {
      # see below for why this is obj. rather than req.
      ban("obj.http.host == " + req.http.host);
      # Throw a synthetic page so the request won't go to the backend.
      return(synth(200, "Ban added"));
   }
}

sub vcl_backend_response {
   # add any portions of the request that would want to be able
   # to BAN on. Doing it in vcl_backend_response means that it
   # will make it into the storage object
   set beresp.http.host = bereq.http.host;
}

sub vcl_deliver {
   # Unless you want the header to actually show in the response,
   # clear them here. So they will be part of the stored object
   # but otherwise invisible
   unset beresp.http.host;
}

Then to do the flush:

curl -X BAN http://example.com;

Solution 4

Purge all Varnish cache from command line (invalidate all the cache):

varnishadm "ban.url ."  # Matches all URLs

Note: Command is purge.url in Varnish 2.x.

We can also ban by a hostname:

varnishadm "ban req.http.host == xxx.com"

View more solutions

Comments

[Carlos Abalde]

That's right, but beware that the ban you're creating in vcl_recv is not lurker friendly. It will be ignored by the ban lurker. That's not critical in some cases, but if your bans are going to match a lot of objects in the storage, you should avoid lazy invalidations and allow the ban lurker to purge objects. Creating lurker friendly bans requires a few more lines of VCL. Please check varnish-cache.org/docs/trunk/users-guide/purging.html#bans for details.

[Joshua DeWald]

That's a great note! I'll modify the answer with that in mind.

[Carlos Abalde]

Not need to write VCL to submit a ban. In any case, the way to go should be creating a ban lurker friendly. See varnish-cache.org/docs/trunk/users-guide/purging.html#bans for details.

[Spikes]

While this works, it's not very efficient for caches with a large number of objects. I'd suggest future readers check out Joshua DeWald's answer below, which is lurker friendly and thereby a bit easier on Varnish to process.

[OZZIE]

how do I unban it after?

[OZZIE]

It doesn't seem possible in any easy way.. #upvoteregrets

[TroodoN-Mike]

This will create downtime on the server. I would recommend to run varnishadm "ban req.url ~ /" instead

[Hoargarth]

Seems like in varnish5 they changed the command slightly. Managed to purge all varnish cache with varnishadm 'ban req.url ~ .'

[benh57]

Message from VCC-compiler: varnish-dev | 'beresp.http.host': cannot be unset in method 'vcl_deliver'. varnish-dev | At: ('/etc/varnish/conf.d/varnish.vcl' Line 195 Pos 10) varnish-dev | unset beresp.http.host;

[Joshua DeWald]

@benh57 I suspect my answer is no longer a good one with newer versions of Varnish :/