CSS File Not Updating on Deploy (Google AppEngine)

Question

I pushed a new version of my website, but now the CSS and static images are not deploying properly.

Here is the messed up page: http://www.gaiagps.com

Appengine shows the latest version as being correct though: http://1.latest.gaiagps.appspot.com/

Any help?

Solution 1

I've seen this before on App Engine, even when using cache-busting query parameters like /stylesheets/default.css?{{ App.Version }}.

Here's my (unconfirmed) theory:

1. You push a new version by deploying or changing a new version to default.
2. While this update is being propagated to all GAE instances running your app...
3. ...someone hits your site.
4. The request for static resource default.css{{ App.Version }} is sent to Google's CDN, which doesn't yet have it.
5. Google's CDN asks GAE for the resource before propagation from step #2 is done for all instances.
6. If you're unlucky, GAE serves up the resource from an instance running the old version...
7. ...which now gets cached in Google's CDN as the authoritative "new" version.

When this (if this is what happens) happens, I can confirm that no amount of cache-busting browser work will help. The Google CDN servers are holding the wrong version.

To fix: The only way I've found to fix this is to deploy another version. You don't run the risk of this happening again (if you haven't made any CSS changes since the race condition), because even if the race condition occurs, presumably your first update is done by the time you deploy your second one, so all instances will be serving the correct version no matter what.

Solution 2

Following is what has worked for me.

1. Serve your css file from the static domain. This is automatically created by GAE.

   //static.{your-app-id}.appspot.com/{css-file-path}

2. Deploy your application. At this point your app will be broken.

3. change the version of the css file

   //static.{your-app-id}.appspot.com/{css-file-path}?v={version-Name}

4. deploy again.

Every time you change the css file. you will have to repeat 2,3 and 4.

Solution 3

Your link looks fine to me, unless I'm missing something.

You may have cached your old CSS, and not getting the new CSS after updating it. Try clearing your browser cache and see if that works.

Going to 1.latest downloads the new CSS since it's not in your cache, so it appears correctly to you.

Solution 4

I had this problem as well. I was using flask with GAE so I didn't have a static handler in my app.yaml. When I added it, the deploy works. Try adding something like this

handlers:
- url: /static
  static_dir: static

to your app.yaml and deploy again. It worked for me. Apparently Google is trying to optimize by not updating files that it thinks users can't see.

Solution 5

As found by shoresh, the docs for the standard environment for Pyhton state that both settings for static cache expiration, the individual element expiration and the top-level element default_expiration, are responsible for defining "the expiration time [that] will be sent in the Cache-Control and Expires HTTP response headers". This means that "files are likely to be cached by the user's browser, as well as by intermediate caching proxy servers such as Internet Service Providers".

The problem here is that "re-deploying a new version of the app will not reset any caches". So if one has set default_expiration to, e.g., 15 days, but makes a change to a CSS or JS file and re-deploy the app, there is no guarantee that those files will be automatically served due to active caches, particularly due to intermediate caching proxy servers, which may include Google Cloud servers - what seems to be the case since accessing your-project-name.appspot.com also serves outdated files.

The same documentation linked above states that "if you ever plan to modify a static file, it should have a short (less than one hour) expiration time. In most cases, the default 10-minute expiration time is appropriate". That is something one should think about before setting any static cache expiration. But for those who, like myself, didn't know all of this beforehand and have already been caught by this problem, I've found a solution.

Even though the documentation states that it's not possible to clear those intermediate caching proxies, one can delete at least the Google Cloud cache.

In order to do so, head to your Google Cloud Console and open your project. Under the left hamburger menu, head to Storage -> Browser. There you should find at least one Bucket: your-project-name.appspot.com. Under the Lifecycle column, click on the link with respect to your-project-name.appspot.com. Delete any existing rules, since they may conflict with the one you will create now.

Create a new rule by clicking on the 'Add rule' button. For the object conditions, choose only the 'Newer version' option and set it to 1. Don't forget to click on the 'Continue' button. For the action, select 'Delete' and click on the 'Continue' button. Save your new rule.

This newly created rule will take up to 24 hours to take effect, but at least for my project it took only a few minutes. Once it is up and running, the version of the files being served by your app under your-project-name.appspot.com will always be the latest deployed, solving the problem. Also, if you are routinely editing your static files, you should remove the default_expiration element from the app.yaml file, which will help avoid unintended caching by other servers.

View more solutions

Comments

[Andrew Johnson]

The menu at the bottom of the menu is not horizontal or big enough, and the images in the slideshow are wrong. I have tried refreshing my cache and loading from a different browser too.

[Jason Hall]

The two sites you linked look exactly the same on Firefox and Chrome for OS X. I suspect there's still some issue that's only affecting you, or your browser, unless someone else can verify that it looks different.

[Emilien]

Try using <Shift>+<F5> to force reload your page (at least in FF). Here everything seems fine, both menu and slideshow images. Good luck.

[Andrew Johnson]

The problem was server-side caching as it turns out. App Engine is weird.

[Andrew Johnson]

I'll buy this... though I'm not sure it's right. It eventually just cleared itself up hours later.

[mainsocial]

I am encountering it now. I've deployed many, many times but it keeps serving the outdated static files. However if I reference the version of the app - eg, alpha5.latest.xxx.appspot.com - then the correct file is served.

[Clint Simon]

I've encountered this problem a few times. For whatever reason, this is the fixed. I think it's something to do with upstream caching from the google instance web host. If you see the problem and request the css file directly in a browser with the querystring, then the issue goes away. It looks like the caching is invalidated the first time a request is made with a unique url to a static file.

[Frank]

Easy and quick solution, I also use date instead of "App.version" ie. foo.css?05022015 where the date when the css was changed.

[Javier Marín]

What about you serve an static file for your domain index? This problem still happen and cannot be solved with a cache buster

[kiwicopple]

I also was encountering this and the only solution I could find was to deploy to a new bucket. To do this, go to console.cloud.google.com/storage/browser and create a new bucket, then redeploy using gcloud app deploy --bucket gs://your-new-bucket. This uploaded all the files again and created a new version in App Engine

[Klemens Zleptnig]

Just be aware that if you have had deployed a static file before, App Engine will still serve the old file, even if the default_expiration is set. A workaround for this is to rename the file.

[danday74]

This answer +1 formed the basis for the solution that finally worked for me which is fully documented here - stackoverflow.com/a/69781887/1205871