How to force BundleCollection to flush cached script bundles in MVC4

Question

... or how I learned to stop worrying and just write code against completely undocumented APIs from Microsoft. Is there any actual documentation of the official System.Web.Optimization release? 'cuz I sure can't find any, there's no XML docs, and all the blog posts refer to the RC API which is substantially different. Anyhoo..

I am writing some code to automatically resolve javascript dependencies and am creating bundles on the fly from those dependencies. Everything works great, except if you edit scripts or otherwise make changes that would affect a bundle without restarting the application, the changes won't be reflected. So I added an option to disable caching of the dependencies for use in development.

However, apparently BundleTables caches the URL even if the bundle collection has changed. For example, in my own code when I want to re-create a bundle I do something like this:

// remove an existing bundle
BundleTable.Bundles.Remove(BundleTable.Bundles.GetBundleFor(bundleAlias));

// recreate it.
var bundle = new ScriptBundle(bundleAlias);

// dependencies is a collection of objects representing scripts, 
// this creates a new bundle from that list. 

foreach (var item in dependencies)
{
    bundle.Include(item.Path);
}

// add the new bundle to the collection

BundleTable.Bundles.Add(bundle);

// bundleAlias is the same alias used previously to create the bundle,
// like "~/mybundle1" 

var bundleUrl = BundleTable.Bundles.ResolveBundleUrl(bundleAlias);

// returns something like "/mybundle1?v=hzBkDmqVAC8R_Nme4OYZ5qoq5fLBIhAGguKa28lYLfQ1"

Whenever I remove & recreate a bundle with the same alias, absolutely nothing happens: the bundleUrl returned from ResolveBundleUrl is the same as before I removed & recreated the bundle. By "the same" I mean that the content hash is unchanged to reflect the new contents of the bundle.

edit... actually, it's much worse than that. The bundle itself is cached somehow outside of the Bundles collection. If I just generate my own random hash to prevent the browser from caching the script, ASP.NET returns the old script. So, apparently, removing a bundle from BundleTable.Bundles does not actually do anything.

I can simply change the alias to get around this problem, and that is OK for development, but I don't like that idea since it means either I have to deprecate aliases after each page load, or have a BundleCollection that grows in size on every page load. If you left this on in a production environment, it would be a disaster.

So it seems that when a script is served, it gets cached independent of the actual BundleTables.Bundles object. So if you re-use a URL, even if you've removed the bundle that it referred to before reusing it, it responds with whatever's in its cache, and altering the Bundles object does not flush the cache -- so only new items (or rather, new items with a different name) would ever be used.

The behavior seems odd... removing something from the collection should remove it from the cache. But it doesn't. There must be a way to flush this cache and have it use the current contents of the BundleCollection instead of what it cached when that bundle was first accessed.

Any idea how I would do this?

There is this ResetAll method which has an unknown purpose but it just breaks things anyway so that isn't it.

Solution 1

We hear your pain on documentation, unfortunately this feature is still changing quite fast, and generating documentation has some lag, and can be outdated almost immediately. Rick's blog post is up to date, and I've tried to answer questions here as well to spread current info in the meantime. We are currently in the process of setting up our official codeplex site which will have always current documentation.

Now in regards to your specific issue of how to flush bundles form the cache.

1. We store the bundled response inside of the ASP.NET cache using a key generated off of the bundle url requested, i.e. Context.Cache["System.Web.Optimization.Bundle:~/bundles/jquery"] we also setup cache dependencies against all of the files and directories that were used to generate this bundle. So if any of the underlying files or directories change, the cache entry will get flushed.

2. We don't really support live updating of the BundleTable/BundleCollection on a per request basis. The fully supported scenario is that bundles are configured during app start(this is so everything works properly in the web farm scenario, otherwise some bundle requests would end up being 404's if sent to the wrong server). Looking at your code example, my guess is that you are trying to modify the bundle collection dynamically on a particular request? Any kind of bundle administration/reconfiguration should be accompanied by an appdomain reset to guarantee everything has been setup correctly.

So avoid modifying your bundle definitions without recycling your app domain. You are free to modify the actual files inside of your bundles, that should automatically be detected and generate new hashcodes for your bundle urls.

Solution 2

I've got a similar problem.
In my class BundleConfig I was trying to see what was the effect of using BundleTable.EnableOptimizations = true.

public class BundleConfig
{
    public static void RegisterBundles(BundleCollection bundles)
    {
        BundleTable.EnableOptimizations = true;

        bundles.Add(...);
    }
}

Everything was working fine.
At some point I was doing some debugging and set the property to false.
I struggled to understand what was happening cause it seemed that the bundle for jquery (the first one) wouldn't be resolved and loaded (/bundles/jquery?v=).

After some swearing I think(?!) I've managed to sort things out. Try to add bundles.Clear() and bundles.ResetAll() at the beginning of the registration and things should start to work again.

public class BundleConfig
{
    public static void RegisterBundles(BundleCollection bundles)
    {
        bundles.Clear();
        bundles.ResetAll();

        BundleTable.EnableOptimizations = false;

        bundles.Add(...);
    }
}

I've realized I need to run these two methods only when I change the EnableOptimizations property.

UPDATE:

Digging deeper I've found out that BundleTable.Bundles.ResolveBundleUrl and @Scripts.Url seem to have problems to resolve the bundle path.

For sake of simplicity I've added a few images:

I have turned off the optimization and bundled a few scripts.

The same bundle is included in the body.

@Scripts.Url gives me the "optimized" path of the bundle while @Scripts.Render generates the proper one.
Same thing happens with BundleTable.Bundles.ResolveBundleUrl.

I am using Visual Studio 2010 + MVC 4 + Framework .Net 4.0.

Solution 3

Bearing in mind Hao Kung's recommendations to not do this because of web farm scenarios, I think there are a lot of scenarios where you might want to do this. Here is a solution:

BundleTable.Bundles.ResetAll(); //or something more specific if neccesary
var bundle = new Bundle("~/bundles/your-bundle-virtual-path");
//add your includes here or load them in from a config file

//this is where the magic happens
var context = new BundleContext(new HttpContextWrapper(HttpContext.Current), BundleTable.Bundles, bundle.Path);
bundle.UpdateCache(context, bundle.GenerateBundleResponse(context));

BundleTable.Bundles.Add(bundle);

You can call the above code at any time and your bundles will get updated. This works both when EnableOptimizations is true or false - in other words, this will throw out the correct markup in debug or live scenarios, with:

@Scripts.Render("~/bundles/your-bundle-virtual-path")

Solution 4

I also ran into issues with updating bundles without rebuilding. Here are the important things to understand:

• The bundle DOES NOT get updated if the file paths change.
• The bundle DOES get updated if the bundle's virtual path changes.
• The bundle DOES get updated if the files on disk change.

So knowing that, if you're doing dynamic bundling, you can write some code to make the bundle's virtual path be based on the file paths. I recommend hashing the file paths and appending that hash to the end of the bundle's virtual path. This way when the file paths change so does the virtual path and the bundle will update.

Here's the code I ended up with that solved the issue for me:

    public static IHtmlString RenderStyleBundle(string bundlePath, string[] filePaths)
    {
        // Add a hash of the files onto the path to ensure that the filepaths have not changed.
        bundlePath = string.Format("{0}{1}", bundlePath, GetBundleHashForFiles(filePaths));

        var bundleIsRegistered = BundleTable
            .Bundles
            .GetRegisteredBundles()
            .Where(bundle => bundle.Path == bundlePath)
            .Any();

        if(!bundleIsRegistered)
        {
            var bundle = new StyleBundle(bundlePath);
            bundle.Include(filePaths);
            BundleTable.Bundles.Add(bundle);
        }

        return Styles.Render(bundlePath);
    }

    static string GetBundleHashForFiles(IEnumerable<string> filePaths)
    {
        // Create a unique hash for this set of files
        var aggregatedPaths = filePaths.Aggregate((pathString, next) => pathString + next);
        var Md5 = MD5.Create();
        var encodedPaths = Encoding.UTF8.GetBytes(aggregatedPaths);
        var hash = Md5.ComputeHash(encodedPaths);
        var bundlePath = hash.Aggregate(string.Empty, (hashString, next) => string.Format("{0}{1:x2}", hashString, next));
        return bundlePath;
    }

Solution 5

Have you tried deriving from (StyleBundle or ScriptBundle), adding no inclusions in your constructor and then overriding

public override IEnumerable<System.IO.FileInfo> EnumerateFiles(BundleContext context)

I do this for dynamic style sheets and EnumerateFiles gets called on every request. It's probably not the greatest solution but it works.

View more solutions

Comments

[Jamie Treworgy]

Hmm... the thing is I don't actually want to clear out the bundle table, because it will contain lots of other ones from different pages (created from different sets of dependencies). But since this is really just for working in a development environment, I think I could copy the contents of it, then clear it, then add them again, if that would flush the cache. Horrendously inefficient but if it works, its good enough for dev.

[LeftyX]

Agree but that's the only option I had. I have spent the whole afternoon trying to understand what was the problem.

[Jamie Treworgy]

I just tried it, STILL not flushing the cache!! I clear it, ResetAll, and have tried setting EnableOptimizations to false both at startup and inline when I need to reset the cache, nothing happening. Argh.

[Jamie Treworgy]

It sure would be nice if the developer could fire off a quick blog post with even a one-liner about the methods in these objects :)

[LeftyX]

I reckon that problem is in BundleTable.Bundles.ResolveBundleUrl and @Scripts.Url If the optimization is off the path should be NOT the one of the bundle.

[Jamie Treworgy]

Saw your update... I need to try this without using ResolveBundleUrl too. This is insidious...

[LeftyX]

Maybe @Hao Kung can try to help us to understand.

[Jamie Treworgy]

+1'd you because this is very helpful, just not sure it's right yet since my problem isn't completely solved!

[Hao Kung]

So just to explain what these methods do: Scripts.Url is just an alias for BundleTable.Bundles.ResolveBundleUrl, it also will resolve non bundle urls, so its a generic url resolver that happens to know about bundles. Scripts.Render uses the EnableOptimizations flag to determine whether to render out a reference to bundles, or the components that make up the bundle.

[Jamie Treworgy]

thank you for bringing your direct knowledge to bear here! Yes - I am trying to modify the bundle collection dynamically. The bundles are built based on a set of dependencies described in another script (that is, itself, not necessarily part of the bundle) -- which is why I'm having this problem. Since changing a script that's in a bundle will force a flush, it can be done - is there a possibility of adding a manual flush method? This isn't crucial --this is for convenience during development - but I hate creating code that could cause problems if accidentally used on prod.

[Jamie Treworgy]

Also can you elaborate on the web farm issue? Would adding a new bundle after application start result it in only being available on the server on which it was created -- or just trying to change an existing one? This would be a bit of a dealkiller for what I'm trying to do since it needs to do runtime resolution of dependencies.

[Hao Kung]

Sure, we could add an explicit cache flush equivalent method, it's already there internally. Regarding the web farm issue, basically imagine you have two webservers A and B, your request goes to A who adds the bundle, and sends down the response, your client now goes to fetch the contents of the bundle, but oops the request goes to server B who did not register the bundle, and there's your 404.

[Jamie Treworgy]

I mean, is it possible for bundles to be just added after app start? What is the mechanism by which they are propogated initially, before they are cached?

[Hao Kung]

The cache update is lazy, the first time the bundle is used (typically via rendering a reference to the bundle), it is added to the cache. If you have an equivalent app start hook where you setup your bundles on all webservers before you start handling requests, that should be fine.

[Jamie Treworgy]

Sorry to keep asking you questions, I really appreciate the feedback. To be sure I understand - it sounds like adding a new bundle dynamically (at the time a particular page is first constructed) WOULD work fine after app start. Its first access would be via a rendered reference - it just happens that the bundle would have been created inline milliseconds before that reference was rendered. Does that make sense?

[Hao Kung]

Going back to the webfarm scenario, the only thing that matters, is if you can guarantee that on all servers, all bundles exist everywhere, irrespective of any requests that have been made to the server. So if some bundles are created on the fly only in response to a particular request, that will not work.

[philw]

As far as I can tell this doesn't work. That is, if I change the constituent file(s), the server cache isn't cleared as stated here. You have to recycle the thing to get any changes out there. Anyone know where that official documentation actually is?

[Zac]

Further reading here which talks a little about the caching and GenerateBundleResponse

[Christophe Blin]

see stackoverflow.com/a/34508048/1545567 to workaround the "I need to restart the pool" limitation from plain html

[ErikE]

I recommend generally avoiding Aggregate for string concatenation, due to the risk of someone not thinking about the inherent Schlemiel the Painter's algorithm in repeatedly using +. Instead, just do string.Join("", filePaths). This will not have that problem, even for very large inputs.

[Talha Junaid]

How to handle the scenario if there is a database e.g. Getting variable values for less from db then generating css file.

[Andreas Warberg]

"we also setup cache dependencies against all of the files and directories that were used to generate this bundle. So if any of the underlying files or directories change, the cache entry will get flushed." Has anyone actually seen this working? I have to cause the AppDomain to reload in order for the bundle to load an included file if it was changed after the first bundle request.