MSBuild Item Include (wildcard) not expanding

Question

This is very weird. We've been trying to figure it out for a while, but it really doesn't make any sense.

Our web project imports a targets file that has a target similar to this:

<Target Name="CSSCheckInternal">
    <ItemGroup>
        <CSSFiles Include="$(MSBuildProjectDirectory)\**\*.css" />
    </ItemGroup>
    <CSSChecker Files="@(CSSFiles)" />
</Target>

At the moment, one branch is building perfectly, executing the task as desired; but the other branch is failing on the above target.

The failure is because the @(CSSFiles) item, when received by the task, appears not to be expanding into an ITaskItem array.

The task is written as follows (up to the point where I get the FullPath metadata):

public class CSSChecker : Task
{
    [Required]
    public ITaskItem[] Files
    {
        get;
        set;
    }

    public override bool Execute()
    {
        string fullFilePath = null;
        if (Files != null)
        {
            foreach (var item in Files)
            {
                fullFilePath = item.GetMetadata("FullPath");
                if(!File.Exists(fullFilePath))
                  throw new InvalidOperationException(
                   string.Format("{0} does not exist", fullFilePath));


        //rest of the code elided

The build that's failing is throwing the InvalidOperationException on the last line there, like this:

File does not exist: C:\Code\Project\**\*.css

So it would seem that MSBuild, instead of expanding the wildcard in the Include attribute, is merely passing the string over, thus creating only one ITaskItem on the task.

The target folder does exist on the disk, and the only difference between the broken project file and the working one is a single file include much earlier in the project file.

Update

I asked Sayed Hashimi on twitter (wrote the MSBuild book) and through that, tried taking out the ** folder wildcard and it's now started working. This isn't really suitable as the task is meant to be re-usable between projects. But it would appear to be something to do with this.

End update

Please if anyone knows under what situation MSBuild would not correctly expand a wildcard, it would be a great help!

Solution 1

I've figured this out - I had to delete the obj\ folder in my project directory, and all of a sudden the folder wildcard starting working again.

The short answer for my situation is that it appears MSBuild's wildcard handling code craps out completely if any path is too long, and just doesn't build an item group.

The thing here being how did I manage to create paths that were that long? Well, I didn't. It was the built-in web publishing task - which I use like this (for a custom deployment that we do):

<MSBuild Projects="$(Proj)" Properties="Platform=$(Platform);
 Configuration=$(Configuration);DeployOnBuild=true;PackageAsSingleFile=False;
 AutoParameterizationWebConfigConnectionStrings=False" />

When you do PackageAsSingleFile=False, which I use to prevent building the zip as I want the website deployables, in the obj folder you get a folder structure like this:

[Project_Dir]\obj\[configuration]\Package\PackageTemp\[Project Dir]\[output *]

If [Project_Dir] is c:\my project\, then the base folder for the temporary package files will be something like c:\my project\obj\debug\Package\PackageTemp\c_c\my project\.

As you can see, that's quite a deep folder structure already, and in reality projects are generally not top-level folders on the root of a drive.

I've found, with some of our projects that use this deployment method, that it becomes impossible to delete the obj\ folder in Explorer or at the command line because the path has got too long. What I do to get around this is to rename as many parent folders as required simply to 1 in order to shorten the full path and then do the delete. I.e in the previous example, I would rename as follows:

c:\my project\obj\1\1\1\1\1

Which works well.

You can imagine - if the project starts off in a deep-enough folder - then the eventual paths of items generated for the publish task will get very long. I've found that if I just use the Publish task from within VS this actually causes an error during the publish - but it would seem that shelling MSBuild in the way I show above, actually somehow sidesteps the folder max path limitation. I'm going to put together a project that proves this soon.

So, in my case, I've had to rewrite my Task to take the base folders that are to be processed, and then have it recurse through the folders and files itself, ignoring any 'obj' folder it finds.

I tried excluding any files in the obj folder using the 'Exclude' attribute, but it made no difference (presumably because both were crapping out!).

Solution 2

I had the same problem and can confirm that it was also related to long paths. Mine were in the node_modules folder which is managed by npm.

I have found that I am able to delete long paths without having to rename folders as Andreas did using the rimraf tool. It is worth just trying again if you get any errors.

Also, if it is not actually true recursion that you are interested in - say if you are only using the double asterisk (**) syntax as a shortcut for including three folder levels - then you could replace the double asterisk syntax with a series of expressions using single asterisks (*).

For example, assuming that there are only three levels of folders that actually need to be included, you could swap this...

C:\**\*.*

...for this.

C:\*\*.*
C:\*\*\*.*
C:\*\*\*\*.*

Solution 3

This problem only happens if any path within the recursive search is longer than MAX_PATH. You can check this by trying to create a directory in your deepest path with explorer, you'll get an error telling you that the path is too long and the directory can't be created. Or if you want to reproduce this issue create a path which is longer than max path using CreateFileW() with the extended \\\?\ path syntax.

MsBuild must internally check the length before it calls FindNextFile() again as you never see any path too long errors in procmon. What you'll see is that one level up before the path is too long it simply gets the expected ERROR_NO_MORE_FILES, then closes the search handle, but doesn't even bother trying the next level down. Once this happens MsBuild will give up its enumeration with no error!

Solution 4

An invalid NTFS filesystem link will also cause this behavior, for example with a hard link in the included folder pointing to a non-existing path.

View more solutions

Comments

[redcalx]

I'm getting a similar issue where a pattern of the form "foo.*.xml" doesn't expand (or match at least), but "*.xml" does.

[Andras Zoltan]

I had to delete my obj folder - due to a web deployment (not a standard one, though) creating folder structures in the obj\ folder that were too deep.

[Sir Crispalot]

This was the answer for me. I knew that my files would be at minimum three levels deep and at most six so could use ..\S\*\*\*\bin\Debug\*Test*.dll, ..\S\*\*\*\*\bin\Debug\*Test*.dll, and so on.

[mdiehl13]

Same as me, I had tests that specifically were creating paths greater than 255 characters. I stopped cleaning up my tests in the ClassCleanup, and this issue started happening. Thanks for helping me connect the dots!