Directory.EnumerateFiles => UnauthorizedAccessException

Question

There is a nice new method in .NET 4.0 for getting files in a directory in a streaming way via enumeration.

The problem here is that if one wishes to enumerate all files one may not know in advance which files or folders are access protected and can throw an UnauthorizedAccessException.

To reproduce, one can just run this fragment:

foreach (var file in Directory.EnumerateFiles(@"c:\", "*", SearchOption.AllDirectories))
{
   // whatever
}

Before this .NET method existed it was possible to achieve roughly the same effect by implementing a recursive iterator on the string-array returning methods. But it's not quite as lazy as the new .NET method is.

So what to do? Can the UnauthorizedAccessException be suppressed or is a fact of life when using this method?

Seems to me that the method should have an overload accepting an action to deal with any exceptions.

Solution 1

Ths issue with the above answer is that is does not take care of exception in sub directories. This would be a better way to handling those exceptions so you get ALL files from ALL subdirectories except those with threw an access exception:

    /// <summary>
    /// A safe way to get all the files in a directory and sub directory without crashing on UnauthorizedException or PathTooLongException
    /// </summary>
    /// <param name="rootPath">Starting directory</param>
    /// <param name="patternMatch">Filename pattern match</param>
    /// <param name="searchOption">Search subdirectories or only top level directory for files</param>
    /// <returns>List of files</returns>
    public static IEnumerable<string> GetDirectoryFiles(string rootPath, string patternMatch, SearchOption searchOption)
    {
        var foundFiles = Enumerable.Empty<string>();

        if (searchOption == SearchOption.AllDirectories)
        {
            try
            {
                IEnumerable<string> subDirs = Directory.EnumerateDirectories(rootPath);
                foreach (string dir in subDirs)
                {
                    foundFiles = foundFiles.Concat(GetDirectoryFiles(dir, patternMatch, searchOption)); // Add files in subdirectories recursively to the list
                }
            }
            catch (UnauthorizedAccessException) { }
            catch (PathTooLongException) {}
        }

        try
        {
            foundFiles = foundFiles.Concat(Directory.EnumerateFiles(rootPath, patternMatch)); // Add files from the current directory
        }
        catch (UnauthorizedAccessException) { }

        return foundFiles;
    }

Solution 2

I Couldn't get the above to work, but here is my implementation, i've tested it on c:\users on a "Win7" box, because if has all these "nasty" dirs:

SafeWalk.EnumerateFiles(@"C:\users", "*.jpg", SearchOption.AllDirectories).Take(10)

Class:

public static class SafeWalk
{
    public static IEnumerable<string> EnumerateFiles(string path, string searchPattern, SearchOption searchOpt)
    {   
        try
        {
            var dirFiles = Enumerable.Empty<string>();
            if(searchOpt == SearchOption.AllDirectories)
            {
                dirFiles = Directory.EnumerateDirectories(path)
                                    .SelectMany(x => EnumerateFiles(x, searchPattern, searchOpt));
            }
            return dirFiles.Concat(Directory.EnumerateFiles(path, searchPattern));
        }
        catch(UnauthorizedAccessException ex)
        {
            return Enumerable.Empty<string>();
        }
    }
}

Solution 3

I understand it's MoveNext that throws the exception.

I tried to write a method that safe-walks a sequence and tries to ignore MoveNext exceptions. However I'm not sure if MoveNext advances position when it throws an exception, so this might as well be infinite loop. It is also bad idea because we would rely on implementation details.

But it's just so much fun!

public static IEnumerable<T> SafeWalk<T> (this IEnumerable<T> source)
{
    var enumerator = source.GetEnumerator();
    bool? hasCurrent = null;

    do {
        try {
            hasCurrent = enumerator.MoveNext();
        } catch {
            hasCurrent = null; // we're not sure
        }

        if (hasCurrent ?? false) // if not sure, do not return value
            yield return enumerator.Current;

    } while (hasCurrent ?? true); // if not sure, continue walking
}

foreach (var file in Directory.EnumerateFiles("c:\\", "*", SearchOption.AllDirectories)
                              .SafeWalk())
{
    // ...
}

This will only work if the following conditions are true about framework's implementation of this iterator (see FileSystemEnumerableIterator<TSource> in Reflector for reference):

• MoveNext advances its position when it fails;
• When MoveNext fails on last element, subsequent calls will return false instead of throwing an exception;
• This behavior is consistent for different versions of .NET Framework;
• I haven't made any logic or syntax mistakes.

Even if it works, please, never use it in production!
But I really wonder if it does.

Solution 4

I'm late, but I suggest using observable pattern instead:

public class FileUtil
{
  private static void FindFiles_(string path, string pattern,
    SearchOption option, IObserver<string> obs, CancellationToken token)
  {
    try
    {
      foreach (var file in Directory.EnumerateFiles(path, pattern,
        SearchOption.TopDirectoryOnly))
      {
        if (token.IsCancellationRequested) break;
        obs.OnNext(file);
      }

      if (option != SearchOption.AllDirectories) return;

      foreach (var dir in Directory.EnumerateDirectories(path, "*", 
        SearchOption.TopDirectoryOnly))
      {
        FindFiles_(dir, pattern, option, obs, token);
      }
    }
    catch (UnauthorizedAccessException) { }
    catch (PathTooLongException) { }
    catch (IOException) { }
    catch (Exception err) { obs.OnError(err); }
  }

  public static IObservable<string> GetFiles(string root, string pattern,
    SearchOption option)
  {
    return Observable.Create<string>(
      (obs, token) =>
        Task.Factory.StartNew(
          () =>
          {
            FindFiles_(root, pattern, option, obs, token);
            obs.OnCompleted();
          },
          token));
  }
}

Solution 5

Posting as an answer since I don't have the rep to add a comment let alone edit existing answers. My requirement was to minimize memory allocations, redundant variables, and have the system do a single enumeration of the directory.

static IEnumerable<string> FindFiles(string path, string filter = "*", bool recursive = false)
{
    IEnumerator<string> fEnum;
    try
    {
        fEnum = Directory.EnumerateFiles(path, filter, recursive ? SearchOption.AllDirectories : SearchOption.TopDirectoryOnly).GetEnumerator();
    }
    catch (UnauthorizedAccessException) { yield break; }
    while (true)
    {
        try { if (!fEnum.MoveNext()) break; }
        catch (UnauthorizedAccessException) { continue; }
        yield return fEnum.Current;
    }
}

Dan Bechard mentions in a comment:

Unfortunately, MoveNext() doesn't advance its position when it throws an exception.

This may have been fixed in either a newer release of .Net or in a Windows 10 build? I'm not having this issue in .NET 5.0 on Windows 10. Tested by searching my entire system drive.

---

In VB.NET:

Public Iterator Function FindFiles(path As String, Optional filter As String = "*", Optional recursive As Boolean = False) As IEnumerable(Of String)

    Dim fEnum As IEnumerator(Of String)
    Dim searchDepth = If(recursive, SearchOption.AllDirectories, SearchOption.TopDirectoryOnly)

    Try
        fEnum = Directory.EnumerateFiles(path, filter, searchDepth).GetEnumerator()
    Catch uae As UnauthorizedAccessException
        Return
    End Try

    Do While True
        Try
            If Not fEnum.MoveNext() Then Exit Do
            Yield fEnum.Current
        Catch uae As UnauthorizedAccessException

        End Try

    Loop

End Function

View more solutions

Comments

[Bent Rasmussen]

Not bad, but it wohn't work for my scenario: I need it to continue and move on when it meets an exception: the only difference between a safe-walk and a normal walk is that the safe-walk just stops enumeration whilst the normal method stops with an exception. I need it to continue and ignore any exceptions in the sense that it should enumerate all directories it can and just skip the ones it doesn't have access to. That, unfortunately it appears, requires a new implementation of the BCL implementation.

[Bent Rasmussen]

...I would have no issue using it in production if it worked ;-) ... but even then it would need a few modifications: for example you don't want to catch all exceptions, it should just be UnauthorizedAccessException or at least it should be filterable via a lambda.

[Till F.]

I also ran into a problem with this. The solution I came up with can be found at stackoverflow.com/questions/13130052/…. It behaves as a true enumerable in the sense that it only does work if you ask for the next item from it.

[Dan Bechard]

Unfortunately, MoveNext() doesn't advance its position when it throws an exception.

[Micha Wiedenmann]

what happens if this hits an a restricted file and the exception is thrown/ignored. Wouldn't it just stop there and ignore all files behind this point?

[Bent Rasmussen]

Something like DirectoryEnumerationPolicy.SkipUnauthorizedPaths would be nice as an additional argument to Directory.Enumerate{Directories,Files,FileSystemEntries}.

[Patrick Stalph]

The fastest option by far. Most solutions load all entries of one level into memory before returning them - which builds up quickly on recursion. This solution magically hopes for MoveNext to work even if it throws. Risky - tested with netstandard 2.0 and it worked to search C:\ recursively while $Recyclebin and system dirs were skipped.

[Massimo]

Partial and incorrect solution. You can access a directory and get access denied on specific files.

[leumasme]

Please do not reference other answers by "The Above Answer"...

[J. Scott Elblein]

You can also get a Could not find a part of the path w/this code.