Is there an async version of DirectoryInfo.GetFiles / Directory.GetDirectories in dotNet?

Question

Is there an asynchronous version of DirectoryInfo.GetFiles / Directory.GetDirectories in dotNet? I'd like to use them in an F# async block, and it'd be nice to have a version that can be called with AsyncCallbacks.

Problem is I'm trying to suck in a bunch of directories, probably on SMB mounts over slow network connections, and I don't want a bunch of thread pool threads sitting around waiting for network reads when they could be doing other work.

Solution 1

No, I don't think there is. The pool thread approach is probably the most pragmatic. Alternatively, I guess you could drop down to P/Invoke - but that would be a lot more work.

Solution 2

I didn't find an async version of GetFiles, however if you look at the sourcecode for other Async operations, they're defined as follows:

module FileExtensions =

        let UnblockViaNewThread f =
            async { //let ctxt = System.Threading.SynchronizationContext.Current
                    do! Async.SwitchToNewThread ()
                    let res = f()
                    do! Async.SwitchToThreadPool ()
                    //do! Async.SwitchTo ctxt
                    return res }

        type System.IO.File with
            static member AsyncOpenText(path)   = UnblockViaNewThread (fun () -> System.IO.File.OpenText(path))
            static member AsyncAppendText(path) = UnblockViaNewThread (fun () -> System.IO.File.AppendText(path))
            static member AsyncOpenRead(path)   = UnblockViaNewThread (fun () -> System.IO.File.OpenRead(path))
            static member AsyncOpenWrite(path)  = UnblockViaNewThread (fun () -> System.IO.File.OpenWrite(path))
            static member AsyncOpen(path,mode,?access,?share) =
                let access = match access with Some v -> v | None -> System.IO.FileAccess.ReadWrite
                let share = match share with Some v -> v | None -> System.IO.FileShare.None
                UnblockViaNewThread (fun () -> System.IO.File.Open(path,mode,access,share))

            static member OpenTextAsync(path)   = System.IO.File.AsyncOpenText(path)
            static member AppendTextAsync(path) = System.IO.File.AsyncAppendText(path)
            static member OpenReadAsync(path)   = System.IO.File.AsyncOpenRead(path)
            static member OpenWriteAsync(path)  = System.IO.File.AsyncOpenWrite(path)
            static member OpenAsync(path,mode,?access,?share) = System.IO.File.AsyncOpen(path, mode, ?access=access, ?share=share)

In other words, the Async file, streamreader, and WebClient operations are just wrappers around the syncronous operations, so you should be able to write your own wrapper around GetFiles/GetDirectories as follows:

module IOExtensions =
    type System.IO.Directory with
        static member AsyncGetFiles(directory) = async { return System.IO.Directory.GetFiles(directory) }
        static member AsyncGetDirectories(path) = async { return System.IO.Directory.GetDirectories(path) }

Solution 3

This may be considered a bit of a hack, but you might consider using the UWP StorageFolder API.

C# example (though F# is probably just as easy):

using Windows.Storage;

...

var folder = await StorageFolder.GetFolderFromPathAsync(path);
var files = await folder.GetFilesAsync();
var folders = await folder.GetFoldersAsync();

You can easily consume these from traditional .NET desktop and console applications by using the UWP for Desktop library by Lucian Wischik (of Microsoft).

Install-Package UwpDesktop

Solution 4

Actually, according to the help for Directory.GetFiles, Directory.EnumerateFiles will return the first result immediately (it's an IEnumerable), rather than wait for the entire list before returning. I believe that's probably what you're looking for.

Solution 5

I've used several times this approach to get Async objects from functions/procedures, and it always worked great:

let AsyncGetDirectories path = 
    let fn = new Func<_, _>(System.IO.Directory.GetDirectories)
    Async.BuildPrimitive(path, fn.BeginInvoke, fn.EndInvoke)

View more solutions

Comments

[Noldorin]

This isn't true async I/O of course, but it's a good practical solution nonetheless. If you did want truly asynchronous operations you'd have to use some horrible Win32 interop, which I'd imagine isn't worth it in the context...

[Drew Noakes]

is this still the case now that .NET 4.5 is out with a bunch of new async APIs? I was looking for this and didn't see it.

[Mauricio Scheffer]

This is lazy, not async

[Jörgen Sigvardsson]

Well, stuff it in an async method/Task, what have you. By simply pushing everything off onto a different thread is a bad way to achieve asynchronicity IMHO. Yes, it runs in parallel, but it's very hard to terminate in a deterministic manner.

[Mauricio Scheffer]

Stuffing it in a async method/task is not the same, and is unrelated to laziness. I agree that "pushing it into a different thread" is "bad async", it should use IOCPs.

[James Moore]

It's fascinating that several answers have come up with this sort of solution, even though it has nothing to do with solving the stated problem. It's also interesting that the C# wrong answer is much longer than the equivalent wrong answers in F# :-).

[James Moore]

Plus, these days I'm using F# on iOS via Xamarin (and soon on Android), so some horrible Win32 interop isn't even possible.

[MvdD]

Not really. Even returning the first file may takes dozens or hundreds of milliseconds (think UNC paths). All that time your execution thread is blocked.

[Scott Hutchinson]

I could not find Princess's answer.

[James Moore]

@ScottHutchinson It's gone, no idea when it was deleted. (My question is from <gulp> 2009). I think it was essentially the same as Juliet's, pointing out that a lot of the async operators don't have good support anyway - they're going to consume threads and not do something sensible with wait handles.

[Scott Hutchinson]

I'm guessing Princess changed her name to Juliet, whose profile says she's a "High Priestess", which is kinda sorta like a princess.

[Dai]

Internally, what API does this library use? Does it call true-async (overlapped IO?) Win32 functions, or just wrap everything in a thread?

[Brugner]

It's still the case as of .NET Core 2.2.

[Dai]

"Alternatively, I guess you could drop down to P/Invoke" - I can't see any Win32 Overlapped IO or native Win32 asynchronous APIs for filesystem operations, so I'm not sure how P/Invoke would help here. As far as I know, the async filesystem API in UWP is also using threading internally instead of being actually overlapped IO or being IO interrupt-driven.

[PRMan]

This is the correct answer, because it's not faking async with threads but actually hooking into real interrupts in hardware/software.

[Matt Johnson-Pint]

It doesn't look like "UWP for Desktop" has stood the test of time. There's probably more modern ways to p/invoke UWP APIs. Maybe .NET 5 will help. :)

[Justin]

Enumerating a drive too over 5 minutes.

[Bent Tranberg]

No, "High Priestess" is a religious title, while a princess would be a member of a royal court, which is... hey, we're way off topic!