Running multiple async tasks and waiting for them all to complete

Question

I need to run multiple async tasks in a console application, and wait for them all to complete before further processing.

There's many articles out there, but I seem to get more confused the more I read. I've read and understand the basic principles of the Task library, but I'm clearly missing a link somewhere.

I understand that it's possible to chain tasks so that they start after another completes (which is pretty much the scenario for all the articles I've read), but I want all my Tasks running at the same time, and I want to know once they're all completed.

What's the simplest implementation for a scenario like this?

Solution 1

Both answers didn't mention the awaitable Task.WhenAll:

var task1 = DoWorkAsync();
var task2 = DoMoreWorkAsync();

await Task.WhenAll(task1, task2);

The main difference between Task.WaitAll and Task.WhenAll is that the former will block (similar to using Wait on a single task) while the latter will not and can be awaited, yielding control back to the caller until all tasks finish.

More so, exception handling differs:

Task.WaitAll:

At least one of the Task instances was canceled -or- an exception was thrown during the execution of at least one of the Task instances. If a task was canceled, the AggregateException contains an OperationCanceledException in its InnerExceptions collection.

Task.WhenAll:

If any of the supplied tasks completes in a faulted state, the returned task will also complete in a Faulted state, where its exceptions will contain the aggregation of the set of unwrapped exceptions from each of the supplied tasks.

If none of the supplied tasks faulted but at least one of them was canceled, the returned task will end in the Canceled state.

If none of the tasks faulted and none of the tasks were canceled, the resulting task will end in the RanToCompletion state. If the supplied array/enumerable contains no tasks, the returned task will immediately transition to a RanToCompletion state before it's returned to the caller.

Solution 2

You could create many tasks like:

List<Task> TaskList = new List<Task>();
foreach(...)
{
   var LastTask = new Task(SomeFunction);
   LastTask.Start();
   TaskList.Add(LastTask);
}

Task.WaitAll(TaskList.ToArray());

Solution 3

You can use WhenAll which will return an awaitable Task or WaitAll which has no return type and will block further code execution simular to Thread.Sleep until all tasks are completed, canceled or faulted.

	WhenAll	WaitAll
Any of the supplied tasks completes in a faulted state	A task with the faulted state will be returned. The exceptions will contain the aggregation of the set of unwrapped exceptions from each of the supplied tasks.	An AggregateException will be thrown.
None of the supplied tasks faulted but at least one of them was canceled	The returned task will end in the TaskStatus.Canceled state	An AggregateException will be thrown which contains an OperationCanceledException in its InnerExceptions collection
An empty list was given	An ArgumentException will be thrown	The returned task will immediately transition to a TaskStatus.RanToCompletion State before it's returned to the caller.
	Doesn't block the current thread	Blocks the current thread

Example

var tasks = new Task[] {
    TaskOperationOne(),
    TaskOperationTwo()
};

Task.WaitAll(tasks);
// or
await Task.WhenAll(tasks);

If you want to run the tasks in a particular/specific order you can get inspiration from this answer.

Solution 4

The best option I've seen is the following extension method:

public static Task ForEachAsync<T>(this IEnumerable<T> sequence, Func<T, Task> action) {
    return Task.WhenAll(sequence.Select(action));
}

Call it like this:

await sequence.ForEachAsync(item => item.SomethingAsync(blah));

Or with an async lambda:

await sequence.ForEachAsync(async item => {
    var more = await GetMoreAsync(item);
    await more.FrobbleAsync();
});

Solution 5

Yet another answer...but I usually find myself in a case, when I need to load data simultaneously and put it into variables, like:

var cats = new List<Cat>();
var dog = new Dog();

var loadDataTasks = new Task[]
{
    Task.Run(async () => cats = await LoadCatsAsync()),
    Task.Run(async () => dog = await LoadDogAsync())
};

try
{
    await Task.WhenAll(loadDataTasks);
}
catch (Exception ex)
{
    // handle exception
}

View more solutions

Comments

[Ravi]

I would recommend WhenAll

[Matt W]

Is it possible to start multiple new threads, at the same time, using the await keyword rather than .Start() ?

[Virus]

@MattW No, when you use await, it would wait for it to complete. In this case you would not be able to create a multi-threaded environment. This is the reason that all the tasks are waited at the end of the loop.

[Zapnologica]

When I try this my tasks run sequentially? Does one have to start each task individually before await Task.WhenAll(task1, task2); ?

[Yuval Itzchakov]

@Zapnologica Task.WhenAll doesn't start the tasks for you. You have to provide them "hot", meaning already started.

[Zapnologica]

Ok. That makes sense. So what will your example do? Because you have not started them?

[Yuval Itzchakov]

@Zapnologica In my example, both methods are the one to kick off the tasks. Task.WhenAll asynchronously waits for both of them to complete, but their execution happens concurrently.

[Emil]

is it the same to call Await Tasks.Task.Run(()=>dowork) for each function sequentially?

[Yuval Itzchakov]

@batmaci No, await on each Task will await each task one at a time. using Task.WhenAll will concurrently wait for the completion of all tasks.

[JRoughan]

Downvote for future readers since it isn't made clear that this is a blocking call.

[Daniel Dušek]

@YuvalItzchakov thank you very much! It is so simple but it helped me a lot today! Is worth at least +1000 :)

[EL MOJO]

See the accepted answer for reasons why not to do this.

[Talha Talip Açıkgöz]

Why don't you just keep it as Task Array?

[Pierre]

if you are not going to use await Task.WhenAll(...) then you can just StartNew on all the tasks and let it be

[Yuval Itzchakov]

@Pierre I'm not following. What does StartNew and spinning new tasks have to do with asynchronously waiting on them all?

[Pierre]

Nothing at all :)

[Ryan]

If there is an exception thrown inside of DoWorkAsync() or DoMoreWorkAsync() before the await calls are reached, won't the exception be lost? docs.microsoft.com/en-us/dotnet/visual-basic/language-refere‌​nce/…

[Yuval Itzchakov]

@Ryan AFAIR Exceptions that happen before the first await point are thrown synchronously.

[DalSoft]

If your not careful @talha-talip-açıkgöz your execute the Tasks when you didn't expect them to execute. Doing it as a Func delegate makes your intent clear.

[dee zg]

sorry for coming to the party late but, why do you have await for each operation and at the same time use WaitAll or WhenAll. Shouldn't tasks in Task[] initialization be without await?

[NtFreX]

@dee zg You are right. The await above defeats the purpose. I ll change my answer and remove them.

[dee zg]

yes, that's it. thanks for clarification! (upvote for nice answer)

[Stephen Kennedy]

If LoadCatsAsync() and LoadDogAsync() are just database calls they are IO-bound. Task.Run() is for CPU-bound work; it adds additional unnecessary overhead if all you are doing is waiting for a response from the database server. Yuval's accepted answer is the right way for IO-bound work.

[Yehor Hromadskyi]

@StephenKennedy could you please clarify what kind of overhead and how much it can impact performance? Thanks!

[Stephen Kennedy]

That would be quite hard to summarise in the comments box :) Instead I recommend reading Stephen Cleary's articles - he's an expert on this stuff. Start here: blog.stephencleary.com/2013/10/…

[PreguntonCojoneroCabrón]

how get the results of Tasks? For example, for merge "rows" (from N tasks in parallel) in a datatable and bind it to gridview asp.net ?

[David Yates]

Another good example of using Task.WhenAll is here and it shows getting result from each afterwards.

[Theodor Zoulias]

This approach introduces the risk of leaking a fire-and-forget task, in case the first task completes with failure before the completion of the second task. The correct way to await multiple tasks is the Task.WhenAll method: await Task.WhenAll(first, second);. Then you can await them individually to get their results, because you know that all have completed successfully.

[Fax]

@TheodorZoulias Is there a problem with leaking fire-and-forget tasks? It seems that for a console application at least, you don't get much benefit from waiting ten minutes on WhenAll to find out that you misspelled the input file name.

[Theodor Zoulias]

It depends on what this fire-and-forget task does. In the best case it just consumes resources, like network bandwidth, that are going to waste. In the worst case it modifies the state of the application, at a time when it's not expected to happen. Imagine that a user clicks a button, they get an error message, the button is re-enabled, and then the ProgressBar continues moving up and down by the ghost task... This never happens by any tool provided by Microsoft (Parallel, PLINQ, TPL Dataflow etc). All these APIs do not return before all internally initiated operations are completed.

[Theodor Zoulias]

If a failure of one task makes the result of another task irrelevant, then the correct course of action is to cancel the still-running task, and await it to complete as well. Awaiting each task sequentially, as your answer suggests, is rarely a good idea. If you decide that leaking fire-and-forget tasks is OK for your use case, then symmetrically a failure on second should also leak the first. Your code doesn't support that. Its leaking behavior is asymmetric.