Executing tasks in parallel

Question

Ok, so basically I have a bunch of tasks (10) and I want to start them all at the same time and wait for them to complete. When completed I want to execute other tasks. I read a bunch of resources about this but I can't get it right for my particular case...

Here is what I currently have (code has been simplified):

public async Task RunTasks()
{
    var tasks = new List<Task>
    {
        new Task(async () => await DoWork()),
        //and so on with the other 9 similar tasks
    }


    Parallel.ForEach(tasks, task =>
    {
        task.Start();
    });

    Task.WhenAll(tasks).ContinueWith(done =>
    {
        //Run the other tasks
    });
}

//This function perform some I/O operations
public async Task DoWork()
{
    var results = await GetDataFromDatabaseAsync();
    foreach (var result in results)
    {
        await ReadFromNetwork(result.Url);
    }
}

So my problem is that when I'm waiting for tasks to complete with the WhenAll call, it tells me that all tasks are over even though none of them are completed. I tried adding Console.WriteLine in my foreach and when I have entered the continuation task, data keeps coming in from my previous Tasks that aren't really finished.

What am I doing wrong here?

Solution 1

You should almost never use the Task constructor directly. In your case that task only fires the actual task that you can't wait for.

You can simply call DoWork and get back a task, store it in a list and wait for all the tasks to complete. Meaning:

tasks.Add(DoWork());
// ...
await Task.WhenAll(tasks);

However, async methods run synchronously until the first await on an uncompleted task is reached. If you worry about that part taking too long then use Task.Run to offload it to another ThreadPool thread and then store that task in the list:

tasks.Add(Task.Run(() => DoWork()));
// ...
await Task.WhenAll(tasks);

Solution 2

If you want to run those task's parallel in different threads using TPL you may need something like this:

public async Task RunTasks()
{
    var tasks = new List<Func<Task>>
    {
       DoWork,
       //...
    };

    await Task.WhenAll(tasks.AsParallel().Select(async task => await task()));

    //Run the other tasks
}

These approach parallelizing only small amount of code: the queueing of the method to the thread pool and the return of an uncompleted Task. Also for such small amount of task parallelizing can take more time than just running asynchronously. This could make sense only if your tasks do some longer (synchronous) work before their first await.

For most cases better way will be:

public async Task RunTasks()
{
    await Task.WhenAll(new [] 
    {
        DoWork(),
        //...
    });
    //Run the other tasks
}

To my opinion in your code:

1. You should not wrap your code in Task before passing to Parallel.ForEach.

2. You can just await Task.WhenAll instead of using ContinueWith.

Solution 3

Essentially you're mixing two incompatible async paradigms; i.e. Parallel.ForEach() and async-await.

For what you want, do one or the other. E.g. you can just use Parallel.For[Each]() and drop the async-await altogether. Parallel.For[Each]() will only return when all the parallel tasks are complete, and you can then move onto the other tasks.

The code has some other issues too:

• you mark the method async but don't await in it (the await you do have is in the delegate, not the method);

• you almost certainly want .ConfigureAwait(false) on your awaits, especially if you aren't trying to use the results immediately in a UI thread.

Solution 4

The DoWork method is an asynchronous I/O method. It means that you don't need multiple threads to execute several of them, as most of the time the method will asynchronously wait for the I/O to complete. One thread is enough to do that.

public async Task RunTasks()
{
    var tasks = new List<Task>
    {
        DoWork(),
        //and so on with the other 9 similar tasks
    };

    await Task.WhenAll(tasks);

    //Run the other tasks            
}

You should almost never use the Task constructor to create a new task. To create an asynchronous I/O task, simply call the async method. To create a task that will be executed on a thread pool thread, use Task.Run. You can read this article for a detailed explanation of Task.Run and other options of creating tasks.

View more solutions

Comments

[Alisson Reinaldo Silva]

By run synchronously until the first await did you want to mean they would actually run async only when we called Task.WhenAll? Thanks!

[i3arnon]

@Alisson I was more referring to the synchronous part of DoWork and not the code calling it.

[SaddamBinSyed]

@i3arnon, as you said, However, async methods run synchronously until the first await on an uncompleted task is reached. ---- SO here can I use "await task.delay(5).ConfigureAwait(false); statement as a first line in my DoWork() method? to return the main thread and start the Threadpool Thread for to execute remianing code in the method, ? Please advise.

[i3arnon]

@SaddamBinSyed you don't need Task.Delay. You have Task.Yield exactly for that purpose. But as I see it using Task.Run is better in that case because it's more explicit about "firing up" a new task.

[Denny Jacob]

An inspiring code snippet, perhaps the best on the net which combines task parallel with async using PLINQ.

[John Henckel]

As a suggestion: using dotNet 4.7+ or Core, you no longer need to the new[] {}. Instead you can just pass multiple tasks such as WhenAll(DoWork(), ....).