Max Degree of Parallelism for AsParallel()

Question

While using Parallel.ForEach we have the option to define the Parallel options and set the Max Degree of Parallelism like :

Parallel.ForEach(values, new ParallelOptions {MaxDegreeOfParallelism = number}, value = > {
    // Do Work
})

But while doing PLINQ like:

Tabel.AsEnumberable()
     .AsParallel()
     .Where(//Logic)

I was not able to find a way to set MaxDegreeOfParallelism. I looked up on the net as well, but didn't find anything. As anyone found a way around this? Any help is appreciated.

Solution 1

You can use ParallelEnumerable.WithDegreeOfParallelism:

Sets the degree of parallelism to use in a query. Degree of parallelism is the maximum number of concurrently executing tasks that will be used to process the query.

var result = Tabel.AsEnumberable()
                  .AsParallel()
                  .WithDegreeOfParallelism(number)
                  .Where(/* predicate */);

Edit:

@svick provided an excellent on ParallelOptions.MaxDegreeOfParallelism vs PLINQ’s WithDegreeOfParallelism which emphasizes the difference between the two:

Parallel works using an under-the-covers concept we refer to as replicating tasks. The concept is that a loop will start with one task for processing the loop, but if more threads become available to assist in the processing, additional tasks will be created to run on those threads. This enables minimization of resource consumption. Given this, it would be inaccurate to state that ParallelOptions enables the specification of a DegreeOfParallelism, because it’s really a maximum degree: the loop starts with a degree of 1, and may work its way up to any maximum that’s specified as resources become available.

PLINQ is different. Some important Standard Query Operators in PLINQ require communication between the threads involved in the processing of the query, including some that rely on a Barrier to enable threads to operate in lock-step. The PLINQ design requires that a specific number of threads be actively involved for the query to make any progress. Thus when you specify a DegreeOfParallelism for PLINQ, you’re specifying the actual number of threads that will be involved, rather than just a maximum.

Solution 2

Yes, you can certainly do that. You just use WithDegreeOfParallelism extension method

yourSequence.AsParallel()
    .WithDegreeOfParallelism(5)//Whatever number as you like
    .Where(...);

Comments

[svick]

You might also want to read ParallelOptions.MaxDegreeOfParallelism vs PLINQ’s WithDegreeOfParallelism on the difference between the two.

[crokusek]

This answer appears to be at odds with this one

[Yuval Itzchakov]

@crokusek I'm not sure where that answer got its references from, but it doesn't seem to back itself up by any offical docs. Also, I'm not sure where they got the notion that the first IO operations would block all other operations. Sounds peculiar.

[crokusek]

Unfortunately I am seeing behavior in line with the other answer--that is that specifying WithDegreeOfParallelism(32) is still hard limited to 8 on a 4 core machine. Does anyone have a proof where the value was used directly where N threads >> 2*core?

[Yuval Itzchakov]

@codingpirate Just wondering, you unmarked my answer as accepted for an answer with an identical answer, almost a year later? Any particular reason why?

[codingpirate]

@YuvalItzchakov - To close the thread I accepted the ans. Having said that I also have your explanation as an accepted answer as well

[omer schleifer]

From a small test I made. it doesn’t seem that the actual number of threads equals DegreeOfParallelism . I assigned DegreeOfParallelism = 100 , and ran a small program that records for each “task” it’s thread id – only 9 threads were actually used. I was running in .Net 4.5.1

[Yuval Itzchakov]

@omer Did you test have enough work for 100 threads to run concurrently?

[omer schleifer]

Each work item consisted of thread.sleep, plus writes thread id. Only 9 threads were used each ran multiple work items

[Yuval Itzchakov]

@omerschleifer Post a gist of your test, I'll take a look at it.