TaskFactory.StartNew -> System.OutOfMemoryException

Question

There around 1000 task running but sometimes i receive the following aout of memory exception thrown by task scheduler. What could be the reason and how to avoid it.

System.Threading.Tasks.TaskSchedulerException: An exception was thrown by a TaskScheduler. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Threading.Thread.StartInternal(IPrincipal principal, StackCrawlMark& stackMark)
   at System.Threading.Thread.Start(StackCrawlMark& stackMark)
   at System.Threading.Thread.Start(Object parameter)
   at System.Threading.Tasks.ThreadPoolTaskScheduler.QueueTask(Task task)
   at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
   at System.Threading.Tasks.Task.InternalStartNew(Task creatingTask, Object action, Object state, CancellationToken cancellationToken, TaskScheduler scheduler, TaskCreationOptions options, InternalTaskOptions internalOptions, StackCrawlMark& stackMark)
   at System.Threading.Tasks.TaskFactory.StartNew(Action action, CancellationToken cancellationToken, TaskCreationOptions creationOptions, TaskScheduler scheduler)
   at App.StartReadSocketTask()

Solution 1

Your (non x64) App has a max memory space of 2GB. Each Thread requires a minimum of 1 MB, typically you can expect OOM before you reach 1000 Threads.

In itself the Task class is supposed to address this (by using the ThreadPool). But when your Tasks take too long (> 500 ms) the TP will slowly add Threads, failing after a few minutes or longer.

The simplest solution might be to look in your code where this unbounded creation of Tasks occurs, and see if you can limit in a way that agrees with your solution. Like if you are using a Producer/Consumer Que, make it a bounded queue.

Otherwise, limit the MaxThreads but this is a blunt, application-wide instrument.

Solution 2

As i was experimenting with testing the limits of the parallel system i encountered this very issue myself. oleksii's comment is spot on (1k threads ~= 1GB of committed memory). It is important to note that this memory is reserved virtual address space NOT the amount of memory actually 'used'. Out of memory exceptions occur when the system cannot commit a contiguous chunk of virtual address space large enough to satisfy your request (insert 'memory fragmentation' rhetoric here). If you view the process in the windows task manager about the time it dies you may see as little as 80-120mb of 'used' memory. To see how much virtual address space is reserved show the "Memory - Commit Size" column in task manager.

To keep this short i was able to break past the ~1k thread limit by switching my build config from x86 to 64 bit. This increases the amount of virtual address space available from (roughly) 2GB to 6TB+ (depending on your OS version) and my OutOfMemoryException went away.

Here's the simple program I created that illustrates this artifact, be sure to run it as x86 and watch it die somewhere between 1k and 1.5k threads - then switch to 64 bit and it should run to completion without failing.

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Threading;

namespace TaskToy
{
    class Program
    {
        static void Main( string[] args )
        {
            List<Task> lTasks = new List<Task>();
            int lWidth = 0;
            for ( int i = 0; i < 5000; i ++ )
            {
                lTasks.Add( new Task( (o) => {

                    Console.WriteLine( "B " + Interlocked.Increment( ref lWidth ) + " tid " + Thread.CurrentThread.ManagedThreadId );
                    Thread.Sleep( 60000 );
                    Console.WriteLine( "E " + Interlocked.Decrement( ref lWidth ) + " tid " + Thread.CurrentThread.ManagedThreadId );
                }, null, TaskCreationOptions.LongRunning ) );
            }

            Parallel.For( 0, lTasks.Count, ( i ) =>
            {
                lTasks[i].Start();
            } );

            Task.WaitAll( lTasks.ToArray() );
            Console.WriteLine( "DONE - press any key..." );
            Console.ReadKey( true );
        }
    }
}

P.S. The 'lWidth' variable indicates the current level of concurrency, that is, how many tasks are actually running at once.

Overall this was a fun academic experiment but it will likely be a 'few' years before running thousands of threads provides a useful return. It is likely advisable to limit the number of threads you spin up to something more practical - likely an order of magnitude less than 'thousands'.

Solution 3

I believe you've run into a fun part of the ThreadPool where it has decided to add more worker threads because your current tasks are "starving" the waiting tasks. Eventually this causes your application to run out of memory.

I suggest adding the TaskCreationOptions.LongRunning flag on creation. This will let the ThreadPool know it should consider oversubscription of the tasks.

From the book Parallel Programming with Microsoft .Net:

As a last result, you can use the SetMaxThreads method to configure the ThreadPool class with an upper limit for the number of worker threads, usually equal to the number of cores (this is the Environment.ProcessorCount property)...

This same book also recommends the following How to: Create a Task Scheduler that Limits the Degree of Concurrency.

Solution 4

It is likely that you starts too many tasks at the same time.

Each task is potentially a separate thread. CLR assigns independent stack memory to each thread. I assume a typical stack takes 1024Kb for an x64 Windows. Simply by spanning the threads you get 1GB of memory purely for the thread stacks. That doesn't include any heap memory nor large objects heap. Plus, you have other processes that consume memory.

View more solutions

Comments

[Henk Holterman]

I agree with the analysis and the 2nd suggestion (SetMaxThreads) but I doubt the LongRunningOption will help here. It suggests to the scheduler to make more threads (sooner).

[Sam]

@Henk: Good catch, I'll remove that suggestion. The Parallel Programming book recommends it in these sort of scenarios, but it was in a section detailing the Parallel class.

[Samet S.]

All my task are started with LongRunning flag set. I'll try the other advices. Hope it helps.

[Samet S.]

The result of ThreadPool.GetMaxThreads is as follows: Maximum worker threads: 1023, Maximum completion port threads: 1000. Will SetMaxThreads work or going over 1023 will cause the threads to wait in a queue until thread pool threads become available?

[Henk Holterman]

@Samuel: You can try SetMaxThreads(32, 1000) and remove the LongRunning option. Vary a little.

[Henk Holterman]

Too many Threads: Yes, too many Tasks: not so clear. 1000 Tasks should not be extreme (by itself).

[Andrea Nagar]

Is there a way I can monitor is the available number of threads is exhausting? After a couple of days my application crashes with that exception and I want to monitor over time if the available space is filling up. Thanks.