Allocating more than 1,000 MB of memory in 32-bit .NET process

Question

I am wondering why I'm not able to allocate more that 1,000 MB of memory in my 32-bit .NET process. The following mini application throws an OutOfMemoryException after having allocated 1,000 MB. Why 1,000 MB, and not say 1.8 GB? Is there a process-wide setting I could change?

static void Main(string[] args)
{
    ArrayList list = new ArrayList();
    int i = 0;
    while (true)
    {
        list.Add(new byte[1024 * 1024 * 10]); // 10 MB
        i += 10;
        Console.WriteLine(i);
    }
}

PS: Garbage collecting does not help.

Edit, to clarify what I want: I have written a server application which deals with very large amounts of data before writing to database/disk. Instead of creating temporary files for everything, I have written an in-memory cache, which makes the whole thing super-fast. But memory is limited, and so I tried to find out what the limits are. And wondered why my small test program threw the OutOfMemoryException after exactly 1,000 MB.

Solution 1

The virtual address space limit of a Win32 process is 1.5GB (not entirely true). Additionally in the .NET frameworks there is a limiter to the % of memory a .NET process can consume. The machine.config has a processModel element with an attribute memoryLimit which is the % of available memory a process can consume. The default value is 60%.

If the machine you're running on has 2GB of memory or you haven't enabled the /3GB switch in your BOOT.INI then you're going to get ~1.3GB of memory per process.

I can't find the KB article but if I remember correctly .NET 1.x cannot address beyond the 1.5GB (1.8GB?) limit regardless of your settings.

http://blogs.msdn.com/tmarq/archive/2007/06/25/some-history-on-the-asp-net-cache-memory-limits.aspx http://social.msdn.microsoft.com/Forums/en-US/clr/thread/c50ea343-b41b-467d-a457-c5a735e4dfff http://www.guidanceshare.com/wiki/ASP.NET_1.1_Performance_Guidelines_-_Caching#Configure_the_Memory_Limit

Solution 2

Having enormous blocks of memory is never a good idea, even in 64bit. You get big problems with contiguous memory and fragmentation.

The problem here is finding a contiguous block. You could try enabling 3gb mode (which might help it find a few more bytes) but I really advise against it. The answers here are:

• use less memory
• use a database/file system
• use x64

You might also want to read Eric Lippert's blog (he seems to have a blog entry for every common .NET question...)

Solution 3

I have recently been doing extensive profiling around memory limits in .NET on a 32bit process. We all get bombarded by the idea that we can allocate up to 2.4GB (2^31) in a .NET application but unfortunately this is not true :(. The application process has that much space to use and the operating system does a great job managing it for us, however, .NET itself seems to have its own overhead which accounts for approximately 600-800MB for typical real world applications that push the memory limit. This means that as soon as you allocate an array of integers that takes about 1.4GB, you should expect to see an OutOfMemoryException().

Obviously in 64bit, this limit occurs way later (let's chat in 5 years :)), but the general size of everything in memory also grows (I am finding it's ~1.7 to ~2 times) because of the increased word size.

What I know for sure is that the Virtual Memory idea from the operating system definitely does NOT give you virtually endless allocation space within one process. It is only there so that the full 2.4GB is addressable to all the (many) applications running at one time.

I hope this insight helps somewhat.

I originally answered something related here (I am still a newby so am not sure how I am supposed to do these links):

Is there a memory limit for a single .NET process

Comments

[Eric Lippert]

Johannes, why would having more address space NOT make it easier to find more contiguous address space??? I am completely confused by your comment. Can you explain?

[David]

Thank you, that gives me an idea. However, the links are not really helpful.

[BlackBear]

I've edited your answer to fix the code formatting. Next time you need to post code paste it, select it and press CTRL+K or the {} icon, it will be automatically formatted

[ezolotko]

Only the server CLR uses the memoryLimit configuration. The workstation CLR always uses as much memory as it can get.

[MyItchyChin]

@ezolotko - I was never aware there was a difference.

[John B. Lambe]

This totally misses the point of the question. The infinite loop allocates memory until it fails, thereby determining the maximum amount of memory the process can consume. Your code doesn't compile because it tries to assign a byte array to a byte. Also, neither of those implements IDisposable. And if it did free the memory on each iteration, it would break the test (it would eventually report about (2^31)/10, but that number would have nothing to do with how much memory it can allocate).