Exception of type 'System.OutOfMemoryException' was thrown

Question

I have suddenly been getting the memory exception errors for two programs running on different machines and even though it seems there is enough memory, it still shows up. I am creating multiple threads in the program so not sure if this appropriate for this forum but could it be something else related to visual studio or is it definitely memory issue.The one program runs on my desktop using visual studio 2008 with 2 gb ram. The other is running on a windows 2003 server with 4 GB ram using visual basic 2008 express. Now the module takes a large xml file that is read into a string and then split and stored in a string array. Now the number of chunks can be upto 10000. Now I know this is big, but I have been running this for over a month now and never had the issue. The only other possible related issue I noticed was that I was running out of space on my harddrive but that was quickly solved with a cleanup. Oh yes the processor for my machine is a duo core set at 2.13 GHZ. It is a console program that makes multiple webrequests but the memory problem arises in one specific module only as I explained above.

Public Shared Function textLoad(ByVal _html As String) As Boolean
    Try 
            //_html is the filestream that was read in

        Dim defaultHeading = "xmlns:gnip=""http://www.gnip.com/schemas/2010"" xmlns=""http://www.w3.org/2005/Atom"""
        Dim header_of_xml As String = "<?xml version=""1.0"" encoding=""utf-8""?>" & vbNewLine & "<entry " & defaultHeading & ">"
        Dim footer_of_xml As String = "</entry>"
        Dim entry As String = String.Empty
        Dim xrs As XmlReaderSettings = New XmlReaderSettings()
        Dim dupeArray As New ArrayList
        Dim stringSplitter() As String = {"</entry>"}

        //split the file content based on the closing entry tag
        sampleResults = Nothing
        sampleResults = _html.Split(stringSplitter, StringSplitOptions.RemoveEmptyEntries)
        entryResults.Clear()

        If getEntryId(sampleResults) Then
           // the following loops seem clumsy but I wanted to dedupe the lists to //ensure that I am not adding duplicate entries and I do this by going to the getEntryID //method and extracting the ids and then comparing them below 

            For i As Integer = 0 To sampleResults.Count - 1
                For j As Integer = 0 To idList.Count - 1
                    If sampleResults(i).Contains(idList.Item(j)) AndAlso Not dupeArray.Contains(idList.Item(j)) Then
                        dupeArray.Add(idList.Item(j))
                        entry = sampleResults(i)

I did look at taskmanager for identifying resources used by this program and this is what is going on:

Parser.exe CPU = 34 MEM usage = 349,500 K

nothing else intensive is running

Edit _-

Figured out exactly where the problem originates:

**sampleResults = _html.Split(stringSplitter, StringSplitOptions.RemoveEmptyEntries)**

Can anyone notice anything wrong with this??

Solution 1

The Split method allocates memory for a string object for each element in the returned array. If you're going to run into a memory issue, it's going to be on that line. Since you're allocating memory for potentially 10,000 large strings at once, on a potentially fragmented heap, it's not surprising that you might come to a point where it just can't find enough contiguous space to allocate for the next string (which would result in the exception you are getting). Do you really need all of those strings at once? Or can you do something like this loop:

1. Read one line at a time (or into a buffer) to build a string.
2. Stop when you've found the first "chunk".
3. Get the info you need from the chunk. (I'm assuming you don't need the whole thing, you're just interested in getting something out of it?).
4. Continue reading the html (Restart at #1), reading it into the same string variable so that the first chunk can get garbage collected.

If you can implement your solution that way, you're issue should disappear.

Solution 2

Note that an OOM doesn't mean you're out of physical RAM, but that you can't fit a new memory block of the desired size into the address space.

There are a number of possible causes for that

• Heap fragmentation
• the 2GB limit of a 32 bit process
• a maximum memory size set in the config

And then there are fake OOM exceptions. For example GDI+ returns out-of-memory errors for a wide class of problems including invalid parameters. These are turned into an OOM exception too.

Solution 3

Have you tried profiling the memory usage as your application is running using the .NET Performance Counters? You can fire up the Performance Monitor (in Windows 2008 R2 it's under Administrative Tools > Performance Monitor) and add counters under .NET Memory for Gen 0 Heap Size, Gen 1 Heap Size, Gen 2 Heap Size and Large Object Heap size. As your application runs, you can use these counters to see where the memory allocations are.

Since you are working with strings, it's important to take into account that strings over 85,000 bytes will get allocated on the Large Object Heap. All of the other heaps are compacted by the garbage collector, but for performance reasons the Large Object Heap is never compacted. That basically means that objects will be cleared from the LOH by the GC as they are disposed, but the remaining objects are never moved from their original locations on the LOH. This behavior of the CLR can lead to LOH fragmentation and eventually OOM exceptions.

Here's a simple example for how LOH framgentation can lead to OOM exceptions. Say we have 100 total units available on the LOH, and we allocate 10 units and then 5 units, leaving us with 85 left. The CLR must allocate the memory continuously, so in our example right now we're taking up the first 15 units of the LOH. We try to allocate 86 units for a really big object, and we get an OOM exception. Realizing that we don't have enough space, we clear the 10 units we first allocated, leaving us with 95 total units avaible. We try again to allocate 86 units, but since the CLR doesn't compact the LOH and the CLR is asking for 86 continuous units, we'll still get OOM exception. We won't be able to allocate our 86 units until we clear the second 5 units as well. This is an oversimplified example, but if you Google LOH fragmentation or check this article you can get a more in depth explanation.

I can't say for certain if this is the problem you are running into, but I would definitely check out the Performance Monitor to see where your memory is being allocated. If you know for a fact you're allocating strings or other objects in excess of 85,000 bytes, then try to split them up further so they stay off the LOH.

Solution 4

I've encoutered this very same problem. Thinking about moving to 64 bit OS. I have some tricks to delay this exception:

1. the most important one is that when heavily working with strings (especially long) use ref to transfer from one method to another. It segnificantly reduces memory and performance.

2. You can use AppDomain to store data. this doubles your memory capasity.

3. Whenever you can - instead of creating objects and strings (WebRequests/ Response) use the same memoryStream or buffer all over again. Allocate one with the max size possile (even *2 the max expected size). the allocation (using new) of Stream/StringBuilder/strings/classes/Buffers Suffocate the heap.

View more solutions

Comments

[vbNewbie]

so do you think this can be remedied by changing the code and using a more effective, effecient logic

[vbNewbie]

Ok I figured out exactly where the problem was occurring - in the following line sampleResults = _html.split(stringsplitter,stringsplitoptions.removeemptyent‌​ryies)

[bensiu]

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes.

[Chris Bennet]

The link is now broken. @bensiu's warning was prescient.