StringBuilder.ToString() throws OutOfMemoryException
Solution 1
What is the reason for the OOM while creating a new string
Because you're running out of memory - or at least, the CLR can't allocate an object with the size you've requested. It's really that simple. If you want to avoid the errors, don't try to create strings that don't fit into memory. Note that even if you have a lot of memory, and even if you're running a 64-bit CLR, there are limits to the size of objects that can be created.
and why it doesn't throw OOM while writing to a file ?
Because you have more disk space than memory.
I'm pretty sure the code isn't exactly as you're describing though. This line would fail to compile:
sw.write(SB.ToString());
... because the method is Write
rather than write
. And if you're actually calling SB.ToString()
, then that's just as likely to fail as str = SB.ToString()
.
It seems more likely that you're actually writing to the file in a streaming fashion, e.g.
using (var writer = File.CreateText(...))
{
for (int i = 0; i < 5000; i++)
{
writer.Write(mytext);
}
}
That way you never need to have huge amounts of text in memory - it just writes it to disk as it goes, possibly with some buffering, but not enough to cause memory issues.
Solution 2
Workaround: Suppose you would want to write a big string stored in StringBuilder to a StreamWriter, I would do a write this way to avoid SB.ToString's OOM exception. But if your OOM exception is due to StringBuilder's content add itself, you should work on that.
public const int CHUNK_STRING_LENGTH = 30000;
while (SB.Length > CHUNK_STRING_LENGTH )
{
sw.Write(SB.ToString(0, CHUNK_STRING_LENGTH ));
SB.Remove(0, CHUNK_STRING_LENGTH );
}
sw.Write(SB);
Solution 3
You have to remember that strings in .NET are stored in memory in 16-bit unicode. This means string of length 132370292 will reqire 260MB of RAM.
Furthermore, while executing
string str = SB.ToString();
you are creating a COPY of your string (another 260MB).
Keep in mind that each process have its own RAM limit so OutOfMemoryException can be thrown even if you have some free RAM left.
Solution 4
Might help someone , if your logic needs large objects then you can change your application to 64bit and also
change your app.config by adding this section
<runtime>
<gcAllowVeryLargeObjects enabled="true" />
</runtime>
gcAllowVeryLargeObjects On 64-bit platforms, enables arrays that are greater than 2 gigabytes (GB) in total size.
Related videos on Youtube
Thiru
Updated on October 28, 2021Comments
-
Thiru over 2 years
I have a created a
StringBuilder
of length "132370292", when I try to get the string using theToString()
method it throwsOutOfMemoryException
.StringBuilder SB = new StringBuilder(); for(int i =0; i<=5000; i++) { SB.Append("Some Junk Data for testing. My Actual Data is created from different sources by Appending to the String Builder."); } try { string str = SB.ToString(); // Throws OOM mostly Console.WriteLine("String Created Successfully"); } catch(OutOfMemoryException ex) { StreamWriter sw = new StreamWriter(@"c:\memo.txt", true); sw.Write(SB.ToString()); //Always writes to the file without any error Console.WriteLine("Written to File Successfully"); }
What is the reason for the OOM while creating a new string and why it doesn't throw OOM while writing to a file?
Machine Details: 64-bit, Windows-7, 2GB RAM, .NET version 2.0
-
Karl-Johan Sjögren almost 10 yearsWhats the content of
mytext
? And why are you writing to aStringBuilder
if and then to a stream? Why no the stream directly using aStringWriter
? -
Syroot almost 10 yearsWhy that exaggerated length? You just reserved 126 MB of memory.
-
Lasse V. Karlsen almost 10 yearsCan you post the actual code from your program?
-
Thiru almost 10 yearsMy Actual code is very different than this, I'm creating the StringBuilder from different sources by Appending the values, and the final Length of the SB is "132370282". SB.ToString() works fine some times and fails some times.
-
Marc Gravell almost 10 yearsThat's 252MB; why would you want a 252MB string? This seems an insanely bad idea - you should really be writing to the writer (
sw
) cumulatively - not building the entire thing in memory. (/cc @DebugErr just to note that this is 252MB, not 126MB) -
Marc Gravell almost 10 yearsAs an unrelated question: why would you install a 64-bit OS on a system with 2GB of memory?
-
Thiru almost 10 yearsI was using 4GB on 64-bit machine. Since I was not able to reproduce the error, I wanted to test this scenario on extreme conditions(that's 2GB VM), and found this.
-
Atur almost 10 yearspossible duplicate of C# Stringbuilder System.OutOfMemoryException
-
-
Lasse V. Karlsen almost 10 yearsIf he calls
ToString
in both cases, wouldn't the fact that he's writing to disk be an irrelevant detail here? -
Alexei Levenkov almost 10 yearsBoth probably off - 64bit process would have plenty of memory, but likely runs as 32bit and hits address space fragmentation; it likely succeeds to write to disk because it already converted result to string once and previous call did not fail (based on sample code)
-
Jon Skeet almost 10 years@AlexeiLevenkov: The sample code isn't the real code anyway, but even in a 64-bit CLR, object size is limited. Also, if you call
ToString()
on aStringBuilder
twice, it still creates two strings - at least in the tests I've just run... (Given the use of .NET 2.0, that might not have been the case back then...) -
Lasse V. Karlsen almost 10 years@JonSkeet Agreed, didn't spot the lowercase w, and since this means it's obviously not the correct code, all bets are off.
-
Marc Gravell almost 10 yearseach
char
is 2 bytes, so you need to double all those numbers -
Thiru almost 10 years@JonSkeet I believe that it may be because of the GC, i.e, when the code is executed in the try block(in the actual app) the availability of free(contiguous) memory may cause OOM and the virtual memory reaches a critical situation and this invokes the GC implicitly before the catch block, GC may free certain amount of memory which is sufficient when the code runs in catch block.
-
Jon Skeet almost 10 years@Thiru: Oh, there's a catch block suddenly, is there? It's a good idea to indicate when you've changed a question significantly after answers have been posted. The GC should collect before the OOM is thrown anyway. I've seen rare occasions when it doesn't, but it's been a long time since I've seen that. Can you reproduce this in a short but complete program? The answer is still the same though, basically - don't try to create such huge strings.
-
goamn about 8 yearsThis was a helpful solution for me in getting to the final solution. I just want to point out 3 errors. First, it should be (SB.Length > 0) as the condition otherwise you will lose the last few thousand pieces of data. Second, you may be on your final run, which means you can't take the full chunk length, and instead you should check for this case and use the remaining length (to avoid an OutOfArgumentRangeException). Third, I believe the last line is surplus and an accident.
-
Titus about 8 yearsYou probably have to think the way it is coded, rather than the way you want the code to be. I believe my above code works well and encourage you to debug and verify it.
-
goamn about 8 yearsYes you are right, it is correct. And it looks better to do it your way. I'm sorry I doubted you :). Although I would name the variables better (stringBuilder, make a "buffer" variable for the ToString).