StringBuilder out of memory error when working with large strings in java

java memory text out-of-memory heap-memory

19,283

Solution 1

Apparently your application doen't have enough memory for complete the operation. So you need to specify memory flags to your virtual machine. You can try the following:

 java -Xms256m -Xmx512m YourApp

Where:

Xms minimun memory allocated by your program at startup (in the example 256 MB)
Xmx maximun memory allocated by your program if it need more (in the example 512 MB)

Solution 2

Well one of the things that might happen is the fact that in Java, java.lang.String gets special treatment. Strings are immutable, and thus the JVM places each String object in a pool .The role of this pool, amongs others is the fact that is used as a sort of "cache" where if you create multiple String instances having the actual "text" value identical, the same instance will be reused from the pool. This way creating what seems to be a big number of String object instances with the same text inside will infact revert to having very few actual String instances in memory. On the other hand, if you use the same text to initialize multiple StringBuilder instances, those will actually be separate instances (containing the same text), and thus occupying more memory.

On the other hand, concatenating Strings (such as String c = "a"+"b";) will in fact create more object instances that if you do it with StribgBuilder (such as new StringBuilder("a").append("b");).

19,283

Author by

montag

Updated on June 04, 2022

Comments

montag almost 2 years

I went from String test += str; where test grew exponentially large with thousands and thousands of characters. It took 45 minutes to run, probably from creating large strings and deleting garbage. I then staggered the input like this which brought it to 30 seconds.

This seems like the cheap way to do it, but it worked well:

  if (secondDump.length() > 50)
  {
     intermedDump = intermedDump + secondDump;
     secondDump = "";
  }      

  if (intermedDump.length() > 100)
  {
     thirdDump = thirdDump + intermedDump;
     intermedDump = "";
  }
  if (thirdDump.length() > 500)
  {
     fourthDump = fourthDump + thirdDump;
     thirdDump = "";
  }
  if (fourthDump.length() > 1000)
  {
     fifthDump = fifthDump + fourthDump;
     fourthDump = "";
  }
  //with just this and not sixth.  Runtime>>>> : 77343
  if (fifthDump.length() > 5000)
  {
     sixthDump = sixthDump + fifthDump;
     fifthDump = "";
  }
  //with just this.  Runtime>>>> : 35903Runtime>>>> : 33780
  if (sixthDump.length() > 10000)
  {
     fillerDump = fillerDump + sixthDump;
     sixthDump = "";
  }

I then discovered that StringBuilder exists, and I've been trying to use it since, replacing all string operations with it.

The problem is, I keep getting an java.lang.OutOfMemoryError with a java memory heap overflow. I think the string is just too long to store in memory as a StringBuilder object, because it makes about 1/50th of the progress that my previous code did before crashing with an out of memory error. It's only working with maybe under a thousand characters.

Why can a string hold the entire output and this can't come close? Also, if I append text to JTextPane, how much memory does that need? If I dump the StringBuilder contents to JTextpane and keep appending and clearing StringBuilder that doesn't seem to work either.

Here is the existing code. Page is just an object being passed:

protected void concatPlates(page PageA) throws IOException
{
   if (backend.isFirstPage == false)
   {
      frontend.fillOutputPane("\n                                 " +
         "                                               \n", PageA);
      frontend.fillOutputPane("                                 " +
         "                                               \n", PageA);
      frontend.fillOutputPane("                                 " +
         "                                               \n", PageA);
   }
   for (int i = 0; i < PLATELEN-1; i++)
   {

      if (arrLeftCol[i].length() == 0)
      {
         /////////////////////////////////////////////////////////
         /////////////////////////////////////////////////////////
         frontend.fillOutputPane(arrLeftCol[i].append(
           arrRightCol[i]));
      }
     else
     {
        PageA.tempStrBuff = new StringBuilder(arrLeftCol[i].substring(0,40));
        frontend.fillOutputPane(PageA.tempStrBuff.append(arrRightCol[i]));
     }
     arrLeftCol[i].append("");
     arrRightCol[i].append("");

     backend.isFirstPage = false;
  }
}


//this is the frontend class
public static void fillOutputPane(String s, page PageA)
{
   fillOutputPane(PageA.getStrBuf());
}
public static void fillOutputPane(StringBuilder stringBuild)
{
  try
  {
     str.append(stringBuild);
  }
  catch (java.lang.OutOfMemoryError e)
  {
     System.out.println((str.length() * 16) /8);
     //System.out.println(str);
     System.out.println("out of memory error");
     System.exit(0);
  }
}

Here is the error:

Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuilder.append(Unknown Source)
at java.lang.StringBuilder.append(Unknown Source)
at backend.fill(backend.java:603)
at frontend$openL.actionPerformed(frontend.java:191)
at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
at javax.swing.AbstractButton.doClick(Unknown Source)
at javax.swing.plaf.basic.BasicMenuItemUI.doClick(Unknown Source)
at javax.swing.plaf.basic.BasicMenuItemUI$Handler.mouseReleased(Unknown Source)
at java.awt.Component.processMouseEvent(Unknown Source)
at javax.swing.JComponent.processMouseEvent(Unknown Source)
at java.awt.Component.processEvent(Unknown Source)
at java.awt.Container.processEvent(Unknown Source)
at java.awt.Component.dispatchEventImpl(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Window.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
at java.awt.EventQueue.access$000(Unknown Source)
at java.awt.EventQueue$1.run(Unknown Source)
at java.awt.EventQueue$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)

I think this is what a stack trace is:

java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(Unknown Source)
at frontend.fillOutputPane(frontend.java:385)
at page.concatPlates(page.java:105)
at backend.setPlate(backend.java:77)
at backend.fill(backend.java:257)
at frontend$openL.actionPerformed(frontend.java:191)
at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
at javax.swing.AbstractButton.doClick(Unknown Source)
at javax.swing.plaf.basic.BasicMenuItemUI.doClick(Unknown Source)
at javax.swing.plaf.basic.BasicMenuItemUI$Handler.mouseReleased(Unknown Source)
at java.awt.Component.processMouseEvent(Unknown Source)
at javax.swing.JComponent.processMouseEvent(Unknown Source)
at java.awt.Component.processEvent(Unknown Source)
at java.awt.Container.processEvent(Unknown Source)
at java.awt.Component.dispatchEventImpl(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Window.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
at java.awt.EventQueue.access$000(Unknown Source)
at java.awt.EventQueue$1.run(Unknown Source)
at java.awt.EventQueue$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
at java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue$2.run(Unknown Source)
at java.awt.EventQueue$2.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)81240560
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)

Shivan Dragon over 12 years

That's sort of a quick fix solution that doesn't really scale or explain how using String vs StringBuilder actually impacts memory consumption.
Puce over 12 years

That's not considered a quick fix - the more data you want to keep in memory the more memory you need.
Puce over 12 years

Althoug StringBuilder might need a bit more memory than a single String object as it keeps open some free slots for new data.
montag over 12 years

I'm a novice and this is my commercial application. Where would you put the code above if you were in eclipse? Just in the run config in args?
montag over 12 years

I see. So do you think this is what could be causing the out of memory error? I didn't think a single StringBuilder object with under a thousand characters would use that much memory.
Puce over 12 years

Yes, in the Arguments tab -> VM arguments
montag over 12 years

So I just put -Xms256m -Xmx512m in VM arguments? It gets a little farther (maybe 1.1 times farther) and then crashes.
montag over 12 years

I just realized it's probably well over a thousand characters.
davidfrancis over 12 years

Can you debug it with Eclipse to find out how long the string is?
davidfrancis over 12 years

Or just add some System.out.println's to tell you how long each string is at various points. Not sure what your code is doing so can't help more than that. Good luck though!
montag over 12 years

Uhh.. yeah my estimate was wrong. When the program crashes, the StringBuilder length is 81240560 characters.
Ernesto Campohermoso over 12 years

You need to check how many memory you program need. The parameter provided are just a start point. You can try with other parameters but remembe let to the OS enough memory for work. By example i f you have 4GB of RAM you can test with -Xmx1024M
montag over 12 years

I did. It gets slightly further and then crashes again.
montag over 12 years

What would that be in bytes? I read it could be up to 4 bytes per char, so that's 324962240 bytes which is 309.65065 MBs.
montag over 12 years

I read it could be up to 4 bytes per char in the string, so that's 324962240 bytes which is 309.65065 MBs. And that was when it crashed.
Shivan Dragon over 12 years

You can do "the long string".getBytes().length to get the (almost exact) bytes size of your string
Ernesto Campohermoso over 12 years

I think you need re-think your algorithm for process the work that you need. What are you trying to do ?
montag over 12 years

I'm reading data that is specifying well locations on a 96 well plate like this 2(3) and there are hundreds of these locations with 4 wells per page. And I'm drawing the plates with characters and this massive string prints at the end of processing.
Ernesto Campohermoso over 12 years

Maybe you can store partially the content of processing in files. Is just an idea.
montag over 12 years

Is there a way to figure out what is using the most memory in debug mode or something?
montag over 12 years

Is it common in mainstream programs to do this when working with large amounts of data?
Shivan Dragon over 12 years

Yes, but not with debug. Install VisualVM (google it, it's an Oracle tool). Open it, and while your Java code is running, start a Sampler on it on the memory consumption. It will instrument your classes such that it shows what classes are instantiated, the object count, and the stacktrace leading to those instances. From this you can deduce who's creating the most memory hungry instances