java.lang.OutOfMemoryError while reading file to byte[] array

Question

Is there a cleaner and faster way to do this:

BufferedReader inputReader = new BufferedReader(new InputStreamReader(context.openFileInput("data.txt")));
String inputString;
StringBuilder stringBuffer = new StringBuilder();
while ((inputString = inputReader.readLine()) != null) {
    stringBuffer.append(inputString + "\n");
}
text = stringBuffer.toString();
byte[] data = text.getBytes();

Basically I'm trying to convert a file into byte[], except if the file is large enough then I run into an outofmemory error. I've been looking around SO for a solution, I tried to do this here, and it didn't work. Any help would be appreciated.

Solution 1

Few suggestions:

1. You don't need to create string builder. You can directly read bytes from the file.
2. If you read multiple files, check for those byte[] arrays remaining in memory even when not required.
3. Lastly increase the maximum memory for your java process using -Xmx option.

Solution 2

As we know the size of this file, somewhat half of the memory can be saved by allocating the byte array of the given size directly rather than expanding it:

byte [] data = new byte[ (int) file.length() ];
FileInputStream fin = new FileInputStream(file);
int n = 0;
while ( (n = fin.read(data, n, data.length() - n) ) > 0);

This will avoid allocating unnecessary additional structures. The byte array is only allocated once and has the correct size from beginning. The while loop ensures all data are loaded ( read(byte[], offset, length) may read only part of file but returns the number of bytes read).

Clarification: When the StringBuilder runs out, it allocates a new buffer that is the two times larger than the initial buffer. At this moment, we are using about twice the amount of memory that would be minimally required. In the most degenerate case (one last byte does not fit into some already big buffer), near three times the minimal amount of RAM may be required.

Solution 3

If you haven't enough memory to store there whole file, you can try rethink your algorithm to process file data while reading it, without constructing large byte[] array data.

If you have already tried increase java memory by playing with -Xmx parameter, then there isn't any solution, which will allow you store data in memory, which can not be located there due to its large size.

Comments

[keyser]

There are lots of good thoughts on the issue in that thread.

[eWizardII]

I tried to implement the actual answer, the only problem is then what do I do with mbb? Like is that already in byte []?

[keyser]

It's confusing to name a StringBuilder stringBuffer, since a StringBuffer is a thread safe version of StringBuilder. Just saying.

[Audrius Meškauskas]

When the StringBuilder runs out, it allocates a new buffer. At that point, we have the two buffers, old and new. Hence at at this point we are using two times more memory than is minimally required.

[keyser]

I also thought that the new buffer would have twice the size of the old one (it's certainly bigger, right? :p). The documentation wasn't clear on this.

[Audrius Meškauskas]

Yes, it will probably be the (old_size + 1) * 2 as can be verified in the source code of OpenJDK. Hence near as much as three times more memory than necessary may be required in the most degenerate corner cases.

[keyser]

That's what I thought. This does help promote your solution ;) Thanks for the link.

[user207421]

And? What does he do if there isn't enough memory? Not an answer.

[user207421]

What if the file size doesn't fit to an int? What if the array is still too large for memory? Not an answer.

[user207421]

You mean ByteArrayOutputStream, but it still doesn't solve the problem.

[Audrius Meškauskas]

The task is to read the file into array so the array itself can be defined and fits into memory.

[Vishy]

@EJP Correct. You may have missed ... but it can still mean you run out of memory. In this case you have to either ...

[JayTee]

If there's not enough memory then it is not doable, the requirement as stated in the question is to have a byte array that holds the entire file contents! Yes I agree with your post, should stream and handle in chunks if possible.

[eWizardII]

Thanks a lot and also from the advice, below, basically had the file upload, then clear out the old one, and continue to do this so it didn't become excessively big.

[eWizardII]

I tried to implement this solution currently testing it I think you meant to say fin.read, instead of file.read. Thanks for the help.

[eWizardII]

Thanks, this seems to be working quite well. I will continue testing it though Only thing I might have run into is the problem about it reading only part of the file - but I will see if this is a glitch in my other code.

[Max]

3 is bad advice. There are cases of people increasing the max heap size, which actually causes the OutOfMemoryError. Here is a good example. I have also seen this occur with the Oracle JDK. I think it has to do with how when you increase the max heap size using -Xmx you also are decreasing the native memory available, and FileInputStream uses native memory, although that is just a theory. The -Xmx flag only increases the max heap size, not the "maximum memory for your java process" as you state, which is limited to 4GB for 32-bit Java.