Android Reading from an Input stream efficiently
Solution 1
The problem in your code is that it's creating lots of heavy String
objects, copying their contents and performing operations on them. Instead, you should use StringBuilder
to avoid creating new String
objects on each append and to avoid copying the char arrays. The implementation for your case would be something like this:
BufferedReader r = new BufferedReader(new InputStreamReader(inputStream));
StringBuilder total = new StringBuilder();
for (String line; (line = r.readLine()) != null; ) {
total.append(line).append('\n');
}
You can now use total
without converting it to String
, but if you need the result as a String
, simply add:
String result = total.toString();
I'll try to explain it better...
-
a += b
(ora = a + b
), wherea
andb
are Strings, copies the contents of botha
andb
to a new object (note that you are also copyinga
, which contains the accumulatedString
), and you are doing those copies on each iteration. -
a.append(b)
, wherea
is aStringBuilder
, directly appendsb
contents toa
, so you don't copy the accumulated string at each iteration.
Solution 2
Have you tried the built in method to convert a stream to a string? It's part of the Apache Commons library (org.apache.commons.io.IOUtils).
Then your code would be this one line:
String total = IOUtils.toString(inputStream);
The documentation for it can be found here: http://commons.apache.org/io/api-1.4/org/apache/commons/io/IOUtils.html#toString%28java.io.InputStream%29
The Apache Commons IO library can be downloaded from here: http://commons.apache.org/io/download_io.cgi
Solution 3
Another possibility with Guava:
dependency: compile 'com.google.guava:guava:11.0.2'
import com.google.common.io.ByteStreams;
...
String total = new String(ByteStreams.toByteArray(inputStream ));
Solution 4
I believe this is efficient enough... To get a String from an InputStream, I'd call the following method:
public static String getStringFromInputStream(InputStream stream) throws IOException
{
int n = 0;
char[] buffer = new char[1024 * 4];
InputStreamReader reader = new InputStreamReader(stream, "UTF8");
StringWriter writer = new StringWriter();
while (-1 != (n = reader.read(buffer))) writer.write(buffer, 0, n);
return writer.toString();
}
I always use UTF-8. You could, of course, set charset as an argument, besides InputStream.
Solution 5
What about this. Seems to give better performance.
byte[] bytes = new byte[1000];
StringBuilder x = new StringBuilder();
int numRead = 0;
while ((numRead = is.read(bytes)) >= 0) {
x.append(new String(bytes, 0, numRead));
}
Edit: Actually this sort of encompasses both steelbytes and Maurice Perry's
Related videos on Youtube
RenegadeAndy
Updated on May 22, 2021Comments
-
RenegadeAndy almost 3 years
I am making an HTTP get request to a website for an android application I am making.
I am using a DefaultHttpClient and using HttpGet to issue the request. I get the entity response and from this obtain an InputStream object for getting the html of the page.
I then cycle through the reply doing as follows:
BufferedReader r = new BufferedReader(new InputStreamReader(inputStream)); String x = ""; x = r.readLine(); String total = ""; while(x!= null){ total += x; x = r.readLine(); }
However this is horrendously slow.
Is this inefficient? I'm not loading a big web page - www.cokezone.co.uk so the file size is not big. Is there a better way to do this?
Thanks
Andy
-
Mike76 almost 5 yearsUnless you are actually parsing the lines it does not make a lot of sense to read line by line. I would rather read char by char via fixed size buffers: gist.github.com/fkirc/a231c817d582e114e791b77bb33e30e9
-
-
RenegadeAndy about 14 yearsIts not that long to be honest - its the page source of the website www.cokezone.co.uk - so really not that big. Definitely less than 100kb.
-
RenegadeAndy about 14 yearsDoes anybody have any other ideas on how this could be made more efficient - or if this is even inefficient!? If the latter is true - why does it take SO long? I dont believe the connection is to blame.
-
RenegadeAndy about 14 yearsHmm. so it would be like this: int offset = 5000; Byte[] bArr = new Byte[100]; Byte[] total = Byte[5000]; while(InputStream.available){ offset = InputStream.read(bArr,offset,100); for(int i=0;i<offset;i++){ total[i] = bArr[i]; } bArr = new Byte[100]; } Is that really more efficient - or have i written it badly! Please give an example!
-
RenegadeAndy about 14 yearsThe problem is - I dont know the size of the thing im reading before i start - so might need some form of array growing as well. Inless you can query an InputStream or URL through http to find out how big the thing im retrieving is to optimise the size of the byte array. I have to be efficient as its on a mobile device which is the main problem! However thanks for that idea - Will give it a shot tonight and let you know how it handles in terms of performance gain!
-
Adrian about 14 yearsI don't think the size of the incoming stream is that important. The above code reads 1000 bytes at a time but you could increase/decrease that size. With my testing it didn't make that much difference weather I used 1000/10000 bytes. That was just a simple Java app though. It may be more important on a mobile device.
-
SteelBytes about 14 yearsno no no no, I mean simply { byte total[] = new [instrm.available()]; instrm.read(total,0,total.length); } and if you then needed it as a String, use { String asString = String(total,0,total.length,"utf-8"); // assume utf8 :-) }
-
Makotosan over 13 yearsI realize this is a late response, but just now happened to stumble across this via a Google search.
-
Charles Ma over 13 yearsThe android API doesn't include IOUtils
-
Makotosan over 13 yearsRight, which is why I mentioned the external library that has it. I added the library to my Android project and it made it easy to read from streams.
-
Jacob Nordfalk over 12 yearsYou could end up with an Unicode entity that is chopped into two subsequent reads. Better to read until some kind of boundary character, like \n, which is exactly what BufferedReader does.
-
safari over 12 yearswhere can I download this, and how did you import that into your android project?
-
Admin over 12 yearsExactly! This is also explained in Java Effective Item 51. I used to experiment with profiling and using StringBuilder instead of String saves a lot of processing time.
-
dokkaebi over 12 yearsFor bonus points, provide an initial capacity to avoid reallocations as the StringBuilder fills up:
StringBuilder total = new StringBuilder(inputStream.available());
-
Nathan Schwermann almost 12 yearsDoesn't this cut out new line characters?
-
botbot almost 12 yearsdon't forget to wrap the while up in try / catch like this: try { while ((line = r.readLine()) != null) { total.append(line); } } catch (IOException e) { Log.i(tag, "problem with readline in inputStreamToString function"); }
-
Matti Virkkunen over 11 years@botbot: Logging and ignoring an exception is not much better than just ignoring the exception...
-
rds about 11 yearsAre you seriously saying the dalvik compiler doesn't optimize a loop of String concatenations?
-
rds about 11 yearsAnd you are right! The compiler replaces the concatenation with a
StringBuilder
only outside of a loop. Same thing with Oracle JDK 6, by the way :( -
Edward Brey over 10 yearsIt's amazing that Android doesn't have a built-in stream-to-string conversion. Having every code snippet on the web and app on the planet re-implement a
readline
loop is ridiculous. That pattern should have died with pea green in the 70s. -
pjw over 10 years@schwiz: yes, I had to use ... total.append(line+"\n");
-
nano over 10 yearsShouldn't the bufferreader be closed at the end of that block?
-
KMI about 10 years@Makotosan :Can you guide how to use this library?Am bit confused
-
B. Clay Shannon-B. Crow Raven about 10 yearsIf you have to download it, I wouldn't call it "built in"; nevertheless, I just downloaded it, and will give it a go.
-
SteveGSD almost 10 yearsThis is much faster than the above and accepted answers. How do you use "Reader" and "Stream" on android?
-
Budimir Grom over 9 years@schwiz: Depends, but I provided an answer which is much better, in this regard and regarding efficiency: stackoverflow.com/a/25343605/3948809
-
Slim_user71169 over 8 yearsThis is not effeciency. Cost about 10s for 20kb of string
-
Jorge Arimany over 8 yearsI would not consider readLine efficient, depending on data this can lead to a OutOfMemory exception
-
Akhil Dad over 8 yearsCan you explain why it would be faster?
-
heiner over 8 yearsIt doesn't scan the input for newline characters, but just reads chunks of 1024 bytes. I'm not arguing this will make any practical difference.
-
Akhil Dad over 8 yearsany comments over @Ronald answer? He is doing the same but for a larger chunk equal to inputStream size. Also how different it is if I scan char array rather than byte array as Nikola answer? Actually I just wanted to know which approach is best in which case? Also readLine removes \n and \r but I seen even google io app code they are using readline
-
mustaq over 7 yearsWhy use append('\n') ?
-
winklerrr over 6 years@dokkaebi I quote from
InputStream.available()
: Note that while some implementations of InputStream will return the total number of bytes in the stream, many will not. It is never correct to use the return value of this method to allocate a buffer intended to hold all data in this stream. -
dokkaebi over 6 years@winklerrr Fair point. The argument to the StringBuilder constructor is just an optimization, though. You won't end up overflowing a buffer if the value is incorrect. Still a good thing to be aware of about InputStream.available.
-
minhazur almost 6 yearsThis is deprecated and causes exception on android 4.1.1. Will suggest to go with buffer reader approach.