Out of memory error while parsing a large JSON using Jackson library on Android

java android json jackson out-of-memory

16,639

Solution 1

With 7-8 megs of JSON, tree model that you are using will typically use 20 - 50 megs of memory (dom models are 3-5x as big, both for XML and JSON). There isn't much you can do about that, regardless of library used: they all build trees using Lists and Maps, which is a heavy-weight way of doing it.

Instead you should consider using Plain Old Java Objects (POJOs), which will use much less memory. For this you need to model POJO that matches your JSON structure; without knowing structure I can't give an example (if you add sample on question, I can), but code to parse is similar to GSON code referenced by another answer:

MyValue value = mapper.readValue(json, MyValue.class);

This will work on Jackson as well as many other Java JSON libs (Gson, Genson at least), and will also be faster method to use. JSON trees are inherently expensive and heavy-weight, and not to be used for multi-megatbyte content.

Finally, if your input consists of a sequence of items, there are even better ways to slice it up (which can be done regardless of whether individual items would be JsonNodes or POJOs!). But I don't know if your content is like that.

Solution 2

We use here the gson lib, and with the code above we can get files larger than 50Mb without problem:

public static <T extends Object> T readFile(String caminho_arquivo, Type type) {

    GsonBuilder gson_builder = new GsonBuilder();

    final SimpleDateFormat sdf_date     = new SimpleDateFormat("yyyy-MM-dd");
    final SimpleDateFormat sdf_datetime = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");

    gson_builder.registerTypeAdapter(Date.class, new JsonDeserializer<Date>(){

        @Override
        public Date deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) throws JsonParseException {

            try {
                if (json.getAsJsonPrimitive().getAsString().length() == 10)
                    return sdf_date.parse(json.getAsJsonPrimitive().getAsString());
                else
                    return sdf_datetime.parse(json.getAsJsonPrimitive().getAsString());

            } catch (ParseException e) {
                Log.e("JSON", "Erro na deserialização de datas no JSON: " + json.getAsJsonPrimitive().getAsString());
                return null;
            }
        }

    });

    Gson gson = gson_builder.create();

    File fileJSON = new File(caminho_arquivo);

    FileReader reader = null;

    try {
        reader = new FileReader(fileJSON);

        return gson.fromJson(reader, type);

    } catch (FileNotFoundException e) {
        e.printStackTrace();
    } finally {
        try {
            reader.close();

            if (fileJSON.exists())
                fileJSON.delete();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    return null;
}

Try this lib, that's a good one, with the jackson we use only in the server side, because jackson is more slow in the Android than gson, at least in our test.

Solution 3

Try and use:

JsonNode rootParser = mapper.readTree(is);

instead.

16,639

Author by

Ramit

Updated on June 11, 2022

Comments

Ramit almost 2 years

I am using Jackson library to parse a large JSON response from server. Size of the json is around 7-8 mb.

I am getting the outOfMemoryError on this piece of code:

ObjectMapper mapper = new ObjectMapper();
JsonNode rootParser = mapper.readValue(is, JsonNode.class);

and this is the exception that I am getting:

    01-14 13:13:20.103: E/AndroidRuntime(25468): FATAL EXCEPTION: Thread-13
    01-14 13:13:20.103: E/AndroidRuntime(25468): java.lang.OutOfMemoryError
    01-14 13:13:20.103: E/AndroidRuntime(25468): at java.util.ArrayList.add(ArrayList.java:123)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.node.ArrayNode._add(ArrayNode.java:722)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.node.ArrayNode.add(ArrayNode.java:203)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:224)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:200)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:224)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:200)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:224)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:200)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:197)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:224)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:200)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:197)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:58)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2732)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1909)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at com.sarla.smartglance.communication.JsonDecoder.decodeResponse(JsonDecoder.java:87)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at com.sarla.smartglance.communication.JsonDecoder.decode(JsonDecoder.java:68)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at com.sarla.smartglance.communication.MHttpManager$1.run(MHttpManager.java:86)

I tried everything but couldn't find any solution to parse such a large amount of data on android.

fge over 11 years

Have you tried to just use .readTree() instead of reading into a JsonNode?
Andrei over 11 years

Give gson a go.
Siddharth over 11 years

How big is the data on the ArrayList, sounds really big.
StaxMan over 11 years

Gson won't help -- tree model is heavy-weight and about same size on different libs.

StaxMan over 11 years

Above code is buggy -- does not specify character encoding to use for FileReader, so will probably barf on non-ASCII input. But there should be no need to construct a Reader, just pass in InputStream`.
StaxMan over 11 years

Also, links to performance test would be great -- the only commonly cited one (martinadamek.com/2011/02/04/…) does not show Gson as faster. Gson does have faster startup time, however; and version 2.1 is ok (earlier versions were very slow).
William Da Silva over 11 years

Well, it's not buggy, it's working very well for me, I had special characters without problem. The "caminho_arquivo" is the location of the file in the SDCard. I use that because I get a zipped file from the server, after that I unzip the file and get the .json file...
William Da Silva over 11 years

And with Jackson don't help me when I need to parse large files, so for my app GSON was the best, even more slow, can read large files.
StaxMan over 11 years

It may work on your platform; but it can easily fail on others; hence it is broken. You should just specify the encoding (UTF-8). This is a common mistake made by Java developers, so perhaps you just cut'n pasted it from somewhere. Was not meant as an insult, although wording may have been strong.
StaxMan over 11 years

As to Gson vs Jackson: just note that code you gave is different from original one -- you are using POJOs (which makes sense!), but Jackson code referred did not. It may be easily changed to do that, and Jackson approach will likewise work. So this is not a real difference between two approaches.
eugen over 11 years

Still can not understand why people use so much DOM apis...I suppose they don't even work directly on those objects so still need to convert it to POJOs... If only a small part of the json stream is needed, to reduce the data in memory jacksons streaming api could be used.
Ramit over 11 years

Thanks a lot StaxMan.. I did notice that it consumed 20-30 MB of memory but there is nothing much I could do to reduce it. For POJO based model, I have to change the entire structure of parsing. My input is a bit complex as it is an analytical data. So if this is the only way then I guess I have to go with that.
StaxMan over 11 years

Another possibility is that if (and only if) part of model is a sequence (List/Array) of things, you can use incremental parsing also. If so, either need to use ObjectMapper.readValues() or raw Streaming API (JsonParser) for iteration, then readValueAs() when located at root of sub-time.