Read Multiple JSON object from a Text File

Question

My Question is similar to what has been asked here . few points :

1. I can not change the format. (No commas to be added etc)
2. This is basically a huge .txt file containing 1000's of Json objects.
3. My Json objects are HUGE.

This is what I am doing right now :

    FileReader fileReader = new FileReader(fileName);
        BufferedReader reader = new BufferedReader(fileReader);
        String data = "";
        while((data = reader.readLine()) != null){
            ObjectMapper mapper = new ObjectMapper();
            Map<String,String> map = mapper.readValue(data, Map.class);
        }

Currently I am using Jackson and Ideally I would like to read one Json Object from the file at a time, Parse it and then move on to the next one. I need to count let say unique number of id's from these Json object and do more operations. It will be best to read them one by one.

Is jackson would be the best way going forward ? This is a good example of parsing huge Json, But it deals with only one object per file. My file has huge Jsons (1000s of them).

Solution 1

Here is a Jackson example that works for me. I have thousands json objects (tokens) in a single json file. This code will iterate through the file read each token and print it's serial.

Required imports:

import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.databind.ObjectMapper;

Using Jackson to read multiple json objects from FileInputStream:

try (FileInputStream fis = new FileInputStream("D:/temp/tokens.json")) {
        JsonFactory jf = new JsonFactory();
        JsonParser jp = jf.createParser(fis);
        jp.setCodec(new ObjectMapper());
        jp.nextToken();
        while (jp.hasCurrentToken()) {
            Token token = jp.readValueAs(Token.class);
            jp.nextToken();
            System.out.println("Token serial "+token.getSerialNumber());
        }
    }

Solution 2

Here is a more JAVA 8ish solution for your query, I always lean toward BufferedReader over InputStreams for any place where parsing is going to be done a lot of time.

 ObjectMapper mapper  = new ObjectMapper();
 JsonFactory jsonFactory = new JsonFactory();
 try(BufferedReader br = new BufferedReader(new FileReader("luser.txt"))) {
     Iterator<luser> value = mapper.readValues( jsonFactory.createParser(br), luser.class);
     value.forEachRemaining((u)->{System.out.println(u);});
 }

The deserialization for each object happens as part of next(), in each iteration.

Solution 3

Here is how I used Gson's JSONReader API to handle similar requirement as above

public static List<YOURPOJO> readTraceLog(String filepath) throws IOException {

         Gson gson = new Gson();
         JsonReader jsonReader = new JsonReader(new FileReader(filepath));
         // important as handles unwanted formatting stuffs such empty spaces 
        jsonReader.setLenient(true);

        boolean start = true; // start of read
        jsonReader.beginObject(); // first object begins
        //List to hold object 
        List<YOURPOJO> completeList = new ArrayList<YOURPOJO>();
        //YOURPOJO has two attributes one is ID and other is list of ANOTHERPOJO

        while (jsonReader.hasNext()) {
            if (!start) {
                //to stop end of Document
                if (jsonReader.peek().toString().matches("END_DOCUMENT")) {
                    break;
                }
                //continue reading object as the come -{
                jsonReader.beginObject();

            }
            start = false;
            YOURPOJO pojo = new YOURPOJO();
            //read attribute id
            String name = jsonReader.nextName();
            pojo.setId(name);

            //list to store ANOTHERPOJO objects
            List<ANOTHERPOJO> tempList = new ArrayList<ANOTHERPOJO>();

            //begin reading list - [
            jsonReader.beginArray();
            while (jsonReader.hasNext()) {
                ANOTHERPOJO t = gson.fromJson(jsonReader, ANOTHERPOJO.class);
                tempList.add(t);
            }
            //end reading list - ]
            jsonReader.endArray();

            //store data
            pojo.setTraceDetails(tempList);
            completeList.add(YOURPOJO);

            //end of object  - }
            jsonReader.endObject();

        }

        jsonReader.close();
        return completeList;
    }

Comments

[Dakshinamurthy Karra]

Does the parser takes a InputStream or Reader as a parameter? In that case it might be possible to loop reading objects till the InputStream reaches EOF (check for available).