How to handle huge data from a REST service

Question

We are using a REST service that returns huge data. In production, the server hardware can handle it. I need to test the REST service by getting the data on my local machine, which cannot handle the huge data. My local machine is a 4G i5. I am getting out of memory exception every time I hit the service.

response.getStatus() returns 200 status. But while collecting the data using input stream reader, I get an out of memory exception.

BufferedReader br = new BufferedReader(new
InputStreamReader(newByteArrayInputStream(response.getEntity().getBytes())));

Is there any other way to collect the data without hitting the memory exception?

I tried pushing my VM size to 2G but, it still doesn't work.

Solution 1

If the service supports it, use the HTTP standard approach of requesting chunks within the range of the entire response content payload.

Another approach (again, assuming the API supports it) is pagination.

Solution 2

You can implement a threshold setting in the REST service configuration that limits the amount of data returned. In production, the threshold would either not be set or would be set to a large value. In your test environment, you can set the threshold to a small, managable value.

Comments

[om39a]

This is a good approach that I can recommend. Thank you Brian!!