Aws Lambda response error

Question

I am running aws lambda which will fetch data from maria DB and return the fetched rows as a JSON object. A total number of item in JSON array is 64K.

I am getting this error:

{ "error": "body size is too long" }

Is there a way I can send all 64K rows by making any configuration change to lambda?

Solution 1

You cannot send the 64K rows (Which goes beyond 6MB body payload size limit) making configuration changes to Lambda. Few alternative options are.

• Query the data and build a JSON file with all the rows in /tmp (Up to 512MB) directory inside Lambda, upload it to S3 and return a CloudFront Signed URL to access the data.
• Split the dataset into multiple pages and do multiple queries.
• Use a EC2 instance or ECS, instead of Lambda.

Note: Based on the purpose of queried data, its size & etc. different mechanisms can be used, efficiently using other AWS services.

Solution 2

This error indicates that your response exceeds the maximum (6 MB), which is maximum data size AWS Lambda can respond.

http://docs.aws.amazon.com/lambda/latest/dg/limits.html

Solution 3

It seems that you're hitting the hard limit of a maximum 6 MB response size. As it's a hard limit there's unfortunately no way to increase this.

You'll need to set up your lambda to be able to send at most 6MB and paginate through the rows you need to retrieve in different invocations until you've fetched all 64K.

Sources: https://docs.aws.amazon.com/lambda/latest/dg/limits.html#limits-list https://forums.aws.amazon.com/thread.jspa?threadID=230229

Comments

[Irfan Pathan]

Dear can you explain me your third point in more details..Do you mean writing API in node js with express and publishing it on EC2 instance ?

[Ashan]

Yes. Either using NodeJS (With ExpressJS or any other suitable framework) in a EC2 instance or using Docker containers, host the service in Container Cluster.

[Irfan Pathan]

Thanks a lot @Ashan for the suggestion, I am working on it now...Cheers