How to return binary data from lambda function in AWS in Python?

Question

I cannot get python lambda to return binary data. The node-template for thumbnail images works fine but I cannot get a python lambda to work. Below is the relevant lines from my lambda. The print("image_data " + image_64_encode) line prints a base64 encoded image to the logs.

def lambda_handler(event, context):
    img_base64 = event.get('base64Image')
    if img_base64 is None:
        return respond(True, "No base64Image key")

    img = base64.decodestring(img_base64)
    name = uuid.uuid4()
    path = '/tmp/{}.png'.format(name)

    print("path " + path)

    image_result = open(path, 'wb')
    image_result.write(img)
    image_result.close()

    process_image(path)

    image_processed_path = '/tmp/{}-processed.png'.format(name)
    print("image_processed_path " + image_processed_path)
    image_processed = open(image_processed_path, 'rb')
    image_processed_data = image_processed.read()
    image_processed.close()
    image_64_encode = base64.encodestring(image_processed_data)

    print("image_data " + image_64_encode)


    return respond(False, image_64_encode)


def respond(err, res):
    return {
        'statusCode': '400' if err else '200',
        'body': res,
        'headers': {
            'Content-Type': 'image/png',
        },
        'isBase64Encoded': 'true'
    }

Any pointers to what I'm doing wrong?

Solution 1

Following all the steps above didn't work on my case, because having the binary support for content-type = */* will convert all responses to binary.

My case:

• Multiple lambda functions returning json (text), just a single lambda returning a binary file. All have lambda proxy enabled.

• The lambdas are in an API Gateway

• The API Gateway is behind CloudFront

Hint: I have notice an important information in the API Gateway -> Settings

Quoting:

API Gateway will look at the Content-Type and Accept HTTP headers to decide how to handle the body.

This means that the Content-Type response header must match Accept request header

Solution:

1. Set Binary Media Types in API gateway to your mime type: image/jpg

2. In your HTTP request set Accept: image/jpg

3. In your HTTP response set Content-Type: image/jpg

{
  "isBase64Encoded": True,
  "statusCode": 200,
  "headers": { "content-type": "image/jpg"},
  "body":  base64.b64encode(content_bytes).decode("utf-8")
}

4. Next we must tell CloudFront to accept the 'Accept' header from the request. So, in CloudFront distribution, click on your API Gateway instance (ID is clickable) and once redirected to CloudFront instance go to Behaviour tab, select the path-pattern of your API (example: /api/*) and click on Edit button.

On the new screen, you have to add Accept header to Whitelist.

Note 1: If you have multiple file types, you must add them all to Binary Media Types in the API gateway settings

Note 2: For those coming from serverless and want to set the binary types when deploying your lambdas, then check this post: setting binary media types for API gateway

plugins:
  - serverless-apigw-binary

custom:
  apigwBinary:
    types:
- 'image/jpeg'

The serverless.yml file for cloudfront should contain:

resources:
    WebAppCloudFrontDistribution:
      Type: AWS::CloudFront::Distribution
      Properties:
        DistributionConfig:
          ...
          CacheBehaviors:
            ...
            - 
              #API calls
              ...
              ForwardedValues:
                ...
                Headers:
                  - Authorization
                  - Accept

Solution 2

I finally figured this out. Returning binary data from a python lambda is doable.

Follow the instructions here: https://aws.amazon.com/blogs/compute/binary-support-for-api-integrations-with-amazon-api-gateway/

Be sure to check the 'Use Lambda Proxy integration' when creating a new method.

Also be sure your Python Lambda response returns a base64-encoded body, sets isBase64Encoded to True, and an appropriate content type:

import base64

def lambda_handler(event, context):
    # ...

    body = base64.b64encode(bin_data)

    return {'isBase64Encoded'   : True,
            'statusCode'        : 200,
            'headers'           : { 'Content-Type': content_type },
            'body'              : body }

THEN:

For each of your routes/methods issue:

apigateway update-integration-response --rest-api-id <api-id> --resource-id <res-id> --http-method POST --status-code 200 --patch-operations "[{\"op\" : \"replace\", \"path\" : \"/contentHandling\", \"value\" : \"CONVERT_TO_BINARY\"}]"

In the AWS console. The and can be seen in the API Gateway 'breadcrumbs' ex:

<api-id> = zdb7jsoey8
<res-id> = zy2b5g

THEN: You need to 'Deploy API'. From what I found only it only worked AFTER deploying the API.

Be sure you setup the 'Binary Media Types' before deploying.

Hint: Nice AWS shell terminal here: https://github.com/awslabs/aws-shell

pip install aws-shell

Solution 3

As far as I can tell, this is also the case with Python 3. I'm trying to return a binary data (bytes). It's not working at all.

I also tried to use base-64 encoding and I have had no success.

This is with API Gateway and Proxy Integration.

[update]

I finally realized how to do this. I enabled binary support for type */* and then returned this:

return({
        "isBase64Encoded": True,
        "statusCode": 200,
        "headers": {
                "content-type": "image/jpg",
        },  
        'body':  base64.b64encode(open('image.jpg', 'rb').read()).decode('utf-8')
})  

Solution 4

I faced the same problem about 6 months ago. Looks like although there is now binary support (and examples in JS) in API Gateway, Python 2.7 Lambda still does not support valid binary response, not sure about Python 3.6.

Base64 encoded response is having problems because of JSON wrapping. I wrote a custom JS on client side taking the base-64 image out of this JSON manually, but this was also a poor solution.

Upload the result to S3 (behind the CloudFront) and return 301 to CloudFront seems to be a good workaround. Works best for me.

View more solutions

Comments

[Rahul]

Where is lambda?

[Onur Degerli]

Do you have any solution? I have the same problem.

[colllin]

Good tips, but didn't quite get it going for me. Can you show me exactly how you're encoding base64_encoded_binary_data?

[driedler]

import base64 base64_encoded_binary_data=base64.b64encode(bin_data)

[colllin]

Thank you @user1495323! That's very clear. I think what I was missing was that the "Binary Media Types" are actually matched with the client's incoming Accept header, not your outgoing Content-Type header, so I needed to add */* to my list of Binary Media Types.

[SangminKim]

docs.aws.amazon.com/apigateway/latest/developerguide/…

[Mihail Malostanidis]

So, basically, the Lambda function will always receive/send base64-encoded data inside a JSON. Only API Gateway will receive/send binary data with the client?

[jscul]

This is giving me back a 0 by 0 image, I have no idea what I'm doing wrong.

[Kyler Laird]

I'd first check the contents of image.jpg and ensure that it's available to the Lambda call. (I didn't do any error checking in the example.) Try wget/curl to capture the result of the Lambda call through API Gateway.

[Tobias Feil]

Thank you, this is working for me! Do you have an idea why you have to encode it base64 first and then decode it utf-8? In my mind, that's reading it as bytes first, then encoding it base64, then decoding the bytes, then decoding the base64. I would have thought you would have to switch the last 2 operations in order to not mingle the encodings?

[Pflugs]

Thank you for this solution. For me, the part I was forgetting was to include the Accept header in the request. Make sure to check that!

[maxwell]

Is there anyway to update the integration response via cloudformation template?