AWS API Gateway base64Decode produces garbled binary?

Question

I'm trying to return a 1px gif from an AWS API Gateway method.

Since binary data is now supported, I return an image/gif using the following 'Integration Response' mapping:

$util.base64Decode("R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7")

However, when I look at this in Chrome, I see the following binary being returned:

Instead of:

Could anyone help me understand why this is garbled and the wrong length? Or what I could do to return the correct binary? Is there some other what I could always return this 1px gif without using the base64Decode function?

Many thanks in advance, this has being causing me a lot of pain!

EDIT

This one gets stranger. It looks like the issue is not with base64Decode, but with the general handling of binary. I added a Lambda backend (previously I was using Firehose) following this blog post and this Stack Overflow question. I set images as binaryMediaType as per this documentation page.

This has let me pass the following image/bmp pixel from Lambda through the Gateway API, and it works correctly:

exports.handler = function(event, context) {

  var imageHex = "\x42\x4d\x3c\x00\x00\x00\x00\x00\x00\x00\x36\x00\x00\x00\x28\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x01\x00\x18\x00\x00\x00\x00\x00\x06\x00\x00\x00\x27\x00\x00\x00\x27\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff\x00\x00";
  context.done(null, { "body":imageHex });

};

However the following images representing an image/png or a image/gif get garbled when passed through:

exports.handler = function(event, context) {

//var imageHex = "\x47\x49\x46\x38\x39\x61\x01\x00\x01\x00\x80\x00\x00\x00\x00\x00\xff\xff\xff\x21\xf9\x04\x01\x00\x00\x00\x00\x2c\x00\x00\x00\x00\x01\x00\x01\x00\x00\x02\x01\x44\x00\x3b";
//var imageHex = "\x47\x49\x46\x38\x39\x61\x01\x00\x01\x00\x80\x00\x00\xff\xff\xff\x00\x00\x00\x21\xf9\x04\x01\x00\x00\x00\x00\x2c\x00\x00\x00\x00\x01\x00\x01\x00\x00\x02\x02\x44\x01\x00\x3b";
  var imageHex = "\x47\x49\x46\x38\x39\x61\x01\x00\x01\x00\x80\x00\x00\x00\x00\x00\x00\x00\x00\x21\xf9\x04\x01\x00\x00\x00\x00\x2c\x00\x00\x00\x00\x01\x00\x01\x00\x00\x02\x02\x44\x01\x00\x3b\x0a"
  context.done(null, { "body":imageHex });

};

This seems to be the same issue as another Stack Overflow question, but I was hoping this would be fixed with the Gateway API binary support. Unfortunately image/bmp doesn't work for my use case as it can't be transparent...

In case it helps anyone, this has been a good tool for converting between base64 and hex.

Solution 1

It looks like this was a known issue previously: https://forums.aws.amazon.com/thread.jspa?messageID=668306&#668306

But it should be possible now that they've added support for binary data: http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-payload-encodings.html

It looks like this is the bit we need: "Set the contentHandling property of the IntegrationResponse resource to CONVERT_TO_BINARY to have the response payload converted from a Base64-encoded string to its binary blob". Then we shouldn't need the base64Decode() function.

Working on a test now to see if this works.

EDIT: I was finally able to get this working. You can see the binary image here: https://chtskiuz10.execute-api.us-east-1.amazonaws.com/prod/rest/image

Here's my Lambda function that returns the base64 encoded PNG as a string: https://gist.github.com/davemaple/73ce3c2c69d5310331395a0210069263

I updated the method response as follows:

I updated the integration response to include a hard-coded image/png header:

The last step was tricky: setting the contentHandling property to "CONVERT_TO_BINARY". I couldn't figure out how to do in the AWS console. I had to use the CLI API to accomplish this:

aws apigateway update-integration-response \
    --profile davemaple \
    --rest-api-id chtskiuzxx \
    --resource-id ki1lxx \
    --http-method GET \
    --status-code 200 \
    --patch-operations '[{"op" : "replace", "path" : "/contentHandling", "value" : "CONVERT_TO_BINARY"}]'

I hope this helps.

Solution 2

To anyone else having problems with this: I was also banging my head against the wall trying to retrieve a binary image over API Gateway proxy integration from lambda, but then I noticed that it says right there in the Binary Support section of Lambda Console:

API Gateway will look at the Content-Type and Accept HTTP headers to decide how to handle the body.

So I added Accept: image/png to the request headers and it worked. Oh the joy, and joyness! No need to manually change content handling to CONVERT_TO_BINARY or muck about with the cli. Of course this rules out using, for example, <img src= directly (can't set headers).

So, in order to get a binary file over API Gateway from lambda with proxy integration:

• List all supported binary content types in the lambda console (and deploy)
• The request Accept header must include the Content-Type header returned from the lambda expression
• The returned body must be base64 encoded
• The result object must also have the isBase64Encoded property set to true

Code:

callback(null, {
    statusCode: 200,
    headers: { 'Content-Type': 'image/png' },
    body: buffer.toString('base64'),
    isBase64Encoded: true
}

Solution 3

Check out this answer. It helped me with exposing PDF file for download through GET request without any additional headers.

Comments

[Dave Maple]

What do the response headers look like in Chrome?

[rjmurt]

Thanks, the response headers are: HTTP/1.1 200 OK Content-Type: image/gif Content-Length: 52 Connection: keep-alive Date: Mon, 02 Jan 2017 22:08:55 GMT x-amzn-RequestId: 0d3f620c-d138-11e6-941a-0f16afc9bdc4 X-Amzn-Trace-Id: Root=1-586acf77-93ce6c87faa62ee76758abf5 X-Cache: Miss from cloudfront Via: 1.1 227087338674ca3d3d23a79539f2998b.cloudfront.net (CloudFront) X-Amz-Cf-Id: 9V1XUr1cPqjm7Bj2HOFfakLlFM5MWo_Ucuv9cdk35xsBz_xhcPvixQ== And response comes back as: data:image/gif;base64,R0lGODlhAQABAO+/vQAAAAAA77+977+977+9Ie‌​+/vQQBAAAAACwAAAAAAQ‌​ABAAACAUQAOw==

[Dave Maple]

What is the backend for your API Gateway? Is it Lambda?

[rjmurt]

Thanks for your help Dave, I've added further details of my investigation above. Would be really interested to hear how you get on!

[Dave Maple]

let me know if you get this working once you have a chance to review.

[rjmurt]

Wow, that worked! I can't thank you enough. I would not have spotted that!

[Dave Maple]

seriously -- the docs could use a little update in this area (is fairly new functionality in their defense).

[Dave Maple]

@tkiethanom : it does work if the response body is a valid base64 encoded image, the method response headers are set, content-type is set to match the image type and you've performed the patch operation to set contentHandling to CONVERT_TO_BINARY. I did get that error at one point but I think it resolved after the patch update.

[tkiethanom]

Actually this worked for me. So the CLI command is basically to get around the fact that when you select Lambda Function from the Create Method screen you aren't able to set the Content Handling dropdown. The dropdown only appears when you select AWS Service.

[Dave Maple]

@tkiethanom :: that's exactly it. It is available in an aws service integration -- but doesn't appear as an option in the console in a lambda integration.

[tkiethanom]

After further exploration I haven't been able to get this to work when checking the Lambda Proxy Integration box. Luckily I can use my Lambda without Proxy Integration, but I wanted to point this out in case anyone else runs into this issue.

[Dave Maple]

@tkiethanom : will try to setup a test this weekend to see what's different in the proxy integration.

[tkiethanom]

@DaveMaple Have you had any luck setting the Content-Type dynamically? When the response is returned as binary the API gateway is having trouble extracting out my integration.response.body.contentType for obvious reasons.

[Dave Maple]

i've tried a lot of things @tkiethanom :: so far sol. without an integration response i haven't found a way.

[Dave Maple]

ok sorry @tkiethanom: I am able to set a content-type to image/png using the lambda proxy, see gist here: gist.github.com/davemaple/47cf6e4dc8b1e23c3562c41a900a69f0. I just can't get the response converted to binary.

[tkiethanom]

@DaveMaple I'm in a similar situation, when I use Lambda Proxy to set Content-Type the CONVERT_TO_BINARY setting doesn't work anymore. So I'm trying to figure out a way to have dynamic Content-Type exported as binary.

[Hassan Siddique]

I want to add contentHandling property to "CONVERT_TO_BINARY" in integration response, but could not find any way in aws web console. Any one did that or any other workaround, please tell.

[Dave Maple]

You can do it in with the AWS skd or cli @HassanSiddique -- but not in the web interface. The accepted answer has a n example using the cli.

[Sabreena]

Could you please explain the last step? I didn't know how to find aws cli and how to add the contentHandling property to "CONVERT_TO_BINARY".

[Dave Maple]

@Sabreena -- try these steps: docs.aws.amazon.com/cli/latest/userguide/installing.html

[Sabreena]

@Dave Maple -- Can i send audio file as multipart/form-data through api gateway to s3? please check stackoverflow.com/questions/43733254/…

[istvanp]

This worked for me. In addition, you could set the Binary Media Types (in the Settings of the API) to */* and then you won't need to set an Accept header. However the caveat is that this prevents you from using text output.

[Benjamin Heinke]

This response object returns a 500 Internal Server error response