AWS API-Gateway communicating to SNS

Question

I am building an API which will be serviced by Lambda functions but I need these to be asynchronous so rather than connecting the API-Gateway directly to the Lambda function I'm using the "AWS Service Proxy" to publish SNS messages and then have the Lambda function subscribe to the relevant SNS topic so it receives delivery of the requests. Here's a picture which illustrates the flow:

I have tested both the Lambda function in isolation as well pub/sub messaging between SNS and Lambda but I am struggling with the API-Gateway to SNS handoff. Documentation is quite light but what I am assuming right now is that the following attributes must be sent in the POST request:

1. Action: the API-Gateway offers to set this in the UI and I have put in the Publish action which is the appropriate SNS action

2. Message: the body of the POST message should be a JSON document. It would be passed by the web client and proxied through the gateway to SNS.

3. TopicArn: indicates the SNS topic that we're publishing to. In my design this would be a static value/endpoint so I'd prefer that the web-client not have to pass this too but if it were easier to do this that would be fine too.

I have tried lots of things but am just stuck. Would love to find a good code example somewhere but any help at all would be appreciated.

---

Wanted to add a little more context on my current attempt:

I have tried publishing my API and using Postman to try and get a valid response. Here's the postman screens(one for header vars, one for JSON body):

This results in the following error message:

{
   "Error": {
     "Code": "InvalidParameter",
     "Message": "Invalid parameter: TopicArn or TargetArn Reason: no value for required parameter",
     "Type": "Sender"
  },
  "RequestId": "b33b7700-e8a3-58f7-8ebe-39e4e62b02d0"
}

the error seems to indicate that the TopicArn parameter is not being sent to SNS but I have included the following in API-Gateway:

Solution 1

I did eventually get this to work after working with AWS support. Here's my solution:

• First of all even though you're sending a POST you will not be able to send a JSON message in the body of the message as you might expect
• Instead you must URL Encode the JSON and pass it as query parameter
• Also remember that the JSON you send should start with a root object of default which in SNS-world means the "default channel"
• Then, eventually Lambda picks up the SNS event you must also abstract away a lot of the noise to get at your JSON message. For this I created the following function that I use within my Lambda function:

/**
 * When this is run in AWS it is run "through" a SNS
 * event wconfig.ich adds a lot of clutter to the event data,
 * this tests for SNS data and normalizes when necessary
 */
function abstractSNS(e) {
  if (e.Records) {
    return JSON.parse(decodeURIComponent(e.Records[0].Sns.Message)).default;
  } else {
    return e;
  }
}

/**
 * HANDLER
 * This is the entry point for the lambda function
 */
exports.handler = function handler(event, context) {
  parent.event = abstractSNS(event);

Solution 2

I'm from the Api Gateway team.

I believe there are a few formats for the HTTP request to the Publish API, but here's the one I used first:

AWS Region us-west-2

AWS Service sns

AWS Subdomain

HTTP method POST

Action Publish

== query strings ==

Subject 'foo'
Message 'bar'
TopicArn 'arn:aws:sns:us-west-2:xxxxxxxxxxxx:test-api'

This worked for me to publish a message.

Let me know if you have further troubles.

Jack

Solution 3

If anybody is still looking for a solution to the original problem, proxying a JSON request body to an SNS topic via API gateway alone, it's possible.

Create the gateway as Ken describes above. Then simply proxy the body to the Integration Request's query parameters. You can also hard code Subject, TopicArn, etc here, or map those from the request's body using a JsonPath.

For example:

{
   //body
   "topic": "arn:aws:sns:1234567:topic"
}

Could be mapped to a header as:

method.request.body.topic

Solution 4

You could use API Gateway to invoke your Lambda function asynchronously by configuring it as an AWS service proxy. The configuration is basically the same you see in this GitHub sample, with the exception that the uri for the Lambda invocation changes to /invoke-async/ instead of just /invoke/

Solution 5

Here is a step by step guide for people who still couldn't figure it out after looking through the above answers. The variable names are case sensitive so make sure you are exact.

1. Open the Post Method
   a. Select Method Request
   b. Change Request Validator to Validate body, query string parameters, and headers
   c. Expand URL Query String Parameters
   d. Add the following two query string parameters
   Name: TopicArn ----> Select Required
   Name: Message -----> Select Required

2. Return to the Post Method and open Integration Request
   a. Expand URL Query String Parameters
   b. Add the following two query string parameters
   Name: TopicArn Mapped from: method.request.querystring.TopicArn
   Name: Message Mapped from: method.request.querystring.Message

3. When testing, alter the below command to match your SNS ARN and put it in Query Strings.
   TopicArn=arn:aws:sns:us-west-2:1234567890:SNSName&Message="Hello from API Gateway"

Sources/Further Information:
API Gateway Proxy Integration Service Guide
SNS Publish Method Documentation

View more solutions

Comments

[ken]

Uh oh. I had posted an answer but it didn't go through. I'll try to get to this tomorrow.

[ken]

I'm a bit confused by your example. Where does this YAML go? Can API-Gateway import Swagger definitions?

[ken]

Also note that invoke-async is deprecated ... they recommend using invoke instead (I believe there is a parameter to set it to async though)

[ken]

Is it possible to export a Swagger YAML definition for an API I setup using the UI? Would be nice to see it in YAML form rather than just the UI.

[ken]

WRT to language, the Lambda functions at the end of this flow are Node JS, the calling web client is a JS SPA (EmberJS). In between -- where the message transformation is taking place -- is all just API Gateway configuration.

[ken]

The example message you've given does look like its urlencoded, the example above it seemed to suggest that you could pass in JSON into the body message (at least that's how I read it).

[ken]

Also, I'm assuming that signing and passing along the Access Key is all handled by the API-Gateway as that would break the "configuration only" approach they seem to be promoting (and the UI does ask for execution arn)

[xpa1492]

Apologies - I am completely off in my answer. I though the API Gateway is a component you wrote (did not realize AWS has a product called this way :).

[ken]

Thanks anyway, I need all the help I can get. :)

[William Francis Gomes]

Hello, can you give some details of your solution like what part you have to change/what part differ from normal setup?? I am having this problem for last 2 days and stuck. Problem is i am not clear with the solution you gave above. Thanks in advance.

[ken]

@WilliamFrancisGomes not sure what else to add ... the "handler" function gives you a event object but if you're using SNS to fire the event you'll get the full SNS event. For me the SNS details are not very useful except for what is in the e.Records[0].Sns.Message so the above function unwraps the part I care about.

[ken]

Note that now API Gateway provides a direct way to call Lambda and invoke it asynchronously so you don't need to use SNS in some cases.

[William Francis Gomes]

my problem is describe here in details: stackoverflow.com/questions/38229941/…

[ken]

@WilliamFrancisGomes what you're trying to do is definitely achievable and a good use-case for SNS. Where in the chain are things breaking down? Is API-Gateway firing the SNS message (if you have any doubts you can just subscribe to the SNS topic via email)? If it is, do your lambda functions execute? I would need more insight into where in the chain things aren't working for you.

[abhiarora]

Can you explain a bit more?

[Anand Bajpai]

Okay so We connect Gateway to Lambda. as per doc: cloudonaut.io/… and then use boto3.readthedocs.io/en/latest/reference/services/… .....to connect lambda to SNS

[Anand Bajpai]

Let me know if this helps or I will post with images,,Only thing is this will use Python

[Rudresh Ajgaonkar]

How do i map the request body and the params sent in the request to the Subject, Message, and TopicArn for SNS?

[Thanh Pham]

@ken, there are still some use cases that messages should be published to SNS from API Gateway instead of invoking Lambda. One is because of Lambda cold start issue and my app needs to publish data in real time.