API Gateway + Lambda + Python: Handling Exceptions

Question

I'm invoking a Python-based AWS Lambda method from API Gateway in non-proxy mode. How should I properly handle exceptions, so that an appropriate HTTP status code is set along with a JSON body using parts of the exception.

As an example, I have the following handler:

def my_handler(event, context):
    try:
        s3conn.head_object(Bucket='my_bucket', Key='my_filename')
    except botocore.exceptions.ClientError as e:
        if e.response['Error']['Code'] == "404":
            raise ClientException("Key '{}' not found".format(filename))
            # or: return "Key '{}' not found".format(filename) ?

class ClientException(Exception):
    pass

Should I throw an exception or return a string? Then how should I configure the Integration Response? Obviously I've RTFM but the FM is FU.

Solution

tl;dr

1. Your Lambda handler must throw an exception if you want a non-200 response.
2. Catch all exceptions in your handler method. Format the caught exception message into JSON and throw as a custom Exception type.
3. Use Integration Response to regex your custom Exception found in the errorMessage field of the Lambda response.

API Gateway + AWS Lambda Exception Handling

There's a number of things you need know about Lambda, API Gateway and how they work together.

Lambda Exceptions

When an exception is thrown from your handler/function/method, the exception is serialised into a JSON message. From your example code, on a 404 from S3, your code would throw:

{
  "stackTrace": [
      [
          "/var/task/mycode.py",
          118,
          "my_handler",
          "raise ClientException(\"Key '{}' not found \".format(filename))"
      ]
  ],
  "errorType": "ClientException",
  "errorMessage": "Key 'my_filename' not found"
}

 API Gateway Integration Response

Overview

"Integration Responses" map responses from Lambda to HTTP codes. They also allow the message body to be altered as they pass through.

By default, a "200" Integration Response is configured for you, which passes all responses from Lambda back to client as is, including serialised JSON exceptions, as an HTTP 200 (OK) response.

For good messages, you may want to use the "200" Integration Response to map the JSON payload to one of your defined models.

Catching exceptions

For exceptions, you'll want to set an appropriate HTTP status code and probably remove the stacktrace to hide the internals of your code.

For each HTTP Status code you wish to return, you'll need to add an "Integration Response" entry. The Integration Response is configured with a regex match (using java.util.regex.Matcher.matches() not .find()) that matches against the errorMessage field. Once a match has been made, you can then configure a Body Mapping Template, to selectively format a suitable exception body.

As the regex only matches against the errorMessage field from the exception, you will need to ensure that your exception contains enough information to allow different Integration Responses to match and set the error accordingly. (You can not use .* to match all exceptions, as this seems to match all responses, including non-exceptions!)

Exceptions with meaning

To create exceptions with enough details in their message, error-handling-patterns-in-amazon-api-gateway-and-aws-lambda blog recommends that you create an exception handler in your handler to stuff the details of the exception into a JSON string to be used in the exception message.

My prefered approach is to create a new top method as your handler which deals with responding to API Gateway. This method either returns the required payload or throws an exception with a original exception encoded as a JSON string as the exception message.

def my_handler_core(event, context):
    try:
        s3conn.head_object(Bucket='my_bucket', Key='my_filename')
        ...
        return something
    except botocore.exceptions.ClientError as e:
        if e.response['Error']['Code'] == "404":
            raise ClientException("Key '{}' not found".format(filename))

def my_handler(event=None, context=None):

    try:
        token = my_handler_core(event, context)
        response = {
            "response": token
        }
        # This is the happy path
        return response
    except Exception as e:
        exception_type = e.__class__.__name__
        exception_message = str(e)

        api_exception_obj = {
            "isError": True,
            "type": exception_type,
            "message": exception_message
        }
        # Create a JSON string
        api_exception_json = json.dumps(api_exception_obj)
        raise LambdaException(api_exception_json)

# Simple exception wrappers
class ClientException(Exception):
    pass

class LambdaException(Exception):
    pass

On exception, Lambda will now return:

{
    "stackTrace": [
        [
            "/var/task/mycode.py",
            42,
            "my_handler",
            "raise LambdaException(api_exception_json)"
        ]
    ],
    "errorType": "LambdaException",
    "errorMessage": "{\"message\": \"Key 'my_filename' not found\", \"type\": \"ClientException\", \"isError\": true}"
}

Mapping exceptions

Now that you have all the details in the errorMessage, you can start to map status codes and create well formed error payloads. API Gateway parses and unescapes the errorMessage field, so the regex used does not need to deal with escaping.

To catch this ClientException as 400 error and map the payload to a clean error model, you can do the following:

1. Create new Error model:

   {
     "type": "object",
     "title": "MyErrorModel",
     "properties": {
       "isError": {
           "type": "boolean"
       },
       "message": {
         "type": "string"
       },
       "type": {
         "type": "string"
       }
     },
     "required": [
       "token",
       "isError",
       "type"
     ]
   }
   

2. Edit "Method Response" and map new model to 400
3. Add new Integration Response
4. Set code to 400
5. Set regex to match "ClientException" types with tolerance for whitespace: .*"type"\s*:\s*"ClientException".*

6. Add a Body Mapping Template for application/json to map the contents of errorMessage to your model:

   #set($inputRoot = $input.path('$'))
   #set ($errorMessageObj = $util.parseJson($input.path('$.errorMessage')))
   {
       "isError" : true,
       "message" : "$errorMessageObj.message",
       "type": "$errorMessageObj.type"
   }
   

Comments

[unclemeat]

I may have missed a step. But once you have the integration response for 200 as default, won't any exception returned which doesn't match one of your specific errors be returned as a 200?

[Jose Alban]

Just came here to say json.dumps is strictly required - as without it, python raises an exception with a dict which is then converted to JSON-with-single-quotes which is invalid for API Gateway's $util.parseJson, which silently fails (yes, I know..) and returns an empty json object.

[Alastair McCormack]

@unclemeat indeed, but that's problem most users have with default setup. The method described above should always return a LambdaException no matter what the original exception was, unless something happens in the outermost exception handling. -- I suppose you could set an error as the default response and match on a known good string when you want to return a 200 status. You will still need to provide a Body Mapping Template to ensure your responses hide your stacktrace.