Python - AWS Lambda extract a key from JSON input

Question

Im trying to implement a function that will get the event from cloudwatch and print the results. I am able to get the event but I want to extract one particular key from that JSON.

Here is my function:

import json
    def lambda_handler(event, context):
        print("Received event: " + json.dumps(event, indent=2)) 
        message = event['Records'][0]['Sns']['Message']
        print(message)

The event got from Cloudwatch:

"Records": [
{
"EventVersion": "1.0", 
"EventSubscriptionArn": "arn:aws:sns:us-east-1:xxxxxxxxxxxxx:bhuvi:XXXXXXXXXXXXXXXXXXXXXXXXXX", 
"EventSource": "aws:sns", 
"Sns": {
"SignatureVersion": "1", 
"Timestamp": "2018-01-13T19:18:44.369Z", 
"Signature": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", 
"SigningCertUrl": "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem", 
"MessageId": "4b76b0ea-5e0f-502f-81ec-e23e03dbaf01", 
"Message": "{\"AlarmName\":\"test\",\"AlarmDescription\":\"test\",\"AWSAccountId\":\"xxxxxxxxxxxxx\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 out of the last 1 datapoints [2.6260535333900545 (13/01/18 19:13:00)] was greater than or equal to the threshold (1.0) (minimum 1 datapoint for OK -> ALARM transition).\",\"StateChangeTime\":\"2018-01-13T19:18:44.312+0000\",\"Region\":\"US East (N. Virginia)\",\"OldStateValue\":\"OK\",\"Trigger\":{\"MetricName\":\"CPUUtilization\",\"Namespace\":\"AWS/RDS\",\"StatisticType\":\"Statistic\",\"Statistic\":\"AVERAGE\",\"Unit\":null,\"Dimensions\":[{\"name\":\"DBInstanceIdentifier\",\"value\":\"myrds\"}],\"Period\":300,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"GreaterThanOrEqualToThreshold\",\"Threshold\":1.0,\"TreatMissingData\":\"\",\"EvaluateLowSampleCountPercentile\":\"\"}}", 
"MessageAttributes":
{}
, 
"Type": "Notification", 
"UnsubscribeUrl": "https://sns.us-east-1.amazonaws.com/?xcsgagrgrwgwrg", 
"TopicArn": "arn:aws:sns:us-east-1:xxxxxxxxxxxxx:bhuvi", 
"Subject": "ALARM: \"test\" in US East (N. Virginia)"
}
}
]
}

My extract command(Upto message) and its result:

message = event['Records'][0]['Sns']['Message']
        print(message)

Result

{
    "AlarmName": "test",
    "AlarmDescription": "test",
    "AWSAccountId": "xxxxxxxxxxxxx",
    "NewStateValue": "ALARM",
    "NewStateReason": "Threshold Crossed: 1 out of the last 1 datapoints [2.6260535333900545 (13/01/18 19:13:00)] was greater than or equal to the threshold (1.0) (minimum 1 datapoint for OK -> ALARM transition).",
    "StateChangeTime": "2018-01-13T19:18:44.312+0000",
    "Region": "US East (N. Virginia)",
    "OldStateValue": "OK",
    "Trigger": {
        "MetricName": "CPUUtilization",
        "Namespace": "AWS/RDS",
        "StatisticType": "Statistic",
        "Statistic": "AVERAGE",
        "Unit": null,
        "Dimensions": [
            {
                "name": "DBInstanceIdentifier",
                "value": "myrds"
            }
        ],
        "Period": 300,
        "EvaluationPeriods": 1,
        "ComparisonOperator": "GreaterThanOrEqualToThreshold",
        "Threshold": 1,
        "TreatMissingData": "",
        "EvaluateLowSampleCountPercentile": ""
    }

I want to extract some values from this message pane.

For eg: I want to extract name. So I tried the below command, but unfortunately its not working. Can anyone help me on this?

my code for this:

message = event['Records'][0]['Sns']['Message']['Trigger']['Dimensions']['name']
    print(message)

ERROR:

{
  "stackTrace": [
    [
      "/var/task/lambda_function.py",
      14,
      "lambda_handler",
      "message = event['Records'][0]['Sns']['Message']['Trigger']['Dimensions']['name']"
    ]
  ],
  "errorType": "TypeError",
  "errorMessage": "string indices must be integers"
}

Solution 1

So there are 3 problems:

Problem 1: In your example event, ['Records'][0]['Sns']['Message'] is a str in JSON format. That means that you need to parse to a dict like this:

 message = event['Records'][0]['Sns']['Message']
 message = json.loads(message)

Problem 2: message['Trigger']['Dimensions'] is a list but you are trying to access it like if it were a dict. So you only need to change your code to:

message = message['Trigger']['Dimensions'][0]['name']

Problem 3: Message is a str that means that you need to verify that is a plain str or json str (otherwise you are going to have problems with multiple structures and types). For that your code could look like:

 message = event['Records'][0]['Sns']['Message']
    if isinstance(message, str):
        try:
            message = json.loads(message)
        except Exception as e:
            print(e) # Or do nothing, this is just to log the error
    elif isinstance(message, list):
        message = message[0]
    # Maybe evaluate bool, tuple, etc other types

    print('RESPONSE', message['Trigger']['Dimensions'][0]['name'] if isinstance(message, dict) else message)

However I would also recommend to make it more extensible iterating the elements that you know are list. And for safety reasons (trying to avoid null pointer exceptions), use the get() function with a default value. http://www.tutorialspoint.com/python/dictionary_get.htm . Try maybe to create a function to parse structures and make it reusable.

Good luck!

Solution 2

Just as Records is a list, so you use ['Records'][0]['Sns']..., so is Dimensions, so again you need to access the first element.

Comments

[TheDataGuy]

Sorry I didn't get you, can you please give me what should I give in the message parameter?

[Juan Urrego]

Ohhh I think that now I see it, Message is a str with JSON content (is not a dict, it think that the print was a little bit confusing). In that case you just need to transform it from json to dict. You can do this: import json message = json.loads(event['Records'][0]['Sns']['Message']) print(message['Trigger']['Dimensions'][0]['name'])

[TheDataGuy]

let me try this one.

[Juan Urrego]

Can you do something? can you show me this print? print(event['Records'][0]['Sns']['Message'].__class__.__name‌​__)

[TheDataGuy]

The captured JSON contents are like this, "Message": "{\"AlarmName\":\"test\". Some " and / characters are there. Was this causing the error?

[Juan Urrego]

No, but check that I'm asking you to print the class name of the object, please try that: print(event['Records'][0]['Sns']['Message'].__class__.__name‌​‌​__)

[TheDataGuy]

sure, give me a moment

[Juan Urrego]

ok, the exception from json is not so meaningful. Can you use simplejson? just to test what is wrong (usually it shows more meaningful error messages). You don't need to do as much just, the pip install and import it like this: import simplejson as json and try again to see the error

[TheDataGuy]

I can't install, because it Lambda(serverless from AWS)

[Juan Urrego]

Ohhhh so you are doing it directly from the AWS editor, I thought you were uploading the project. In that case you can't install libraries that's true :S. But this is odd, it should be working, the only thing that I can imagine is that in some cases the str is empty or None. Ok let's this before applying the loads: message.replace('\"', '"')

[TheDataGuy]

So what should I give in message parameter?

[Juan Urrego]

Make something like this: message = event['Records'][0]['Sns']['Message'] then message = json.loads(message.replace('\"', '"')) just give it a try, maybe its having some kind of "extra" escape characters

[TheDataGuy]

Again Im getting the same error :( ` "errorMessage": "string indices must be integers"`

[Juan Urrego]

that's weird, after the json loads what is the class of message? I mean: print(message.__class__.__name‌​__) what is returning

[TheDataGuy]

seems json.loads doing something, whenever I gave this im getting this error(even this time too) ` "errorMessage": "No JSON object could be decoded" `

[Juan Urrego]

can you share the complete code fragment maybe in codepen, gist or something similar as it is (with the changes made), just to test it locally

[Juan Urrego]

Ok, I just took your example event and test it in my AWS Lambda and is working for me. BTW I'm using python 3 (and I didn't do the replace stuff). Can you send me a print of your event (not using the json.dumps that you were doing, pure print(event))

[TheDataGuy]

gits updated with print(event), meanwhile i'll try to execute this with Python 3.6

[Juan Urrego]

I commented your gist with the piece of code that worked for me (using your example event).

[TheDataGuy]

Is this worked on Lambda? I tried 2.7 and 3.6, but no luck :(

[Juan Urrego]

Yes, AWS Lambda with python 3.6 using the example event that you published in gist :S

[TheDataGuy]

Anyhow its works for you, So update this in your answer, I'll mark it as answered.

[Juan Urrego]

Now you changed the structure of your event, now message is just a plain str, evaluate that case. I commented your gist with that solution. Please always verify the structure of your message using the .get() function. Maybe you should create kind of a parser to verify multiple structures

[TheDataGuy]

Awesome, it works like a charm, Can you please update your answer which you mentioned in the Gits, So I'll make it as answered.