Using Boto3 in python to acquire results from dynamodb and parse into a usable variable or dictionary

Question

I'm trying to acquire the most recent entry into DynamoDB or to parse out the results I get, so I can skim the most recent item off the top.

This is my code

from __future__ import print_function # Python 2/3 compatibility
import boto3
import json
import decimal
from boto3.dynamodb.conditions import Key, Attr

# Helper class to convert a DynamoDB item to JSON.
class DecimalEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, decimal.Decimal):
            if o % 1 > 0:
                return float(o)
            else:
                return int(o)
        return super(DecimalEncoder, self).default(o)

dynamodb = boto3.resource('dynamodb', region_name='us-west-2',
endpoint_url="https://foo.foo.foo/aws")

table = dynamodb.Table('footable')
response = table.scan(
    Select="ALL_ATTRIBUTES",
    )

for i in response['Items']:
    print(json.dumps(i, cls=DecimalEncoder))

My results are a lot of the following that I'd like to either parse or if someone knows the code to just select the top entry that would be great.

{"MinorID": 123, "Location": "123westsideave"}
{"MinorID": 321, "Location": "456nowhererd"}
{"MinorID": 314, "Location": "123westsideave"}

Solution 1

at the end of my code where it says "print(json.dumps(i, cls=DecimalEncoder))" I changed that to "d = ast.literal_eval((json.dumps(i, cls=DecimalEncoder)))" I also added import ast at the top. It worked beautifully.

import ast

table = dynamodb.Table('footable')
response = table.scan(
    Select="ALL_ATTRIBUTES",
    )

for i in response['Items']:
    d = ast.literal_eval((json.dumps(i, cls=DecimalEncoder)))

Solution 2

import boto3
import json
import decimal
from boto3.dynamodb.conditions import Key, Attr

# Helper class to convert a DynamoDB item to JSON.
class DecimalEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, decimal.Decimal):
            return str(o)
        if isinstance(o, set):  #<---resolving sets as lists
            return list(o)
        return super(DecimalEncoder, self).default(o)


dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('mytable')

response = table.query(
    KeyConditionExpression=Key('object_type').eq("employee")
)

print(json.dumps((response), indent=4, cls=DecimalEncoder))

Comments

[Davos]

This seems redundant. i in response['Items'] is already a dictionary, then you are using json.dumps to serialize it to a string of JSON, then using ast.literal_eval to compile that string as literal python code as an initialized value of d, i.e. back to a dictionary. The only purpose of the DecimalEncoder is to serialize decimal objects in the dictionary, solely for the purpose of printing them. json.dumps is a handy method, but json is ambiguous and the json library takes an encoder class so you can be explicit about how you want to serialize ambiguous datatypes.

[Davos]

I suspect the issue arises from the fact that the dynamodb basic example shows a stub piece of code that just prints the data from the dynamodb response using the json.dumps method, which then has this weird serialization requirement. It's a bunch of code set up explicitly to print data, which is useful for a hello world look at the output, but will probably never be what you want to do in a real application that uses dynamodb as a data store. If you do need datatype conversions, then forcing it to a string then back to python objects is a really brutal way of doing it.

[Inga]

it still failed with 'set' object . error TypeError: Object of type set is not JSON serializable. i post update with a fix worked for me