Localhost Endpoint to DynamoDB Local with Boto3

Question

Although Amazon provides documentation regarding how to connect to dynamoDB local with Java, PHP and .Net, there is no description of how to connect to localhost:8000 using Python. Existing documentation on the web points to the use of the DynamoDBConnection method inside boto.dynamodb2.layer1, but this creates an incompatibility between live and test environments that use the boto3 protocol to manage dynamoDB.

In boto3, you can make a request to dynamo using the following constructor and variables set into the environment:

client = boto3.client('dynamodb')
table = client.list_tables()

Whereas the boto.dynamodb2.layer1 package requires you to construct the following:

client = DynamoDBConnection(
    host='localhost',
    port=8000,
    aws_access_key_id='anything',
    aws_secret_access_key='anything',
    is_secure=False)
table = client.list_tables()

Although it is possible to create logic which determines the proper constructor based upon the local environment, I am wary of building a set of methods which treat each constructor as the same. Instead, I would prefer to use boto3 for everything and to be able to set the endpoint for dynamoDB in the environmental variables. Unfortunately, that option does not appear to be currently be available.

Is there any way to use boto3 to define a dynamoDB local endpoint (like the other languages)? Or any chance that Amazon will plan to support this feature?

Solution 1

It does support DynamoDB Local. You just need to set the appropriate endpoint such as you can do with other language SDKs

Here is a code snippet of how you can use boto3's client and resource interface via DynamoDB Local:

import boto3

# For a Boto3 client.
ddb = boto3.client('dynamodb', endpoint_url='http://localhost:8000')
response = ddb.list_tables()
print(response)

# For a Boto3 service resource
ddb = boto3.resource('dynamodb', endpoint_url='http://localhost:8000')
print(list(ddb.tables.all()))

Solution 2

Note: You will want to extend the above response to include region. I have appended to Kyle's code above. If your initial attempt is greeted with a region error, this will return the appropriate '[]' response.

import boto3

## For a Boto3 client ('client' is for low-level access to Dynamo service API)
ddb1 = boto3.client('dynamodb', endpoint_url='http://localhost:8000', region_name='us-west-2')
response = ddb1.list_tables()
print(response)

# For a Boto3 service resource ('resource' is for higher-level, abstracted access to Dynamo)
ddb2 = boto3.resource('dynamodb', endpoint_url='http://localhost:8000', region_name='us-west-2')
print(list(ddb2.tables.all()))

Solution 3

use dummy access key and id otherwise it will throw an exception on running the methods.

import boto3

dynamodb = boto3.session('dynamodb',
                          aws_access_key_id="anything",
                          aws_secret_access_key="anything",
                          region_name="us-west-2",
                          endpoint_url="http://localhost:8000")

Solution 4

This is the tutorial python DynamoDb. It depicts how to connect to local instance.

http://docs.aws.amazon.com/amazondynamodb/latest/gettingstartedguide/GettingStarted.Python.01.html

It seems like the minimum required parameters are the following with the help of aws configuration (below).

dynamodb = boto3.resource('dynamodb', endpoint_url='http://localhost:8000/')

The region, access key and secret key parameters can be omitted when configure profile parameters using aws configure command (require install aws cli). However, you can create aws configuration files manually in your home (in case, you don't want to use aws cli).

file ~/.aws/config

[default]
output = json
region = anywhere

file ~/.aws/credentials

[default]
aws_access_key_id = whatever_id
aws_secret_access_key = whatever_key 

You can consult the aws configuration in http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html

Note in the local DynamoDb development region, aws_access_key_id and aws_secret_access_key values in those files can be anything. But if you want to use aws cli with the AWS then you must put the valid region, valid id and keys. They are available when you register to the AWS services.

More information, when you call

db = boto3.client('dynamodb')

The host the boto3 connect will base on the region parameter e.g. region=us-west-1 when call above api, it will connect to dynamodb.us-west-1.amazonaws.com. However, when pass parameter endpoint_url the region will not be used. For more AWS endpoint list please go to http://docs.aws.amazon.com/general/latest/gr/rande.html.

View more solutions

Comments

[R J]

Thanks for the explanation. Works perfectly, so I'm already adjusting my code. Do you know if this feature is documented somewhere? The logical place would be here. But, I looked around a bit before and didn't find any indication that boto3.client accepts **kwargs.

[Erin Schoonover]

This is required if you do not start DynamoDB Local with -sharedDb

[Bruno Bronosky]

"Is the feature documented anywhere?" HA! No, not with Amazon. They don't even document that there is a JS Web Shell at localhost:8000/shell Even though there is a "tour" in that shell, there is no documentation of the tutorial that you can go through using tutorial.start() in that shell. More info on those at youtu.be/tDqLwzQEOmM?t=1383 Finally, they don't "document" a version number, so this product has had to be removed from Homebrew. github.com/Homebrew/homebrew-core/pull/9175 Sigh.

[Supawat Pusavanno]

boto3.session(...) doesn't work. it doesn't recognize endpoint_url. for more info please refer to boto3.readthedocs.io/en/latest/reference/core/session.html

[Mieczysław Daniel Dyba]

In response to @RJ you can find documentation of endpoint_url here as well as other keyword arguments for the client method. I was directed to that link from here

[ketan]

@Kyle Knapp- To run this code credentials are necessary? I'm getting "Unable to locate credentials" error.

[ketan]

To run this code credentials are necessary? I'm getting "Unable to locate credentials" error.

[Dambre]

@kit Maybe a late one, but I was also jumping into "Unable to locate credentials" error. It can be simply solved by setting AWS_ACCESS_KEY_ID=RANDOM and AWS_SECRET_ACCESS_KEY=RANDOM environment variables.

[Mark Ransom]

Does it really make a difference which region_name you use when you're connecting to localhost? That doesn't make logical sense.