How to move data from Glue to Dynamodb

10,865

Solution 1

You can add the following lines to your Glue ETL script:

    glueContext.write_dynamic_frame.from_options(frame =DynamicFrame.fromDF(df, glueContext, "final_df"), connection_type = "dynamodb", connection_options = {"tableName": "pceg_ae_test"})

df should be of type DynamicFrame

Solution 2

I am able to write using boto3... definitly its not best approach to load but its working one. :)

dynamodb = boto3.resource('dynamodb','us-east-1') table = 
dynamodb.Table('BULK_DELIVERY')

print "Start testing"

for row in df1.rdd.collect():
    var1=row.sourceCid 
    print(var1) table.put_item( Item={'SOURCECID': "{}".format(var1)} )

print "End testing"
Share:
10,865
Robby
Author by

Robby

Updated on July 25, 2022

Comments

  • Robby
    Robby over 1 year

    We are designing an Big data solution for one of our dashboard applications and seriously considering Glue for our initial ETL. Currently Glue supports JDBC and S3 as the target but our downstream services and components will work better with dynamodb. We are wondering what is the best approach to eventually move the records from Glue to Dynamo.

    Should we write to S3 first and then run lambdas to insert the data into Dynamo? Is that the best practice? OR Should we use a third party JDBC wrapper for Dynamodb and use Glue to directly write to Dynamo (Not sure if this is possible, sounds a bit scary) OR Should we do something else?

    Any help is greatly appreciated. Thanks!

  • Nicus
    Nicus almost 4 years
    "AWS Glue does not currently support writing to Amazon DynamoDB." docs.aws.amazon.com/glue/latest/dg/…
  • Prashanth S.
    Prashanth S. over 3 years
    By reading the documentation I thought I can't write to dynamodb directly, but I tried the above script and it did work
  • Eldhose
    Eldhose over 2 years
    Hey, I want to update an entry if its already there in dynamodb, how can I achieve that? glueContext.write_dynamic_frame is failing when there is already an entry present with the same primary key. Please help
  • Cristián Vargas Acevedo
    Cristián Vargas Acevedo about 2 years
    Officially only Glue versión 1 is compatible for write on dynamodb