Can we copy the files and folders recursively between aws s3 buckets using boto3 Python?
26,177
S3 store object, it doesn't store folder, even '/' or '\' is part of the object key name. You just need to manipulate the key name and copy the data over.
import boto3
old_bucket_name = 'SRC'
old_prefix = 'A/B/C/'
new_bucket_name = 'TGT'
new_prefix = 'L/M/N/'
s3 = boto3.resource('s3')
old_bucket = s3.Bucket(old_bucket_name)
new_bucket = s3.Bucket(new_bucket_name)
for obj in old_bucket.objects.filter(Prefix=old_prefix):
old_source = { 'Bucket': old_bucket_name,
'Key': obj.key}
# replace the prefix
new_key = obj.key.replace(old_prefix, new_prefix, 1)
new_obj = new_bucket.Object(new_key)
new_obj.copy(old_source)
Optimized technique of defining new_key
suggested by zvikico:
new_key = new_prefix + obj.key[len(old_prefix):]
Author by
Gowthaman Javi
Updated on September 14, 2020Comments
-
Gowthaman Javi over 3 years
Is it possible to copy all the files in one source bucket to other target bucket using boto3. And source bucket doesn't have regular folder structure.
Source bucket: SRC Source Path: A/B/C/D/E/F.. where in D folder it has some files, E folder has some files Target bucket: TGT Target path: L/M/N/
I need to copy all the files and folders from above SRC bucket from folder C to TGT bucket under N folder using boto3.
Can any one aware of any API or do we need to write new python script to complete this task.