Write csv file and save it into S3 using AWS Lambda (python)

Question

I'm trying to write a csv file into an S3 bucket using AWS Lambda, and for this I used the following code:

data=[[1,2,3],[23,56,98]]
with open("s3://my_bucket/my_file.csv", "w") as f:
   f.write(data)

And this raises the following error:

[Errno 2] No such file or directory: u's3://my_bucket/my_file.csv': IOError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 51, in lambda_handler
with open("s3://my_bucket/my_file.csv", "w") as f:
IOError: [Errno 2] No such file or directory: u's3://my_bucket/my_file.csv'

Can I have some help with this please ?

PS: I'm using python 2.7

Thanking you in advance

Solution

Better to answer later than never. There are four steps to get your data in S3:

• Call the S3 bucket
• Load the data into Lambda using the requests library (if you don't have it installed, you are gonna have to load it as a layer)
• Write the data into the Lambda '/tmp' file
• Upload the file into s3

Something like this:

import csv
import requests
#all other apropriate libs already be loaded in lambda

#properly call your s3 bucket
s3 = boto3.resource('s3')
bucket = s3.Bucket('your-bucket-name')
key = 'yourfilename.txt'

#you would need to grab the file from somewhere. Use this incomplete line below to get started:
with requests.Session() as s:
    getfile = s.get('yourfilelocation')

#Only then you can write the data into the '/tmp' folder.
with open('/tmp/yourfilename.txt', 'w', newline='') as f:
    w = csv.writer(f)
    w.writerows(filelist)
#upload the data into s3
bucket.upload_file('/tmp/yourfilename.txt', key)

Hope it helps.

Comments

[jarmod]

Lambda doesn't have native device driver support for s3:// URIs like that. Write the CSV file to local file system (/tmp) and then use boto3's put_object() method. You can also stream the file contents into S3 using boto3, if preferred.

[jarmod]

See stackoverflow.com/questions/40336918/…

[Ryan]

Okay so does the s3://my_bucket/ directory actually exist?

[kab]

Yesn but the file does not exist

[Mark B]

The s3://my_bucket/ "directory" doesn't exist locally... It's on S3. And it's not even a directory, it's an S3 bucket. It can't be accessed like a local file, you have to use boto3.