How to mount S3 bucket as local FileSystem?

Question

I have a python app running on a Jupiter-notebook on AWS. I loaded a C-library into my python code which expects a path to a file. I would like to access this file from the S3 bucket.

I tried to use s3fs:

s3 = s3fs.S3FileSystem(anon=False)

using s3.ls('..') lists all my bucket files... this is ok so far. But, the library I am using should actually use the s3 variable inside where I have no access. I can only pass the path to the c library.

Is there a way to mount the s3 bucket in a way, where I don't have to call s3.open(), and can just call open(/path/to/s3) were somewhere hidden the s3 bucket is really mounted as a local filesystem?

I think it should work like this without using s3. Because I can't change the library I am using internally to use the s3 variable...

with s3.open("path/to/s3/file",'w') as f:
      df.to_csv(f)

with open("path/to/s3/file",'w') as f:
      df.to_csv(f)

Or am I doing it completely wrong?

The c library iam using is loaded as DLL in python and i call a function :

lib.OpenFile(path/to/s3/file)

I have to pass the path to s3 into the library OpenFile function.

Solution 1

If you're looking to mount the S3 bucket as part of the file system, then use s3fs-fuse

https://github.com/s3fs-fuse/s3fs-fuse

That will make it part of the file system, and the regular file system functions will work as you would expect.

Solution 2

If you are targeting windows, it is possible to use rclone along with winfsp to mount a S3 bucket as local FileSystem

The simplified steps are :
rclone config to create a remote
rclone mount remote:bucket * to mount

https://github.com/rclone/rclone
https://rclone.org/

https://github.com/billziss-gh/winfsp
http://www.secfs.net/winfsp/

Might not the completely relevant to this question, but I am certain it will be to a lot of users coming here.

Comments

[Khan]

i thought s3fs is doing the same

[Allan Elder]

The code you have is doing it in the context of a process; it is only available to that process. If you use Fuse, that makes it mounted at the system level, and the drive path available to all processes.