Adding s3 bucket as docker volume

Question

I have one spring boot application which is in our internal data center, which process files from a specific folder on the host.

we wanted to deploy this to aws and wanted to use s3 bucket to upload files for processing.

is there any way we can add s3 bucket space as docker volume?

Solution 1

UPD: See at the bottom of this answer.

Other answers mistakenly say that AWS S3 is an object store and you can not mount it as volume to docker. Which is not correct. AWS S3 has a 3rd party FUSE driver, which allows it to be mounted as local filesystem and operate on objects as if those were files.

However it does not seem this FUSE driver has been made available as storage plugin for Docker just yet.

Edit: well, i have to correct myself after just a couple of minutes posting this. There in fact is a FUSE based driver for Docker to get volume mounted from AWS S3. See REX-ray and also here for possible configuration issue.

Solution 2

Other answers have correctly pointed out that : AWS S3 is an object store and you can not mount it as volume to docker.

That being said, using S3 with spring application is super easy and there is framework developed called spring-cloud. spring-cloud works excellent with AWS.

Here is sample code :

public void uploadFiles(File file, String s3Url) throws IOException {
   WritableResource resource = (WritableResource) resourceLoader.getResource(s3Url);

   try (OutputStream outputStream = resource.getOutputStream()) {
       Files.copy(file.toPath(), outputStream);
   }
}

You can find detailed blog over here.

Solution 3

S3 is an object store, not a file system. You should have S3 trigger a message to SQS when new objects are added to the bucket. Then you can code your application running in the Docker container to poll SQS for new messages, and us the S3 location in the message to copy the object from S3 to local storage (using the appropriate AWS SDK) for processing.

Comments

[Nemo]

They want to upload files to S3, but not poll from S3.

[Kevin]

2022: the S3 FUSE driver is active, but REX-ray seems to have quit being maintained in 2019, and their homepage is no longer hosted.