Sonatype Nexus: How to use Amazon S3 as a storage for maven artifacts?

Question

I've got a task to examine how to make our internal Nexus installation (installed on CentOs/RHEL) to store artifacts in Amazon S3 cloud storage (or any other cheap cloud storage).

So far, I had found only one solution - mount s3 as a file system, f.e. using s3fs. But I've met few problems using s3fs (beyond the scope of this topic).

Maybe someone already succeeded in the search of the solution for similar task, and could share the knowledge? Or maybe anyone had better ideas?

Thanks in advance.

Solution 1

Edit: Since version 3.12, S3 blobstore is included in nexus.

Sonatype has an open source plugin that does that:

https://github.com/sonatype/nexus-blobstore-s3

you compile it, put it on the nexus deploy directory, setup your S3 account and you're good to test it.

Solution 2

Nexus Sonatype does not support the use of S3 as it's filesystem store.

Why not use Nexus to manage your master repository and use a tool like s3sync to mirror the repository file contents into S3? Would be a useful way to provide versioned releases to customers.

Comments

[Aleksey Timohin]

>Nexus Sonatype does not support the use of S3 as it's filesystem store. Yep. There's no support out of the box. But, I'm sure, there is a way to do that, and I just want to find it.

[Mark O'Connor]

@AlekseyTimohin You can try and fool Nexus into thinking it's using a local file system, but didn't you say there are probelem with s3fs? Personally, I want my repository safe and reliable. For performance reasons one would need to be careful to ensure that indexes are stored locally. I'm guessing the extra latency imposed by a HTTP interface to S3 would kill performance. All in all mirroring would same to be a much safer option.

[Mark O'Connor]

@AlekseyTimohin Have you considered just publishing direct to S3? tech-frontlines.blogspot.com/2012/02/…

[Aleksey Timohin]

thank you for suggestion and link. From now I'll try to figure out if it'll work for me. Definitely, I don't want to make my artifacts public and available to everyone, but I hadn't find yet a way to make all the files available only to some subset of ip-addresses. But I will research that.

[quintonm]

The reason for wanting Nexus to use S3 for storage is rather simple. With S3, you get virtually unlimited storage and you only pay for what you actually use. If you are hosting Nexus in AWS, you currently have to provision EBS storage to hold the artifacts. You are then paying for the max amount of storage that you need. If you run low on storage, you have to provision more and then move everything to the new file system. You also have to worry about file system corruption which just isn't an issue with S3.

[pppai]

I tried using this Nexus plugin, I could create blob storage on S3. However, it only stored metadata files of artifacts in the S3 bucket. Is there a way to store artifacts in S3 through Nexus?

[pppai]

I tried using this Nexus plugin, I could create blob storage on S3 bucket. However, it only stored metadata files of artifacts in the S3 bucket (and the artifacts were stored in the same EC2 instance). Is there a way to store artifacts in S3 through Nexus?

[wargre]

Since 3.12 you don't need this plugin, it is integrated in nexus. The artifact are stored in S3, but remember nexus 3 stores things in blobstores and not as simple file ( you want to store a pom and a jar you will end up with 2 blob names ADDFSDCVSD and CDSFFGRJ in your S3 ). There is an integrated database to link artifact to blobstore, this database is not in S3 but in the sonatype work directory.

[pppai]

Yes, there are 2 files created per artifact in the Blob store. I wanted to store the artifacts in S3 instead of the attached EBS volume. Can I modify the OrientDB database such that artifacts are uploaded/downloaded from S3?

[wargre]

if your blob are written to EBS then when you created the blobstore you did not set type to S3 or when you create your repository you did not link the repository to the S3 blobstore