S3-compatible Object Storage systems

Question

Has anyone come across a near 100% Amazon S3 RESTful API compatible object storage system?

What I am after is a layer that sit on top of any (preferably a POSIX) filesystem that provides Amazon S3 style RESTful API to store (PUT), retrieve (GET), stat (HEAD) and delete (DELETE), with decent authentication.

Commercial projects/ideas are welcome too.

NOTE:

I have so far tried Eucalyptus and Cumulus; of which Eucalyptus seems to be blindly calling itself S3-compatible. The response XML documents are not at all compatible, and is patchy in certain places with no XML documents at all. Cumulus has managed to keep the response documents quite similar, but seems to have forgotten about data integrity!

Let me explain the latter part: both Eucalyptus and Cumulus have no support for integrity verification that Amazon S3 offers. What you can do with S3 is that you can provide a Base64(MD5(FILE)) along with the PUT request, which then gets verified by S3 before they respond with a success. Eucalyptus and Cumulus doesn't support this. With, Eucalyptus we can at least get around this by checking the MD5 given in the response document (not S3-compatible behaviour). In Cumulus, this is not possible as it doesnt respond with anything (like S3). It is made even worse by Cumulus, by not giving out an ETag in the HEAD request.

Solution 1

In the end, I had to move forward with Eucalyptus Walrus. Eventhough I ended up finding some fundamental issues with it, they all had possible work-arounds.

Thanks everyone for your valuable input!

Solution 2

Swift is OpenStack's object storage engine, and as of the Bexar release in Feb 2011, it claims an experimental S3-compatible middleware. As OpenStack is getting a lot of attention (Canonical are shifting Ubuntu to it over Eucalyptus this October, for example) it might be worth a look.

Solution 3

Newdream's Ceph has a S3-compatible fastcgi gateway which uses the same object storage system as the Ceph Filesystem itself. (http://ceph.newdream.net/wiki/RADOS_Gateway) Still not production ready, but making regular updates.

I found Openstack's object store more reliable in my testing though not S3 compatible with the s3 libraries I used or really with rackspace's own java client for their cloudfiles storage service (which uses their own version of swift). Though the jclouds project which has both s3 and Swift support worked for me.

Solution 4

[DISCLAIMER: I work for Scality]

Amazon stays very active on their AWS products and makes additions and changes/improvements to their API very frequently. It is then hard to find a near-100% compatible product (especially in commercial products, probably OSS products would be more reactive to any changes). Many object/cloud storage vendors have been toying around with their own protocols for some time (Atmos, DDN even OpenStack's swift..) but they've mostly all abandoned the idea of their own protocol becoming as widely used as S3. They are all (almost?) announcing an S3-compatible interface now or in the months to come. It is after all the industry de facto standard.The only contender left might be CDMI just because of its Standardized nature and the fact it comes from SNIA, a non-profit, non-vendor organization. But it's clearly not being adopted or talked about as much as S3.

There are a few options as far as S3-compatible interface goes, like Mezeo, the ones you've mentioned (Eucalyptus and Cumulus), OpenStack (although it's not out yet i believe). Scality is another one of those.

Scality's S3 implementation (called RS2 - REST Storage Service) is very close to S3 in that our requests/replies are consistent with what you'd get from S3. Some of the functionalities tied to S3's infrastructure are not present for obvious reasons (Region selection for instance). But the necessary commands are there: GET/PUT/DELETE an object, bucket creation and listing ; and react just like S3 would.

I would just add, to answer your second problem, that Scality's product, RING, is a software object storage platform that solves the data integrity issues you're referring to by checksumming and doing background integrity checks of all objects stored on the RING. Storing an object is basically a transactional query, and once an object has been claimed as "successfully stored", it will forever be, with constant integrity checks (on every read for example).

I won't go into details. you can go to our website to understand how we alleviate storage admins from the common data management nightmares and let them sleep at night knowing their data is available. Always. :)

Best, Marc Villemade @mastachand

Solution 5

You could also look at Cloud Foundry vBlob: https://github.com/cloudfoundry/vblob

It's a node.js app (with Ruby wrapper for CF integration) that implements a fair chunk of the S3 protocol on top of any file system that the VM can "see".

View more solutions

Comments

[crb]

S3FS is the reverse order to what the poster wants. It makes a filesystem out of an S3 bucket. The OP wants to make a S3 bucket out of a filesystem.

[CodeMedic]

Sean; I am not looking for a client library or a client too to interact with storage; more interested in the server itself.