Move file to a different s3Location in Java

Question

My usecase is as follows. I want to move a file from a certain s3Location to a different s3Location, using the Java S3 SDK. For instance, if the file is in bucket/current, I want to move it to bucket/old.

I currently can download a file as an S3Object, turn that object into a File (java.io, since the S3 Java client for reasons I don't understand does not allow you to upload an S3Object, the very same object you download!) and upload that file. I'm curious if there is a better approach to this.

Thanks!

Solution 1

There is no direct implementation of a rename or move operation in S3. Instead, the typical solution is to copy the object to the new location and then delete the original. You can accomplish this with the AmazonS3#copyObject and AmazonS3#deleteObject methods of the AWS SDK for Java.

This is more efficient than the technique you described in your question of downloading the file locally and then re-uploading it under the new key. copyObject internally makes use of S3 server-side copy provided in the S3 REST API PUT Object - Copy operation. The copy is performed on the S3 server side, so you won't have to pay the I/O costs (and real money costs if transiting out of AWS servers) compared to a local file download/upload.

Please be aware that this is much different from the rename operation as provided in a typical local file system, for multiple reasons:

• It is not atomic. Most local file systems provide an atomic rename operation, which is useful for building safe "commit" or "checkpoint" constructs to publish the fact that a file is done being written and ready for consumption by some other process.
• It is not as fast as a local file system rename. For typical local file systems, rename is a metadata operation that involves manipulating a small amount of information in inodes. With the copy/delete technique I described, all of the data must be copied, even if that copy is performed on the server side by S3.
• Your application may be subject to unique edge cases caused by the Amazon S3 Data Consistency Model.

Solution 2

You can use moveObject of the StorageService class.