Set Expires header for an existing S3 object using AWS Java SDK

Question

I'm updating existing objects in an Amazon S3 bucket to set some metadata. I'd like to set the HTTP Expires header for each object to better handle HTTP/1.0 clients.

We're using the AWS Java SDK, which allows for metadata changes to an object without re-uploading the object content. We do this using CopyObjectRequest to copy an object to itself. The ObjectMetadata class allows us to set the Cache-Control, Content-Type and several other headers. But not the Expires header.

I know that S3 stores and serves the Expires header for objects PUT using the REST API. Is there a way to do this from the Java SDK?

Updated to indicate that we are using CopyObjectRequest

Solution 1

To change the metadata of an existing Amazon S3 object, you need to copy the object to itself and provide the desired new metadata on the fly, see copyObject():

By default, all object metadata for the source object are copied to the new destination object, unless new object metadata in the specified CopyObjectRequest is provided.

This can be achieved like so approximately (fragment from the top of my head, so beware):

AmazonS3 s3 = new AmazonS3Client();

String bucketName = "bucketName ";
String key = "key.txt";
ObjectMetadata newObjectMetadata = new ObjectMetadata();
// ... whatever you desire, e.g.:
newObjectMetadata.setHeader("Expires", "Thu, 21 Mar 2042 08:16:32 GMT");

CopyObjectRequest copyObjectRequest = new CopyObjectRequest()
                 .WithSourceBucketName(bucketName)
                 .WithSourceKey(key)
                 .WithDestinationBucket(bucketName)
                 .WithDestinationKey(key)
                 .withNewObjectMetadata(newObjectMetadata);

s3.copyObject(copyObjectRequest);

Please be aware of the following easy to miss, but important copyObject() constraint:

The Amazon S3 Acccess Control List (ACL) is not copied to the new object. The new object will have the default Amazon S3 ACL, CannedAccessControlList.Private, unless one is explicitly provided in the specified CopyObjectRequest.

This is not accounted for in my code fragment yet!

Good luck!

Solution 2

We were looking for a similar solution and eventually settled for max-age cache-control directive. And we eventually realized that hte cache-control overrides the Expires even if expires is more restrictive. And anyways cache-control met our requirement as well.

Comments

[bkirkbri]

Thanks. You are absolutely correct Cache-Control supersedes the Expires header. Still, we'd like to include it for HTTP/1.0 clients that do not respect Cache-Control.

[bkirkbri]

I should have explained better that we are using CopyObjectRequest to do this now. I've updated the question to reflect this.

[bkirkbri]

The ObjectMetadata.setHeader is marked as internal use only. Have you used this successfully? Our code is only going to be run once, so we won't need to worry about it later if Amazon makes changes. But YMMV.

[Steffen Opel]

Hmmh, I wasn't aware of that restriction indeed, but recall having used Expires: at some point a while back; I might be wrong though, insofar I often use other SDKs for interacting with S3 (e.g. C#/Python, which do definitely support this) and could have mixed that up - the code itself doesn't differ from the other setXYZHeader() methods currently (see ObjectMetadata.java), so the restriction would be based on a non visible side effect, if any.

[bkirkbri]

It's probably used by the other header methods internally and internal only because only certain headers will work. They should add a setExpires method on ObjectMetadata as it's the only header that doesn't have it's own method.

[Dima Deplov]

@SteffenOpel What is the "right" date for Expires value? 2042 or 2014 or 2024? What is depends from this date? Is this a reason to set date to 2042 (or any far date)?

[Steffen Opel]

@flinth - there's no 'right' date (I just made that up back then), it entirely depends on your use case; you might want to read Yahoo's classic Best Practices for Speeding Up Your Web Site though, which explains one specific use case for implementing a "Never expire" policy by setting far future Expires header for static components, see section Add an Expires or a Cache-Control Header on the linked page for details - please also check Controlling Freshness with the Expires HTTP Header in case.

[Dima Deplov]

@SteffenOpel thanks for your reply, I will check this sources, right now I asked a question about cache-control and expires, maybe this links helps me, thank you. May be you'll find interesting my question. stackoverflow.com/questions/18642446/…