Amazon S3 lifecycle retroactive application

Question

Fairly straightforward question. Do Amazon S3 lifecycle rules that I set get applied to data retroactively? If so, what sort of delay might I see before older data begins to be archived or deleted?

By way of example, let's say I have a bucket with 3 years of backed up data. If I create a new lifecycle where that data will be archived after 31 days, and deleted after 365 days, will that new rule be applied to the existing data? How soon will it begin to be enforced?

Solution 1

Yes it's retroactive (i.e. all the things that are already there and match will get the rule).
There may be a slight delay (i.e. rules will have day granularity and run on daily basis), but the rules will take effect immediately. Depending on how much data you have to remove/move it may take a while if you have a lot of pre-existing data.

Source: S3 FAQ here: http://aws.amazon.com/s3/faqs/

Lifecycle policies apply to both existing and new S3 objects, ensuring that you can optimize storage and maximize cost savings for all current data and any new data placed in S3 without time-consuming manual data review and migration.

After an Object Expiration rule is added, the rule is applied to objects that already exist in the bucket as well as new objects added to the bucket. Once objects are past their expiration date, they are identified and queued for removal. You will not be billed for storage for objects on or after their expiration date, though you may still be able to access those objects while they are in queue before they are removed. As with standard delete requests, Amazon S3 doesn’t charge you for removing objects using Object Expiration. You can set Expiration rules for your versioning-enabled or versioning-suspended buckets as well.

Solution 2

S3 bucket lifecycle rules apply to both existing and new data in S3. In addiiton to Mircea answer:

• Lifecycle rules run once a day at midnight Universal Coordinated Time (UTC). For example, if you create a lifecycle rule on October 15 at 4:00 PM Pacific Standard Time (PST) that transitions objects after one day, then the objects are transitioned to the new storage class on October 17 at 12:00 AM UTC.
• New lifecycle rules can take up to 48 hours to complete the first run. If there are millions of objects for the rule, then it might take a few runs for all the objects to be transitioned into another storage class.

Comments

[Marco Arruda]

Is there any cost applied to the first-run lifecycle for existing objects? Let's suppose I have 1TB bucket on Standard Storage. After I create my lifecycle, 50% of my objects fall into its transition rule to Standard-IA. Does it apply any costs? (transfer, requests, etc..)?

[Marco Arruda]

Is there any cost applied to the first-run lifecycle for existing objects? Let's suppose I have 1TB bucket on Standard Storage. After I create my lifecycle, 50% of my objects fall into its transition rule to Standard-IA. Does it apply any costs? (transfer, requests, etc..)?

[Ihor Konovalenko]

@MarcoArruda According to AWS docs, "There are per-request ingest fees when using PUT, COPY, or lifecycle rules to move data into any S3 storage class". A transition request is charged per object when an object becomes eligible for transition according to the Lifecycle rule. So consider the ingest or transition cost before moving objects into any storage class. For prices see: aws.amazon.com/s3/pricing (page Requests & data retrievals). Also may be usful link: aws.amazon.com/s3/faqs (see "S3 Lifecycle Management" chapter)

[Marco Arruda]

Thanks for the answer. I think the one applied in my case is the Lifecycle Transition requests (per 1,000 requests), showing in the tab Requests and data retrieval.

[ddelange]

Yes, there will be PUT requests to move objects to a different storage class. Here's an example calculation