Amazon Simple Storage Service
Developer Guide (API Version 2006-03-01)
« PreviousNext »
View the PDF for this guide.Go to the AWS Discussion Forum for this product.Go to the Kindle Store to download this guide in Kindle format.Did this page help you?  Yes | No |  Tell us about it...

Object Lifecycle Management

Lifecycle management defines how Amazon S3 manages objects during their lifetime.

Some objects that you store in an Amazon S3 bucket might have a well-defined lifecycle:

  • If you are uploading periodic logs to your bucket, your application might need these logs for a week or a month after creation, and after that you might want to delete them.

  • Some documents are frequently accessed for a limited period of time. After that, you might not need real-time access to these objects, but your organization might require you to archive them for a longer period and then optionally delete them later. Digital media archives, financial and healthcare records, raw genomics sequence data, long-term database backups, and data that must be retained for regulatory compliance are some kinds of data that you might upload to Amazon S3 primarily for archival purposes.

For such objects, you can define rules that identify the affected objects, a timeline, and specific actions you want Amazon S3 to perform on the objects.

Amazon S3 manages object lifetimes with a lifecycle configuration, which is assigned to a bucket and defines rules for individual objects. Each rule in a lifecycle configuration consists of the following:

  • An object key prefix that identifies one or more objects to which the rule applies.

  • An action or actions that you want Amazon S3 to perform on the specified objects.

  • A date or a time period, specified in days since object creation, when you want Amazon S3 to perform the specified action.

You can add these rules to your bucket using either the Amazon S3 console or programmatically.

Lifecycle Configuration Rules

A rule can apply to a single object or multiple objects whose key name begins with the prefix specified in the rule. For example, suppose you have the following objects:

logs/day1

logs/day2

logs/day3

ExampleObject.jpg

If you specify ExampleObject.jpg as the prefix, the rule applies to the specific object. If you specify logs/ as the prefix, the rule applies to the three objects whose key name begins with the string "logs/".

A rule can specify the following actions:

  • Transition— When a specified date or time period in the object’s lifetime is reached, Amazon S3 sets the storage class to Glacier. For more information about storage classes, see Storage Classes.

    For example, you can set a rule for Amazon S3 to transition "MyArchiveObject.jpg" to the Glacier storage class 30 days after creation or to transition all objects with the key prefix "glacierobjects/" to the Glacier storage class a year after creation.

  • Expiration—When the specified time period is reached in the object's lifetime, Amazon S3 deletes it.

    For example, you can set a rule with an expiration action so that Amazon S3 will delete objects with the key prefix "log2012-01-01" 30 days after creation.

    Expiration applies to all Amazon S3 objects, including those that are archived in Amazon Glacier.

You can specify a date or a time period in days since object creation when the action will be taken on the specified object or objects.

Note

In Amazon S3 you cannot modify objects. If you replace an existing object, it is a DELETE followed by a PUT operation, and therefore you are creating a new object. So in Amazon S3 the last modified date is same as object creation date, and Amazon S3 uses this date in its calculation.

  • When you specify a number of days, Amazon S3 calculates the time by adding the number of days specified in the rule to the object creation time and rounding the resulting time to the next day midnight UTC. For example, if an object was created on 1/15/2012 10:30 a.m. UTC and you specify 3 days in a transition rule, then the transition date of the object would be calculated as 1/19/2012 00:00 UTC.

  • When you specify a date in a rule, the specified action is executed on the specified date. That is, Amazon S3 transitions or expires the objects on the specified date.

These rules are available as a lifecycle subresource on the bucket. The following is an example lifecycle configuration:

<LifecycleConfiguration>
  <Rule>
    <ID>Example Rule</ID>
    <Prefix>projectdocs/</Prefix>
    <Status>Enabled</Status>
    <Transition>
      <Days>365</Days>
      <StorageClass>GLACIER</StorageClass>
    </Transition>
    <Expiration>
      <Days>3650</Days>
    </Expiration>
  </Rule>
</LifecycleConfiguration>

A lifecycle configuration can contain as many as 1000 rules. The preceding rule defines a Transition action and an Expiration action. The rule applies to all objects whose key name starts with the prefix projectdocs/. The rule requests Amazon S3 to transition objects to the Glacier storage class 365 days after creation and to delete the objects 3650 days after creation.

Important

In the preceding example, the time period specified in both the transition and expiration actions is relative to the date the objects are created.

There is usually some lag before a new or updated lifecycle configuration is fully propagated to all the Amazon S3 systems. Expect a delay of a few minutes before the lifecycle configuration fully takes effect. This delay can also occur when you delete a lifecycle configuration.

Note

If your bucket is version-enabled or versioning is suspended, you cannot add a lifecycle configuration.

When you add a lifecycle configuration to a bucket, the configuration rules apply to both existing objects and to objects that you add later. For example, if you add a lifecycle configuration rule with an expiration action today that causes objects with a specific prefix to expire 30 days after creation, Amazon S3 will queue for removal any existing objects that are more than 30 days old.

Before You Decide to Archive Objects

The Transition action in the lifecycle configuration rule enables you to transition objects to the Glacier storage class—that is, archive data to Amazon Glacier. Before you decide to archive objects, note the following:

  • Objects in the Glacier storage class are not available in real time.

    Archived objects are Amazon S3 objects, but before you can access an archived object, you must first restore a temporary copy of it. The restored object copy is available only for the duration you specify in the restore request. After that, Amazon S3 deletes the temporary copy, and the object remains archived in Amazon Glacier.

    Note that object restoration from an archive can take from three to five hours.

  • The transition action allows only one-way transition to the Glacier storage class.

    You cannot use a lifecycle configuration rule to convert a Glacier object to a Standard or Reduced Redundancy Storage (RRS) object. If you want to change the storage class of an already archived object to either Standard or RRS, you must use the restore operation to make a temporary copy first. Then use the copy operation to overwrite the object as the Standard or the RRS object.

  • Glacier objects are visible and available only through Amazon S3, not through Amazon Glacier

    Amazon S3 stores the archived objects in Amazon Glacier; however, these are Amazon S3 objects, and you can access them only by using the Amazon S3 console or the API. You cannot access the archived objects through the Amazon Glacier console or the API.

  • A rule with an empty key prefix applies to all the objects in the bucket

    If you specify an empty prefix, the rule applies to all objects in the bucket. So if the rule specifies a transition action with an empty prefix, you are requesting Amazon S3 to archive all the objects in the bucket at a specific period in the objects' lifetime.

For more information, see Object Archival (Transition Objects to the Glacier Storage Class). For information about Amazon S3 storage classes, see Storage Classes.

Before You Decide to Expire Objects

The Expiration action in the lifecycle configuration rule enables you to request Amazon S3 to delete objects. Before you decide to set this action in the lifecycle configuration, note the following:

  • The Expiration action deletes objects

    You might have objects in Amazon S3 or archived to Amazon Glacier. No matter where these objects are, Amazon S3 will delete them. You will no longer be able to access these objects.

  • A rule with an empty key prefix applies to all the objects in the bucket

    If you specify an empty prefix, the rule applies to all objects in the bucket. So if the rule specifies an expiration action with an empty prefix, you are requesting Amazon S3 to delete all the objects in the bucket at a specific period in the objects' lifetime.

Specifying a Lifecycle Configuration

You can set a lifecycle configuration on a bucket either programmatically or by using the Amazon S3 console.

Specify a Lifecycle Configuration Using the Amazon S3 Console

With the Amazon S3 console, you can configure as many as 1000 lifecycle rules for objects in a bucket. For more information, see Manage Object Lifecycle Using the AWS Management Console.

Specify a Lifecycle Configuration Programmatically

The Amazon S3 provides API actions to add a lifecycle configuration to the bucket. Amazon S3 stores the lifecycle configuration in the lifecycle subresource on your bucket. For more information, see Manage Object Lifecycle Using the REST API.

You can also use the AWS SDKs to add a lifecycle configuration to your bucket.

Whether you are using the Amazon S3 API or the AWS SDK to manage your lifecycle configuration, the process for defining it is the same. A lifecycle configuration is specified in XML. It contains a set of rules, as shown in the following fragment:

<LifecycleConfiguration>
  <Rule>
      ...
  </Rule>
  <Rule>
      ...
  </Rule>
   ...
</LifecycleConfiguration>

Each rule identifies an object or group of objects by key prefix, and it specifies an action for Amazon S3 to perform at a specified time or after a specified time period following object creation. The following XML elements define a rule:

  • <ID> element—A unique identifier for the rule.

  • <Status> element— Enabled or Disabled. Amazon S3 will not evaluate any disabled rules.

  • <Prefix> element—A key prefix that identifies an object or objects to which the rule applies. If you do not supply a Prefix value, the rule applies to all objects in the bucket.

  • Action elements—In a single rule, you can define either or both the following actions, although in a rule an action can appear at most once.

    • <Expiration> action—Defines in terms of a specific date or number of days since creation when the object lifetime expires. It can have <Date> or <Days> element children.

    • <Transition> action—Defines in terms of date or days since creation when the object transition to the Glacier storage class. It can have <Date> or <Days> element children.

The following are examples of the lifecycle configuration. For more information about the REST API or the AWS SDKs, click one of the links provided at the beginning of this section.

Example 1: Specify a Rule

The following lifecycle configuration has a rule with two actions. It requests Amazon S3 to transition objects with key prefix projectdocs/ to Amazon Glacier 365 days after creation and delete those objects 3650 days after creation.

<LifecycleConfiguration>
  <Rule>
    <ID>Transition and Expiration Rule</ID>
    <Prefix>projectdocs/</Prefix>
    <Status>Enabled</Status>
    <Transition>
      <Days>365</Days>
      <StorageClass>GLACIER</StorageClass>
    </Transition>
    <Expiration>
      <Days>3650</Days>
    </Expiration>
  </Rule>
</LifecycleConfiguration>

Instead of a time period, you can specify a date for each action; however, you cannot use both a date and a time period in the same rule.

In the preceding example, there is one rule with two actions. You can achieve the same result by defining two rules for one object, each of which performs one of the actions:

<LifecycleConfiguration>
  <Rule>
    <ID>Transition Rule</ID>
    <Prefix>projectdocs/</Prefix>
    <Status>Enabled</Status>
    <Transition>
      <Days>365</Days>
      <StorageClass>GLACIER</StorageClass>
    </Transition>
  </Rule>
  <Rule>
    <ID>Expiration Rule</ID>
    <Prefix>projectdocs/</Prefix>
    <Status>Enabled</Status>
    <Expiration>
      <Days>3650</Days>
    </Expiration>
  </Rule>
</LifecycleConfiguration>

Example 2: Specify a Rule that Applies to All Objects in the Bucket

The following lifecycle configuration specifies an empty prefix and zero transition days. If you don't specify a prefix, the rule applies to all objects in the bucket. Because the Days element is set to 0, all objects are immediately eligible for archival to Amazon Glacier, and Amazon S3 archives these objects using processes that run periodically.

<LifecycleConfiguration>
  <Rule>
    <ID>Archive all object immediately upon creation</ID>
    <Prefix></Prefix>
    <Status>Enabled</Status>
    <Transition>
      <Days>0</Days>
      <StorageClass>GLACIER</StorageClass>
    </Transition>
  </Rule>
</LifecycleConfiguration>

Example 3: Disabling a Rule

The following lifecycle configuration specifies two rules; however, one of them is disabled. Amazon S3 will not perform any action specified in a rule that is disabled.

<LifecycleConfiguration>
  <Rule>
    <ID>30 days log objects expire rule</ID>
    <Prefix>logs/</Prefix>
    <Status>Enabled</Status>
    <Expiration>
      <Days>30</Days>
    </Expiration>
  </Rule>
  <Rule>
    <ID>1 year documents expire rule</ID>
    <Prefix>documents/</Prefix>
    <Status>Disabled</Status>
    <Expiration>
      <Days>365</Days>
    </Expiration>
  </Rule>
</LifecycleConfiguration>

Example 4: Overlapping Rules Not Allowed

Rules cannot overlap. For example, the following lifecycle configuration has a rule that sets objects with the prefix "documents/" to expire after 30 days. In another rule, objects with the prefix "documents/2011" are set to expire after 365 days. In this case, Amazon S3 returns an error message.

<LifecycleConfiguration>
  <Rule>
    <ID>111</ID>
    <Prefix>documents/</Prefix>
    <Status>Enabled</Status>
    <Expiration>
      <Days>30</Days>
    </Expiration>
  </Rule>
  <Rule>
    <ID>222</ID>
    <Prefix>documents/2011</Prefix>
    <Status>Enabled</Status>
    <Expiration>
      <Days>365</Days>
    </Expiration>
  </Rule>
</LifecycleConfiguration>