Setting lifecycle configuration on a bucket - Amazon Simple Storage Service

Setting lifecycle configuration on a bucket

This section explains how you can set a S3 Lifecycle configuration on a bucket using AWS SDKs, the AWS CLI, or the S3 console. For information about S3 Lifecycle configuration, see Managing your storage lifecycle.

You can use lifecycle rules to define actions that you want Amazon S3 to take during an object's lifetime (for example, transition objects to another storage class, archive them, or delete them after a specified period of time).

Before you set a lifecycle configuration, note the following:

Propagation delay

When you add an S3 Lifecycle configuration to a bucket, there is usually some lag before a new or updated Lifecycle configuration is fully propagated to all the Amazon S3 systems. Expect a delay of a few minutes before the configuration fully takes effect. This delay can also occur when you delete an S3 Lifecycle configuration.

Disabling or deleting Lifecycle rules

When you disable or delete Lifecycle rules, Amazon S3 stops scheduling new objects for deletion or transition after a small delay. Any objects that were already scheduled are unscheduled and are not deleted or transitioned.

Existing and new objects

When you add a Lifecycle configuration to a bucket, the configuration rules apply to both existing objects and objects that you add later. For example, if you add a Lifecycle configuration rule today with an expiration action that causes objects with a specific prefix to expire 30 days after creation, Amazon S3 will queue for removal any existing objects that are more than 30 days old.

Changes in billing

There may be a lag between when the Lifecycle configuration rules are satisfied and when the action triggered by satisfying the rule is taken. However, changes in billing happen as soon as the Lifecycle configuration rule is satisfied, even if the action is not yet taken.

For example, after the object expiration time, you are not charged for storage, even if the object is not deleted immediately. Another example, as soon as the object transition time elapses, you are charged Amazon S3 Glacier storage rates, even if the object is not immediately transitioned to the S3 Glacier storage class. Lifecycle transitions to the S3 Intelligent-Tiering storage class are the exception. Changes in billing do not happen until the object has transitioned into the S3 Intelligent-Tiering storage class.

You can define a lifecycle rules for all objects or a subset of objects in the bucket by using a shared prefix (objects names that begin with a common string) or a tag. Using a lifecycle rule you can define actions specific to current and non-current object versions. For more information, see the following:

To create a lifecycle rule

  1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.

  2. In the Buckets list, choose the name of the bucket that you want to create a lifecycle rule for.

  3. Choose the Management tab, and choose Create lifecycle rule.

  4. In Lifecycle rule name, enter a name for your rule.

    The name must be unique within the bucket.

  5. Choose the scope of the lifecycle rule:

    • To apply this lifecycle rule to all objects with a specific prefix or tag, choose Limit the scope to specific prefixes or tags.

      • To limit the scope by prefix, in Prefix, enter the prefix.

      • To limit the scope by tag, choose Add tag, and enter the tag key and value.

      For more information about object name prefixes, see Creating object key names. For more information about object tags, see Categorizing your storage using tags.

    • To apply this lifecycle rule to all objects in the bucket, choose This rule applies to all objects in the bucket, and choose I acknowledge that this rule applies to all objects in the bucket.

  6. Under Lifecycle rule actions, choose the actions that you want your lifecycle rule to perform:

    • Transition current versions of objects between storage classes

    • Transition previous versions of objects between storage classes

    • Expire current versions of objects

    • Permanently delete previous versions of objects

    • Delete expired delete markers or incomplete multipart uploads

    Depending on the actions that you choose, different options appear.

  7. To transition current versions of objects between storage classes, under Transition current versions of objects between storage classes:

    1. In Storage class transitions, choose the storage class to transition to:

      • Standard-IA

      • Intelligent-Tiering

      • One Zone-IA

      • Glacier

      • Glacier Deep Archive

    2. In Days after object creation, enter the number of days after creation to transition the object.

    For more information about storage classes, see Using Amazon S3 storage classes. You can define transitions for current or previous object versions or for both current and previous versions. Versioning enables you to keep multiple versions of an object in one bucket. For more information about versioning, see Using the S3 console.

    Important

    When you choose the Glacier or Glacier Deep Archive storage class, your objects remain in Amazon S3. You cannot access them directly through the separate Amazon S3 Glacier service. For more information, see Transitioning objects using Amazon S3 Lifecycle.

  8. To transition non-current versions of objects between storage classes, under Transition non-current versions of objects between storage classes:

    1. In Storage class transitions, choose the storage class to transition to:

      • Standard-IA

      • Intelligent-Tiering

      • One Zone-IA

      • Glacier

      • Glacier Deep Archive

    2. In Days after object becomes non-current, enter the number of days after creation to transition the object.

  9. To expire current versions of objects, under Expire previous versions of objects, in Number of days after object creation, enter the number of days.

    Important

    In a non-versioned bucket the expiration action results in Amazon S3 permanently removing the object. For more information about lifecycle actions, see Elements to describe lifecycle actions.

  10. To permanently delete previous versions of objects, under Permanently delete previous versions of objects, in Number of days after objects become previous versions, enter the number of days.

  11. Under Delete expired delete markers or incomplete multipart uploads, choose Delete expired object delete markers and Delete incomplete multipart uploads. Then, enter the number of days after the multipart upload initiation that you want to end and clean up incomplete multipart uploads.

    For more information about multipart uploads, see Uploading and copying objects using multipart upload.

  12. Choose Create rule.

    If the rule does not contain any errors, Amazon S3 enables it, and you can see it on the Management tab under Lifecycle rules.

For information about CloudFormation templates and examples, see Working with AWS CloudFormation templates and AWS::S3::Bucket in the AWS CloudFormation User Guide.

You can use the following AWS CLI commands to manage S3 Lifecycle configurations:

  • put-bucket-lifecycle-configuration

  • get-bucket-lifecycle-configuration

  • delete-bucket-lifecycle

For instructions on setting up the AWS CLI, see Developing with Amazon S3 using the AWS CLI.

The Amazon S3 Lifecycle configuration is an XML file. But when using the AWS CLI, you cannot specify the XML. You must specify the JSON instead. The following are example XML Lifecycle configurations and equivalent JSON that you can specify in an AWS CLIcommand.

Consider the following example S3 Lifecycle configuration.

Example 1

JSON
{ "Rules": [ { "Filter": { "Prefix": "documents/" }, "Status": "Enabled", "Transitions": [ { "Days": 365, "StorageClass": "GLACIER" } ], "Expiration": { "Days": 3650 }, "ID": "ExampleRule" } ] }
XML
<LifecycleConfiguration> <Rule> <ID>ExampleRule</ID> <Filter> <Prefix>documents/</Prefix> </Filter> <Status>Enabled</Status> <Transition> <Days>365</Days> <StorageClass>GLACIER</StorageClass> </Transition> <Expiration> <Days>3650</Days> </Expiration> </Rule> </LifecycleConfiguration>

Example 2

JSON
{ "Rules": [ { "ID": "id-1", "Filter": { "And": { "Prefix": "myprefix", "Tags": [ { "Value": "mytagvalue1", "Key": "mytagkey1" }, { "Value": "mytagvalue2", "Key": "mytagkey2" } ] } }, "Status": "Enabled", "Expiration": { "Days": 1 } } ] }
XML
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>id-1</ID> <Expiration> <Days>1</Days> </Expiration> <Filter> <And> <Prefix>myprefix</Prefix> <Tag> <Key>mytagkey1</Key> <Value>mytagvalue1</Value> </Tag> <Tag> <Key>mytagkey2</Key> <Value>mytagvalue2</Value> </Tag> </And> </Filter> <Status>Enabled</Status> </Rule> </LifecycleConfiguration>

You can test the put-bucket-lifecycle-configuration as follows.

To test the configuration

  1. Save the JSON Lifecycle configuration in a file (lifecycle.json).

  2. Run the following AWS CLI command to set the Lifecycle configuration on your bucket.

    $ aws s3api put-bucket-lifecycle-configuration  \ --bucket bucketname  \ --lifecycle-configuration file://lifecycle.json
  3. To verify, retrieve the S3 Lifecycle configuration using the get-bucket-lifecycle-configuration AWS CLI command as follows.

    $ aws s3api get-bucket-lifecycle-configuration  \ --bucket bucketname
  4. To delete the S3 Lifecycle configuration use the delete-bucket-lifecycle AWS CLI command as follows.

    aws s3api delete-bucket-lifecycle \ --bucket bucketname
Java

You can use the AWS SDK for Java to manage the S3 Lifecycle configuration of a bucket. For more information about managing S3 Lifecycle configuration, see Managing your storage lifecycle.

Note

When you add S3 Lifecycle configuration to a bucket, Amazon S3 replaces the bucket's current Lifecycle configuration, if there is one. To update a configuration, you retrieve it, make the desired changes, and then add the revised configuration to the bucket.

The following example shows how to use the AWS SDK for Java to add, update, and delete the Lifecycle configuration of a bucket. The example does the following:

  • Adds a Lifecycle configuration to a bucket.

  • Retrieves the Lifecycle configuration and updates it by adding another rule.

  • Adds the modified Lifecycle configuration to the bucket. Amazon S3 replaces the existing configuration.

  • Retrieves the configuration again and verifies that it has the right number of rules by the printing number of rules.

  • Deletes the Lifecycle configuration and verifies that it has been deleted by attempting to retrieve it again.

For instructions on creating and testing a working sample, see Testing the Amazon S3 Java Code Examples.

import com.amazonaws.AmazonServiceException; import com.amazonaws.SdkClientException; import com.amazonaws.auth.profile.ProfileCredentialsProvider; import com.amazonaws.regions.Regions; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3ClientBuilder; import com.amazonaws.services.s3.model.BucketLifecycleConfiguration; import com.amazonaws.services.s3.model.BucketLifecycleConfiguration.Transition; import com.amazonaws.services.s3.model.StorageClass; import com.amazonaws.services.s3.model.Tag; import com.amazonaws.services.s3.model.lifecycle.LifecycleAndOperator; import com.amazonaws.services.s3.model.lifecycle.LifecycleFilter; import com.amazonaws.services.s3.model.lifecycle.LifecyclePrefixPredicate; import com.amazonaws.services.s3.model.lifecycle.LifecycleTagPredicate; import java.io.IOException; import java.util.Arrays; public class LifecycleConfiguration { public static void main(String[] args) throws IOException { Regions clientRegion = Regions.DEFAULT_REGION; String bucketName = "*** Bucket name ***"; // Create a rule to archive objects with the "glacierobjects/" prefix to Glacier immediately. BucketLifecycleConfiguration.Rule rule1 = new BucketLifecycleConfiguration.Rule() .withId("Archive immediately rule") .withFilter(new LifecycleFilter(new LifecyclePrefixPredicate("glacierobjects/"))) .addTransition(new Transition().withDays(0).withStorageClass(StorageClass.Glacier)) .withStatus(BucketLifecycleConfiguration.ENABLED); // Create a rule to transition objects to the Standard-Infrequent Access storage class // after 30 days, then to Glacier after 365 days. Amazon S3 will delete the objects after 3650 days. // The rule applies to all objects with the tag "archive" set to "true". BucketLifecycleConfiguration.Rule rule2 = new BucketLifecycleConfiguration.Rule() .withId("Archive and then delete rule") .withFilter(new LifecycleFilter(new LifecycleTagPredicate(new Tag("archive", "true")))) .addTransition(new Transition().withDays(30).withStorageClass(StorageClass.StandardInfrequentAccess)) .addTransition(new Transition().withDays(365).withStorageClass(StorageClass.Glacier)) .withExpirationInDays(3650) .withStatus(BucketLifecycleConfiguration.ENABLED); // Add the rules to a new BucketLifecycleConfiguration. BucketLifecycleConfiguration configuration = new BucketLifecycleConfiguration() .withRules(Arrays.asList(rule1, rule2)); try { AmazonS3 s3Client = AmazonS3ClientBuilder.standard() .withCredentials(new ProfileCredentialsProvider()) .withRegion(clientRegion) .build(); // Save the configuration. s3Client.setBucketLifecycleConfiguration(bucketName, configuration); // Retrieve the configuration. configuration = s3Client.getBucketLifecycleConfiguration(bucketName); // Add a new rule with both a prefix predicate and a tag predicate. configuration.getRules().add(new BucketLifecycleConfiguration.Rule().withId("NewRule") .withFilter(new LifecycleFilter(new LifecycleAndOperator( Arrays.asList(new LifecyclePrefixPredicate("YearlyDocuments/"), new LifecycleTagPredicate(new Tag("expire_after", "ten_years")))))) .withExpirationInDays(3650) .withStatus(BucketLifecycleConfiguration.ENABLED)); // Save the configuration. s3Client.setBucketLifecycleConfiguration(bucketName, configuration); // Retrieve the configuration. configuration = s3Client.getBucketLifecycleConfiguration(bucketName); // Verify that the configuration now has three rules. configuration = s3Client.getBucketLifecycleConfiguration(bucketName); System.out.println("Expected # of rules = 3; found: " + configuration.getRules().size()); // Delete the configuration. s3Client.deleteBucketLifecycleConfiguration(bucketName); // Verify that the configuration has been deleted by attempting to retrieve it. configuration = s3Client.getBucketLifecycleConfiguration(bucketName); String s = (configuration == null) ? "No configuration found." : "Configuration found."; System.out.println(s); } catch (AmazonServiceException e) { // The call was transmitted successfully, but Amazon S3 couldn't process // it, so it returned an error response. e.printStackTrace(); } catch (SdkClientException e) { // Amazon S3 couldn't be contacted for a response, or the client // couldn't parse the response from Amazon S3. e.printStackTrace(); } } }
.NET

You can use the AWS SDK for .NET to manage the S3 Lifecycle configuration on a bucket. For more information about managing Lifecycle configuration, see Managing your storage lifecycle.

Note

When you add a Lifecycle configuration, Amazon S3 replaces the existing configuration on the specified bucket. To update a configuration, you must first retrieve the Lifecycle configuration, make the changes, and then add the revised Lifecycle configuration to the bucket.

The following example shows how to use the AWS SDK for .NET to add, update, and delete a bucket's Lifecycle configuration. The code example does the following:

  • Adds a Lifecycle configuration to a bucket.

  • Retrieves the Lifecycle configuration and updates it by adding another rule.

  • Adds the modified Lifecycle configuration to the bucket. Amazon S3 replaces the existing Lifecycle configuration.

  • Retrieves the configuration again and verifies it by printing the number of rules in the configuration.

  • Deletes the Lifecycle configuration.and verifies the deletion.

For instructions on how to create and test a working sample, see Running the Amazon S3 .NET Code Examples.

using Amazon; using Amazon.S3; using Amazon.S3.Model; using System; using System.Collections.Generic; using System.Threading.Tasks; namespace Amazon.DocSamples.S3 { class LifecycleTest { private const string bucketName = "*** bucket name ***"; // Specify your bucket region (an example region is shown). private static readonly RegionEndpoint bucketRegion = RegionEndpoint.USWest2; private static IAmazonS3 client; public static void Main() { client = new AmazonS3Client(bucketRegion); AddUpdateDeleteLifecycleConfigAsync().Wait(); } private static async Task AddUpdateDeleteLifecycleConfigAsync() { try { var lifeCycleConfiguration = new LifecycleConfiguration() { Rules = new List<LifecycleRule> { new LifecycleRule { Id = "Archive immediately rule", Filter = new LifecycleFilter() { LifecycleFilterPredicate = new LifecyclePrefixPredicate() { Prefix = "glacierobjects/" } }, Status = LifecycleRuleStatus.Enabled, Transitions = new List<LifecycleTransition> { new LifecycleTransition { Days = 0, StorageClass = S3StorageClass.Glacier } }, }, new LifecycleRule { Id = "Archive and then delete rule", Filter = new LifecycleFilter() { LifecycleFilterPredicate = new LifecyclePrefixPredicate() { Prefix = "projectdocs/" } }, Status = LifecycleRuleStatus.Enabled, Transitions = new List<LifecycleTransition> { new LifecycleTransition { Days = 30, StorageClass = S3StorageClass.StandardInfrequentAccess }, new LifecycleTransition { Days = 365, StorageClass = S3StorageClass.Glacier } }, Expiration = new LifecycleRuleExpiration() { Days = 3650 } } } }; // Add the configuration to the bucket. await AddExampleLifecycleConfigAsync(client, lifeCycleConfiguration); // Retrieve an existing configuration. lifeCycleConfiguration = await RetrieveLifecycleConfigAsync(client); // Add a new rule. lifeCycleConfiguration.Rules.Add(new LifecycleRule { Id = "NewRule", Filter = new LifecycleFilter() { LifecycleFilterPredicate = new LifecyclePrefixPredicate() { Prefix = "YearlyDocuments/" } }, Expiration = new LifecycleRuleExpiration() { Days = 3650 } }); // Add the configuration to the bucket. await AddExampleLifecycleConfigAsync(client, lifeCycleConfiguration); // Verify that there are now three rules. lifeCycleConfiguration = await RetrieveLifecycleConfigAsync(client); Console.WriteLine("Expected # of rulest=3; found:{0}", lifeCycleConfiguration.Rules.Count); // Delete the configuration. await RemoveLifecycleConfigAsync(client); // Retrieve a nonexistent configuration. lifeCycleConfiguration = await RetrieveLifecycleConfigAsync(client); } catch (AmazonS3Exception e) { Console.WriteLine("Error encountered ***. Message:'{0}' when writing an object", e.Message); } catch (Exception e) { Console.WriteLine("Unknown encountered on server. Message:'{0}' when writing an object", e.Message); } } static async Task AddExampleLifecycleConfigAsync(IAmazonS3 client, LifecycleConfiguration configuration) { PutLifecycleConfigurationRequest request = new PutLifecycleConfigurationRequest { BucketName = bucketName, Configuration = configuration }; var response = await client.PutLifecycleConfigurationAsync(request); } static async Task<LifecycleConfiguration> RetrieveLifecycleConfigAsync(IAmazonS3 client) { GetLifecycleConfigurationRequest request = new GetLifecycleConfigurationRequest { BucketName = bucketName }; var response = await client.GetLifecycleConfigurationAsync(request); var configuration = response.Configuration; return configuration; } static async Task RemoveLifecycleConfigAsync(IAmazonS3 client) { DeleteLifecycleConfigurationRequest request = new DeleteLifecycleConfigurationRequest { BucketName = bucketName }; await client.DeleteLifecycleConfigurationAsync(request); } } }
Ruby

You can use the AWS SDK for Ruby to manage S3 Lifecycle configuration on a bucket by using the class AWS::S3::BucketLifecycleConfiguration. For more information about using the AWS SDK for Ruby with Amazon S3, see Using the AWS SDK for Ruby - Version 3. For more information about managing lifecycle configuration, see Managing your storage lifecycle.

YThe following sections in the Amazon Simple Storage Service API Reference describe the REST API related to the S3 Lifecycle configuration.