Lifecycle management in Security Lake - Amazon Security Lake
Retention managementRollup Regions

Lifecycle management in Security Lake

You can customize Security Lake to store data in your preferred AWS Regions for your preferred amount of time. Lifecycle management can help you comply with different compliance requirements.

Retention management

To manage your data so that it is stored cost effectively, you can configure retention settings for the data. Because Security Lake stores your data as objects in Amazon Simple Storage Service (Amazon S3) buckets, the retention settings correspond to an Amazon S3 Lifecycle configuration. By configuring these settings, you can specify your preferred Amazon S3 storage class and the time period for S3 objects to stay in that storage class before they transition to a different storage class or expire. For more information about Amazon S3 Lifecycle configurations, see Managing your storage lifecycle in the Amazon Simple Storage Service User Guide.

In Security Lake, you specify retention settings at the Region level. For example, you might choose to transition all S3 objects in a specific AWS Region to the S3 Standard-IA storage class 30 days after they're written to the data lake. The default Amazon S3 storage class is S3 Standard.

Important

Security Lake doesn't support Amazon S3 Object Lock. When the data lake buckets are created, S3 Object Lock is disabled by default. Enabling S3 Object Lock with default retention mode interrupts the delivery of normalized log data to the data lake.

Configuring retention settings when enabling Security Lake

Follow these instructions to configure retention settings for one or more Regions when you're onboarding to Security Lake. If you don't configure retention settings, Security Lake uses the default settings for an Amazon S3 Lifecycle configuration—store the data indefinitely using the S3 Standard storage class.

Console

  1. Open the Security Lake console at https://console.aws.amazon.com/securitylake/.

  2. When you reach Step 2: Define target objective of the onboarding workflow, choose Add transition under Select storage classes. Then choose the Amazon S3 storage class that you want to transition S3 objects to. (The unlisted, default storage class is S3 Standard.) Also specify a retention period (in days) for that storage class. To transition objects to another storage class after that time, choose Add transition and enter the settings for the subsequent storage class and retention period.

  3. To specify when you want S3 objects to expire, choose Add transition. Then, for storage class, choose Expire. For retention period, enter the total number of days that you want to store objects in Amazon S3, using any storage class, after objects are created. When this time period ends, objects expire and Amazon S3 deletes them.

  4. When you finish, choose Next.

Your changes will apply to all the Regions that you enabled Security Lake in during earlier onboarding steps.

API

To configure retention settings programmatically when you're onboarding to Security Lake, use the CreateDataLake operation of the Security Lake API. If you're using the AWS CLI, run the create-data-lake command. Specify the retention settings you want in the lifecycleConfiguration parameters as follows:

  • For transitions, specify the total number of days (days) that you want to store S3 objects in a particular Amazon S3 storage class (storageClass).

  • For expiration, specify the total number of days that you want to store objects in Amazon S3, using any storage class, after objects are created. When this time period ends, objects expire and Amazon S3 deletes them.

Security Lake applies the settings to the Region that you specify in the region field of the configurations object.

For example, the following command enables Security Lake in the us-east-1 Region. In this Region, objects expire after 365 days, and objects transition to the ONEZONE_IA S3 storage class after 60 days. This example is formatted for Linux, macOS, or Unix, and it uses the backslash (\) line-continuation character to improve readability.

$ aws securitylake create-data-lake \
--configurations '[{"encryptionConfiguration": {"kmsKeyId":"S3_MANAGED_KEY"},"region":"us-east-1","lifecycleConfiguration": {"expiration":{"days":365},"transitions":[{"days":60,"storageClass":"ONEZONE_IA"}]}}]' \
--meta-store-manager-role-arn "arn:aws:securitylake:ap-northeast-2:123456789012:data-lake/default"

Updating retention settings

Follow these instructions to update retention settings for one or more Regions after enabling Security Lake.

Console

  1. Open the Security Lake console at https://console.aws.amazon.com/securitylake/.

  2. In the navigation pane, choose Regions

  3. Select a Region, and then choose Edit.

  4. In the Select storage classes section, enter the settings that you want. For storage class, choose the Amazon S3 storage class that you want to transition S3 objects to. (The unlisted, default storage class is S3 Standard.) For retention period, enter the number of days that you want to store objects in that storage class. You can specify multiple transitions.

    To also specify when you want S3 objects to expire, choose Expire for storage class. Then, for retention period, enter the total number of days that you want to store objects in Amazon S3, using any storage class, after objects are created. When this time period ends, objects expire and Amazon S3 deletes them.

  5. When you finish, choose Save.

API

To update retention settings programmatically, use the UpdateDataLake operation of the Security Lake API. f you're using the AWS CLI, run the update-data-lake command. In your request, use the lifecycleConfiguration parameter to specify the new settings:

  • To change the transition settings, use the transitions parameters to specify each new time period in days (days) that you want to store S3 objects in a particular Amazon S3 storage class (storageClass).

  • To change the overall retention period, use the expiration parameter to specify the total number of days that you want to store S3 objects, using any storage class, after objects are created. When this retention period ends, objects expire and Amazon S3 deletes them.

Security Lake applies the settings to the Region that you specify in the region field of the configurations object.

For example, the following AWS CLI command updates the data expiration settings and storage transition settings for the us-east-1 Region. In this Region, objects expire after 500 days, and objects transition to the ONEZONE_IA S3 storage class after 30 days. This example is formatted for Linux, macOS, or Unix, and it uses the backslash (\) line-continuation character to improve readability.

$ aws securitylake update-data-lake \
--configurations '[{"encryptionConfiguration": {"kmsKeyId":"S3_MANAGED_KEY"},"region":"us-east-1","lifecycleConfiguration": {"expiration":{"days":500},"transitions":[{"days":30,"storageClass":"ONEZONE_IA"}]}}]' \
--meta-store-manager-role-arn "arn:aws:securitylake:ap-northeast-2:123456789012:data-lake/default"

Rollup Regions

A rollup Region consolidates data from one or more contributing Regions. This can help you comply with regional data compliance requirements.

For instructions on configuring rollup Regions, see Configuring rollup Regions.