Lifecycle management in Security Lake

You can customize Security Lake to store data in your preferred AWS Regions for your preferred amount of time. Lifecycle management can help you comply with different compliance requirements.

Retention management

To manage your data so that it is stored cost effectively, you can configure retention settings for the data. Because Security Lake stores your data as objects in Amazon Simple Storage Service (Amazon S3) buckets, the retention settings correspond to an Amazon S3 Lifecycle configuration. By configuring these settings, you can specify your preferred Amazon S3 storage class and the time period for S3 objects to stay in that storage class before they transition to a different storage class or expire. For more information about Amazon S3 Lifecycle configurations, see Managing your storage lifecycle in the Amazon Simple Storage Service User Guide.

In Security Lake, you specify retention settings at the Region level. For example, you might choose to transition all S3 objects in a specific AWS Region to the S3 Standard-IA storage class 30 days after they're written to the data lake. The default Amazon S3 storage class is S3 Standard.

Note

Security Lake does not support Amazon S3 object locking. When the data lake buckets are created, Amazon S3 object lock is disabled by default. If object locking is enabled on the bucket, delivery of normalized log data to the data lake will be interrupted.

Configuring retention settings when enabling Security Lake

Follow these instructions to configure retention settings for one or more Regions when you're onboarding to Security Lake. If you don't configure retention settings, Security Lake uses the default settings for an Amazon S3 Lifecycle configuration—store the data indefinitely using the S3 Standard storage class.

Console

Open the Security Lake console at https://console.aws.amazon.com/securitylake/.
When you reach Step 2: Define target objective of the onboarding workflow, choose Add transition under Select storage classes. Then choose the Amazon S3 storage class that you want to transition S3 objects to. (The unlisted, default storage class is S3 Standard.) Also specify a retention period (in days) for that storage class. To transition objects to another storage class after that time, choose Add transition and enter the settings for the subsequent storage class and retention period.
To specify when you want S3 objects to expire, choose Add transition. Then, for storage class, choose Expire. For retention period, enter the total number of days that you want to store objects in Amazon S3, using any storage class, after objects are created. When this time period ends, objects expire and Amazon S3 deletes them.
When you finish, choose Next.

Your changes will apply to all the Regions that you enabled Security Lake in during earlier onboarding steps.

API

To configure retention settings programmatically when you're onboarding to Security Lake, use the CreateDataLake operation of the Security Lake API. In your request, use the lifecycleConfiguration parameters to specify the settings that you want:

For transitions, specify the total number of days (days) that you want to store S3 objects in a particular Amazon S3 storage class (storageClass).
For expiration, specify the total number of days that you want to store objects in Amazon S3, using any storage class, after objects are created. When this time period ends, objects expire and Amazon S3 deletes them.

Security Lake applies the settings to the Region that you specify in the region field of the configurations object.

AWS CLI

If you're using the AWS Command Line Interface (AWS CLI) to onboard to Security Lake, run the create-data-lake command. When you run the command, use the lifecycleConfiguration parameters to specify the retention settings that you want:

For transitions, specify the total number of days (days) that you want to store S3 objects in a particular Amazon S3 storage class (storageClass).
For expiration, specify the total number of days that you want to store objects in Amazon S3, using any storage class, after objects are created. When this time period ends, objects expire and Amazon S3 deletes them.

Security Lake applies the settings to the Region that you specify in the region field of the configurations object.

Updating retention settings

Follow these instructions to update retention settings for one or more Regions after enabling Security Lake.

Console

Open the Security Lake console at https://console.aws.amazon.com/securitylake/.
In the navigation pane, choose Regions
Select a Region, and then choose Edit.
In the Select storage classes section, enter the settings that you want. For storage class, choose the Amazon S3 storage class that you want to transition S3 objects to. (The unlisted, default storage class is S3 Standard.) For retention period, enter the number of days that you want to store objects in that storage class. You can specify multiple transitions.

To also specify when you want S3 objects to expire, choose Expire for storage class. Then, for retention period, enter the total number of days that you want to store objects in Amazon S3, using any storage class, after objects are created. When this time period ends, objects expire and Amazon S3 deletes them.
When you finish, choose Save.

API

To update retention settings programmatically, use the UpdateDataLake operation of the Security Lake API. In your request, use the lifecycleConfiguration parameters to specify the new settings:

To change the transition settings, use the transitions parameters to specify each new time period in days (days) that you want to store S3 objects in a particular Amazon S3 storage class (storageClass).
To change the overall retention period, use the expiration parameter to specify the total number of days that you want to store S3 objects, using any storage class, after objects are created. When this retention period ends, objects expire and Amazon S3 deletes them.

Security Lake applies the settings to the Region that you specify in the region field of the configurations object.

AWS CLI

To update retention settings by using the AWS Command Line Interface (AWS CLI), run the update-data-lake command. When you run the command, use the lifecycleConfiguration parameters to specify the new settings:

To change the transition settings, use the transitions parameters to specify each new time period in days (days) that you want to store S3 objects in a particular Amazon S3 storage class (storageClass).
To change the overall retention period, use the expiration parameter to specify the total number of days that you want to store S3 objects, using any storage class, after objects are created. When this retention period ends, objects expire and Amazon S3 deletes them.

Security Lake applies the settings to the Region that you specify in the region field of the configurations object.

Rollup Regions

A rollup Region consolidates data from one or more contributing Regions. This can help you comply with regional data compliance requirements.

Before adding a rollup Region, you first need to create two different roles in AWS Identity and Access Management (IAM):

Prerequisite: IAM role for data replication
Prerequisite: IAM role to register AWS Glue partitions

Note

Security Lake creates these IAM roles or uses existing roles on your behalf when you use the Security Lake console. However, you must create these roles when using the Security Lake API or AWS CLI.

Prerequisite: IAM role for data replication

This IAM role grants permission to Amazon S3 to replicate source logs and events across multiple Regions.

To grant these permissions, create an IAM role that starts with the prefix SecurityLake, and attach the following sample policy to the role. You'll need the Amazon Resource Name (ARN) of the role when you create a rollup Region in Security Lake. In this policy, sourceRegions are contributing Regions, and destinationRegions are rollup Regions.


{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadS3ReplicationSetting",
      "Action": [
        "s3:ListBucket",
        "s3:GetReplicationConfiguration",
        "s3:GetObjectVersionForReplication",
        "s3:GetObjectVersion",
        "s3:GetObjectVersionAcl",
        "s3:GetObjectVersionTagging",
        "s3:GetObjectRetention",
        "s3:GetObjectLegalHold"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::aws-security-data-lake-[[sourceRegions]]*",
        "arn:aws:s3:::aws-security-data-lake-[[sourceRegions]]*/*"
      ],
      "Condition": {
        "StringEquals": {
          "s3:ResourceAccount": [
            "{{bucketOwnerAccountId}}"
          ]
        }
      }
    },
    {
      "Sid": "AllowS3Replication",
      "Action": [
        "s3:ReplicateObject",
        "s3:ReplicateDelete",
        "s3:ReplicateTags"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::aws-security-data-lake-[[destinationRegions]]*/*"
      ],
      "Condition": {
        "StringEquals": {
          "s3:ResourceAccount": [
            "{{bucketOwnerAccountId}}"
          ]
        }
      }
    }
  ]
}

For more information on replication roles, see Setting up permissions in the Amazon Simple Storage Service User Guide.

Prerequisite: IAM role to register AWS Glue partitions

This IAM role grants permissions for the partition updater AWS Lambda function used by Security Lake to register AWS Glue partitions for the S3 objects that were replicated from other regions. Without creating this role, subscribers can't query events from those objects.

To grant these permissions, create a role named AmazonSecurityLakeMetaStoreManager (you may have already created this role while onboarding to Security Lake). For more information about this role, including a sample policy, see Create necessary IAM roles.

In the Lake Formation console, you need to grant AmazonSecurityLakeMetaStoreManager permission as a data lake administrator by following these steps:

Open the Lake Formation console at https://console.aws.amazon.com/lakeformation/.
Sign in as an administrative user.
If a Welcome to Lake Formation window appears, choose the user that you created or selected in Step 1, and then choose Get started.
If you don't see a Welcome to Lake Formation window, then perform the following steps to configure a Lake Formation Administrator.
1. In the navigation pane, under Permissions, choose Administrative Roles and tasks. In the Data lake administrators section of the console page, choose Choose administrators.
2. In the Manage data lake administrators dialog box, for IAM users and roles, choose the AmazonSecurityLakeMetaStoreManager IAM role that you created, and then choose Save.

For more information about changing permission for data lake administrators, see Create a data lake administrator in the AWS Lake Formation Developer Guide.

Configuring rollup Regions when creating your data lake

Follow these instructions to add a rollup Region in Security Lake when you're onboarding to Security Lake.

Note

A Region can contribute data to multiple rollup Regions. However, a rollup Region cannot be a contributing Region for another rollup Region.

Console

When you reach Step 2: Define target objective of the onboarding workflow, choose Add rollup Region under Select rollup Regions. Specify the rollup Region and the Regions that will contribute to it. You can add one or more rollup Regions.
For Service access, create a new IAM role, or use an existing IAM role that gives Security Lake permission to replicate data across multiple Regions.
When you finish, choose Next.
Review your settings in Step 3: Review and create. Then choose Create.

API

To configure rollup Regions programmatically when you're onboarding to Security Lake, use the CreateDataLake operation of the Security Lake API.

In your request, use the replicationConfiguration parameters to specify each Region (regions) that you want the specified Region (region) to contribute data to. For the rollup Region, don't specify any values for the replicationConfiguration parameters.

AWS CLI

If you're using the AWS Command Line Interface (AWS CLI) to onboard to Security Lake, run the create-data-lake command.

When you run the command, use the replicationConfiguration parameters to specify each Region (regions) that you want the specified Region (region) to contribute data to. For the rollup Region, don't specify any values for the replicationConfiguration parameters.

Updating or removing rollup Regions

Choose your preferred access method, and follow these steps to update or remove rollup Regions after enabling Security Lake.

Console

Open the Security Lake console at https://console.aws.amazon.com/securitylake/.
In the navigation pane, under Settings, choose Rollup Regions.
Choose Modify.
To add a rollup Region, choose Add rollup Region, and then specify the contributing Regions.
To remove a rollup Region, choose Remove next to the Region.
When you finish, choose Save.

API

To configure rollup Regions programmatically, use the UpdateDataLake operation of the Security Lake API. In your request, use the supported parameters to specify the rollup settings:

To add a contributing Region, use the region field to specify the Region code for the Region to add. In the regions array of the replicationConfiguration object, specify the Region code for each rollup Region to contribute data to.
To remove a contributing Region, use the region field to specify the Region code for the Region to remove. For the replicationConfiguration parameters, don't specify any values.

For example, the following request specifies the following:

The us-west-2 Region should contribute data to the us-east-1 Region. us-east-1 is the rollup Region.
The eu-west-1 Region should contribute data to the eu-central-1 Region. eu-central-1 is the rollup Region.


{
    configurations: [
        {
            "region": "us-west-2",
            "replicationConfiguration": {
                "regions": [
                    "us-east-1"
                ],
                "roleArn": "arn:aws:iam::123456789012:role/service-role/AmazonSecurityLakeS3ReplicationRole"
            }
        },
        {
            "region": "eu-west-1",
            "replicationConfiguration": {
                "regions": [
                    "eu-central-1"
                ],
                "roleArn": "arn:aws:iam::123456789012:role/service-role/AmazonSecurityLakeS3ReplicationRole"
            }
        }
    ]
}

In the preceding example, Security Lake has already been enabled and configured in the specified rollup Regions, us-east-1 and eu-central-1.

For a list of Region codes, see Amazon Security Lake endpoints in the AWS General Reference.

AWS CLI

To configure rollup Regions by using the AWS Command Line Interface (AWS CLI), run the update-data-lake command. When you run the command, use the supported parameters to specify the rollup settings:

To add a contributing Region, use the region field to specify the Region code for the Region to add. In the regions list of the replicationConfiguration object, specify the Region code for each rollup Region to contribute data to.
To remove a contributing Region, use the region field to specify the Region code for the Region to remove. For the replicationConfiguration parameters, don't specify any values.

For example, the following command specifies the following:

The us-west-2 Region should contribute data to the us-east-1 Region. us-east-1 is the rollup Region.
The eu-west-1 Region should contribute data to the eu-central-1 Region. eu-central-1 is the rollup Region.


$ aws securitylake update-data-lake --configurations file:///update-data-lake.json

Where the contents of the update-data-lake.json file are:


{
    configurations: [
        {
            "region": "us-west-2",
            "replicationConfiguration": {
                "regions": [
                    "us-east-1"
                ],
                "roleArn": "arn:aws:iam::123456789012:role/service-role/AmazonSecurityLakeS3ReplicationRole"
            }
        },
        {
            "region": "eu-west-1",
            "replicationConfiguration": {
                "regions": [
                    "eu-central-1"
                ],
                "roleArn": "arn:aws:iam::123456789012:role/service-role/AmazonSecurityLakeS3ReplicationRole"
            }
        }
    ]
}

In the preceding example, Security Lake has already been enabled and configured in the specified rollup Regions, us-east-1 and eu-central-1.

For a list of Region codes, see Amazon Security Lake endpoints in the AWS General Reference.

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

Security Lake queries

Open Cybersecurity Schema Framework (OCSF)