Getting started with Amazon Security Lake

This section explains how to enable and start using Security Lake. You'll learn how to configure your data lake settings and set up log collection. You can enable and use Security Lake through the AWS Management Console or programmatically. Whichever method you use, you must first set up an AWS account and an administrative user. The steps after that differ based on the method of access. The Security Lake console offers a streamlined process for getting started, and creates all necessary AWS Identity and Access Management (IAM) roles that you need to create your data lake.

Initial AWS account setup

Sign up for an AWS account

If you do not have an AWS account, complete the following steps to create one.

To sign up for an AWS account

1. Open https://portal.aws.amazon.com/billing/signup.

2. Follow the online instructions.

   Part of the sign-up procedure involves receiving a phone call and entering a verification code on the phone keypad.

   When you sign up for an AWS account, an AWS account root user is created. The root user has access to all AWS services and resources in the account. As a security best practice, assign administrative access to a user, and use only the root user to perform tasks that require root user access.

AWS sends you a confirmation email after the sign-up process is complete. At any time, you can view your current account activity and manage your account by going to https://aws.amazon.com/ and choosing My Account.

Create a user with administrative access

After you sign up for an AWS account, secure your AWS account root user, enable AWS IAM Identity Center, and create an administrative user so that you don't use the root user for everyday tasks.

Secure your AWS account root user

1. Sign in to the AWS Management Console as the account owner by choosing Root user and entering your AWS account email address. On the next page, enter your password.

   For help signing in by using root user, see Signing in as the root user in the AWS Sign-In User Guide.

2. Turn on multi-factor authentication (MFA) for your root user.

   For instructions, see Enable a virtual MFA device for your AWS account root user (console) in the IAM User Guide.

Create a user with administrative access

1. Enable IAM Identity Center.

   For instructions, see Enabling AWS IAM Identity Center in the AWS IAM Identity Center User Guide.

2. In IAM Identity Center, grant administrative access to a user.

   For a tutorial about using the IAM Identity Center directory as your identity source, see Configure user access with the default IAM Identity Center directory in the AWS IAM Identity Center User Guide.

Sign in as the user with administrative access

• To sign in with your IAM Identity Center user, use the sign-in URL that was sent to your email address when you created the IAM Identity Center user.

  For help signing in using an IAM Identity Center user, see Signing in to the AWS access portal in the AWS Sign-In User Guide.

Assign access to additional users

1. In IAM Identity Center, create a permission set that follows the best practice of applying least-privilege permissions.

   For instructions, see Create a permission set in the AWS IAM Identity Center User Guide.

2. Assign users to a group, and then assign single sign-on access to the group.

   For instructions, see Add groups in the AWS IAM Identity Center User Guide.

Identify the account that you'll use to enable Security Lake

Security Lake integrates with AWS Organizations to manage log collection across multiple accounts in an organization. If you want to use Security Lake for an organization, you must use your Organizations management account to designate a delegated Security Lake administrator. Then, you must use the credentials of the delegated administrator to enable Security Lake, add member accounts, and enable Security Lake for them. For more information, see Managing multiple accounts with AWS Organizations.

Alternatively, you can use Security Lake without the Organizations integration for a standalone account that's not part of an organization.

Considerations when enabling Amazon Security Lake

Before enabling Security Lake, consider the following:

• Security Lake provides cross-region management features, which means you can create your data lake and configure log collection across AWS Regions. To enable Security Lake in all supported Regions, you can choose any supported Regional endpoint. You can also add rollup Regions to aggregate data from multiple regions to a single Region.

• We recommend activating Security Lake in all of the supported AWS Regions. If you do this, Security Lake can collect data that's connected to unauthorized or unusual activity even in Regions that you aren't actively using. If Security Lake is not activated in all supported Regions, its ability to collect data from other services that you use in multiple Regions is reduced.

• When you enable Security Lake for the first time in any Region, it creates a service-linked role for your account called AWSServiceRoleForSecurityLake. This role includes the permissions to call other AWS services on your behalf and operate the security data lake. For more information about how service-linked roles work, see Using service-linked roles in the IAM User Guide. If you enable Security Lake as the delegated Security Lake administrator, Security Lake creates the service-linked role in each member account in the organization.

• Security Lake doesn't support Amazon S3 Object Lock. When the data lake buckets are created, S3 Object Lock is disabled by default. Enabling Object Lock on a bucket interrupts the delivery of normalized log data to the data lake.

Getting started on the console

This tutorial explains how to enable and configure Security Lake through the AWS Management Console. As part of the AWS Management Console, the Security Lake console offers a streamlined process for getting started, and creates all necessary AWS Identity and Access Management (IAM) roles that you need to create your data lake.

Step 1: Configure sources

Security Lake collects log and event data from a variety of sources and across your AWS accounts and AWS Regions. Follow these instructions to identify which data you want Security Lake to collect. You can only use these instructions to add a natively-supported AWS service as a source. For information about adding a custom source, see Collecting data from custom sources.

To configure log source collection

1. Open the Security Lake console at https://console.aws.amazon.com/securitylake/.

2. By using the AWS Region selector in the upper-right corner of the page, select a Region. You can enable Security Lake in the current Region and other Regions while onboarding.

3. Choose Get started.

4. For Select log and event sources, choose one of the following options:

   1. Ingest default AWS sources – When you choose the recommended option, CloudTrail - S3 data events isn't included for ingestion. This is because ingesting high volume of CloudTrail - S3 data events might impact the usage cost significantly. To ingest this source, select the Ingest specific AWS sources option.

   2. Ingest specific AWS sources – With this option, you can select one or more log and event sources that you want to ingest.

   Note

   When you enable Security Lake in an account for the first time, all the selected log and event sources will be a part of a 15-day free trial period. For more information about usage statistics, see Reviewing usage and estimated costs.

5. For Versions, chose the version of data source from which you want to ingest log and event sources.

   Important

   If you don't have the required role permissions to enable the new version of the AWS log source in the specified Region, contact your Security Lake administrator. For more information, see Update role permissions.

6. For Select Regions, choose whether to ingest log and event sources from all supported Regions or specific Regions. If you choose Specific Regions, select which Regions to ingest data from.

7. For Service access, create a new IAM role or use an existing IAM role that gives Security Lake permission to collect data from your sources and add them to your data lake. One role is used across all Regions in which you enable Security Lake.

8. Choose Next.

Step 2: Define storage settings and rollup Regions (optional)

You can specify the Amazon S3 storage class in which you want Security Lake to store your data and for how long. You can also specify a rollup Region to consolidate data from multiple Regions. These are optional steps. For more information, see Lifecycle management in Security Lake.

To configure storage and rollup settings

1. If you want to consolidate data from multiple contributing Regions to a rollup Region, for Select rollup Regions, choose Add rollup Region. Specify the rollup Region and the Regions that will contribute to it. You can set up one or more rollup Regions.

2. For Select storage classes, choose an Amazon S3 storage class. The default storage class is S3 Standard. Provide a retention period (in days) if you want the data to transition to another storage class after that time, and choose Add transition. After the retention period ends, the objects expire and Amazon S3 deletes them. For more information about Amazon S3 storage classes and retention, see Retention management.

3. If you selected a rollup Region in the first step, for Service access, create a new IAM role or use an existing IAM role that gives Security Lake permission to replicate data across multiple Regions.

4. Choose Next.

Step 3: Review and create data lake

Review the sources that Security Lake will collect data from, your rollup Regions, and your retention settings. Then, create your data lake.

To review and create the data lake

1. While enabling Security Lake, review Log and event sources, Regions, Rollup Regions, and Storage classes.

2. Choose Create.

After creating your data lake, you will see the Summary page on the Security Lake console. This page provides an overview of the number of Regions and Rollup Regions, information about subscribers, and Issues.

The Issues menu shows you a summary of issues from the last 14 days that are impacting the Security Lake service or your Amazon S3 buckets. For additional details about each issue, you can go to the Issues page of the Security Lake console.

Step 4: View and query your own data

After creating your data lake, you can use Amazon Athena or similar services to view and query your data from AWS Lake Formation databases and tables. When you use the console, Security Lake automatically grants database view permissions to the role that you use to enable Security Lake. At a minimum, the role must have Data analyst permissions. For more information on permission levels, see Lake Formation personas and IAM permissions reference. For instructions on granting SELECT permissions, see Granting Data Catalog permissions using the named resource method in the AWS Lake Formation Developer Guide.

Step 5: Create subscribers

After creating your data lake, you can add subscribers to consume your data. Subscribers can consume data by directly accessing objects in your Amazon S3 buckets or by querying the data lake. For more information about subscribers, see Subscriber management in Amazon Security Lake.

Getting started programmatically

This tutorial explains how to enable and start using Security Lake programmatically. The Amazon Security Lake API gives you comprehensive, programmatic access to your Security Lake account, data, and resources. Alternatively, you can use AWS command line tools— the AWS Command Line Interface or the AWS Tools for PowerShell—or the AWS SDKs to access Security Lake.

Step 1: Create IAM roles

If you access Security Lake programmatically, it's necessary to create some AWS Identity and Access Management (IAM) roles in order to configure your data lake.

Important

It's not necessary to create these IAM roles if you use the Security Lake console to enable and configure Security Lake.

You must create roles in IAM if you'll be taking one or more of the following actions (choose the links to see more information about IAM roles for each action):

• Creating a custom source – Custom sources are sources other than natively-supported AWS services that send data to Security Lake.

• Creating a subscriber with data access – Subscribers with permissions can directly access S3 objects from your data lake.

• Creating a subscriber with query access – Subscribers with permissions can query data from Security Lake using services like Amazon Athena.

• Configuring a rollup Region – A rollup Region consolidates data from multiple AWS Regions.

After creating the roles previously mentioned, attach the AmazonSecurityLakeAdministrator AWS managed policy to the role that you're using to enable Security Lake. This policy grants administrative permissions that allow a principal to onboard to Security Lake and access all Security Lake actions.

Attach the AmazonSecurityLakeMetaStoreManager AWS managed policy to create your data lake or query data from Security Lake. This policy is necessary for Security Lake to support extract, transform, and load (ETL) jobs on raw log and event data that it receives from sources.

Step 2: Enable Amazon Security Lake

To enable Security Lake programmatically, use the CreateDataLake operation of the Security Lake API. If you're using the AWS CLI, run the create-data-lake command. In your request, use the region field of the configurations object to specify the Region code for the Region in which to enable Security Lake. For a list of Region codes, see Amazon Security Lake endpoints in the AWS General Reference.

Example 1

The following example command enables Security Lake in the us-east-1 and us-east-2 Regions. In both Regions, this data lake is encrypted with Amazon S3 managed keys. Objects expire after 365 days, and objects transition to the ONEZONE_IA S3 storage class after 60 days. This example is formatted for Linux, macOS, or Unix, and it uses the backslash (\) line-continuation character to improve readability.

$ aws securitylake create-data-lake \
--configurations '[{"encryptionConfiguration": {"kmsKeyId":"S3_MANAGED_KEY"},"region":"us-east-1","lifecycleConfiguration": {"expiration":{"days":365},"transitions":[{"days":60,"storageClass":"ONEZONE_IA"}]}}, {"encryptionConfiguration": {"kmsKeyId":"S3_MANAGED_KEY"},"region":"us-east-2","lifecycleConfiguration": {"expiration":{"days":365},"transitions":[{"days":60,"storageClass":"ONEZONE_IA"}]}}]' \
--meta-store-manager-role-arn "arn:aws:iam:us-east-1:123456789012:role/service-role/AmazonSecurityLakeMetaStoreManager"

Example 2

The following example command enables Security Lake in the us-east-2 Region. This data lake is encrypted with a customer managed key that was created in AWS Key Management Service (AWS KMS). Objects expire after 500 days, and objects transition to the GLACIER S3 storage class after 30 days. This example is formatted for Linux, macOS, or Unix, and it uses the backslash (\) line-continuation character to improve readability.

$ aws securitylake create-data-lake \
--configurations '[{"encryptionConfiguration": {"kmsKeyId":"1234abcd-12ab-34cd-56ef-1234567890ab"},"region":"us-east-2","lifecycleConfiguration": {"expiration":{"days":500},"transitions":[{"days":30,"storageClass":"GLACIER"}]}}]' \
--meta-store-manager-role-arn "arn:aws:iam:us-east-1:123456789012:role/service-role/AmazonSecurityLakeMetaStoreManager"

Note

If you've already enabled Security Lake and want to update the configuration settings for a Region or source, use the UpdateDataLake operation, or if using the AWS CLI, the update-data-lake command. Don't use the CreateDataLake operation.

Step 3: Configure sources

Security Lake collects log and event data from a variety of sources and across your AWS accounts and AWS Regions. Follow these instructions to identify which data you want Security Lake to collect. You can only use these instructions to add a natively-supported AWS service as a source. For information about adding a custom source, see Collecting data from custom sources.

To define one or more collection sources programmatically, use the CreateAwsLogSource operation of the Security Lake API. For each source, specify a Regionally unique value for the sourceName parameter. Optionally use additional parameters to limit the scope of the source to specific accounts (accounts) or a specific version (sourceVersion).

Note

If you don't include an optional parameter in your request, Security Lake applies your request to all accounts or all versions of the specified source, depending on the parameter that you exclude. For example, if you're the delegated Security Lake administrator for an organization and you exclude the accounts parameter, Security Lake applies your request to all the accounts in your organization. Similarly, if you exclude the sourceVersion parameter, Security Lake applies your request to all versions of the specified source.

If your request specifies a Region in which you haven't enabled Security Lake, an error occurs. To address this error, ensure that the regions array specifies only those Regions in which you've enabled Security Lake. Alternatively, you can enable Security Lake in the Region, and then submit your request again.

When you enable Security Lake in an account for the first time, all the selected log and event sources will be a part of a 15-day free trial period. For more information about usage statistics, see Reviewing usage and estimated costs.

Step 4: Configure storage settings and rollup Regions (optional)

You can specify the Amazon S3 storage class in which you want Security Lake to store your data and for how long. You can also specify a rollup Region to consolidate data from multiple Regions. These are optional steps. For more information, see Lifecycle management in Security Lake.

To define a target objective programmatically when you enable Security Lake, use the CreateDataLake operation of the Security Lake API. If you've already enabled Security Lake and want to define a target objective, use the UpdateDataLake operation, not the CreateDataLake operation.

For either operation, use the supported parameters to specify the configuration settings that you want:

• To specify a rollup Region, use the region field to specify the Region that you want to contribute data to the rollup Regions. In the regions array of the replicationConfiguration object, specify the Region code for each rollup Region. For a list of Region codes, see Amazon Security Lake endpoints in the AWS General Reference.

• To specify retention settings for your data, use the lifecycleConfiguration parameters:

  • For transitions, specify the total number of days (days) that you want to store S3 objects in a particular Amazon S3 storage class (storageClass).

  • For expiration, specify the total number of days that you want to store objects in Amazon S3, using any storage class, after objects are created. When this retention period ends, objects expire and Amazon S3 deletes them.

  Security Lake applies the specified retention settings to the Region that you specify in the region field of the configurations object.

For example, the following command creates a data lake with ap-northeast-2 as a rollup Region. The us-east-1 Region will contribute data to the ap-northeast-2 Region. This example also establishes a 10-day expiration period for objects that are added to the data lake.

$ aws securitylake create-data-lake \
--configurations '[{"encryptionConfiguration": {"kmsKeyId":"S3_MANAGED_KEY"},"region":"us-east-1","replicationConfiguration": {"regions": ["ap-northeast-2"],"roleArn":"arn:aws:iam::123456789012:role/service-role/AmazonSecurityLakeS3ReplicationRole"},"lifecycleConfiguration": {"expiration":{"days":10}}}]' \
--meta-store-manager-role-arn "arn:aws:iam::123456789012:role/service-role/AmazonSecurityLakeMetaStoreManager"

You have now created your data lake. Use the ListDataLakes operation of the Security Lake API to verify enablement of Security Lake and your data lake settings in each Region.

If issues or errors arise in the creation of your data lake, you can view a list of exceptions by using the ListDataLakeExceptions operation, and notify users of exceptions with the CreateDataLakeExceptionSubscription operation. For more information, see Troubleshooting data lake status.

Step 5: View and query your own data

After creating your data lake, you can use Amazon Athena or similar services to view and query your data from AWS Lake Formation databases and tables. When you programmatically enable Security Lake, database view permissions aren't granted automatically. The data lake administrator account in AWS Lake Formation must grant SELECT permissions to the IAM role you want to use to query the relevant databases and tables. At a minimum, the role must have Data analyst permissions. For more information on permission levels, see Lake Formation personas and IAM permissions reference. For instructions on granting SELECT permissions, see Granting Data Catalog permissions using the named resource method in the AWS Lake Formation Developer Guide.

Step 6: Create subscribers

After creating your data lake, you can add subscribers to consume your data. Subscribers can consume data by directly accessing objects in your Amazon S3 buckets or by querying the data lake. For more information about subscribers, see Subscriber management in Amazon Security Lake.