Managing data access for Security Lake subscribers - Amazon Security Lake

Managing data access for Security Lake subscribers

Subscribers with data access to source data in Amazon Security Lake are notified of new objects for the source as the data is written to the S3 bucket. By default, subscribers are notified about new objects through an HTTPS endpoint that they provide. Alternatively, subscribers can be notified about new objects by polling an Amazon Simple Queue Service (Amazon SQS) queue.

Prerequisites to creating a subscriber with data access

You must complete the following prerequisites before you can create a subscriber with data access in Security Lake.

Verify permissions

To verify your permissions, use IAM to review the IAM policies that are attached to your IAM identity. Then, compare the information in those policies to the following list of (permissions) actions that you must have to notify subscribers when new data is written to the data lake.

You will need permission to perform the following actions:

  • iam:CreateRole

  • iam:DeleteRolePolicy

  • iam:GetRole

  • iam:PutRolePolicy

  • lakeformation:GrantPermissions

  • lakeformation:ListPermissions

  • lakeformation:RegisterResource

  • lakeformation:RevokePermissions

  • ram:GetResourceShareAssociations

  • ram:GetResourceShares

  • ram:UpdateResourceShare

In addition to the preceding list, you also need permission to perform the following actions:

  • events:CreateApiDestination

  • events:CreateConnection

  • events:DescribeRule

  • events:ListApiDestinations

  • events:ListConnections

  • events:PutRule

  • events:PutTargets

  • s3:GetBucketNotification

  • s3:PutBucketNotification

  • sqs:CreateQueue

  • sqs:DeleteQueue

  • sqs:GetQueueAttributes

  • sqs:GetQueueUrl

  • sqs:SetQueueAttributes

Get the subscriber's external ID

To create a subscriber, apart from the subscriber's AWS account ID, you will also need to get their external ID. The external ID is a unique identifier that the subscriber provides to you. Security Lake adds the external ID to the subscriber IAM role that it creates. You use the external ID when you create a subscriber in the Security Lake console, through the API, or AWS CLI.

For more information about external IDs, see How to use an external ID when granting access to your AWS resources to a third party in the IAM User Guide.

Important

If you plan to use the Security Lake console to add a subscriber, you can skip the next step and proceed to Creating a subscriber with data access. The Security Lake console offers a streamlined process for getting started, and creates all necessary IAM roles or uses existing roles on your behalf.

If you plan to use Security Lake API or AWS CLI to add a subscriber, continue with the next step to create an IAM role to invoke EventBridge API destinations.

Create IAM role to invoke EventBridge API destinations (API and AWS CLI-only step)

If you're using Security Lake through API or AWS CLI, create a role in AWS Identity and Access Management (IAM) that grants Amazon EventBridge permissions to invoke API destinations and send object notifications to the correct HTTPS endpoints.

After creating this IAM role, you'll need the Amazon Resource Name (ARN) of the role in order to create the subscriber. This IAM role isn't necessary if the subscriber polls data from an Amazon Simple Queue Service (Amazon SQS) queue or directly queries data from AWS Lake Formation. For more information about this type of data access method (access type), see Managing query access for Security Lake subscribers.

Attach the following policy to your IAM role:

{ "Version": "2012-10-17", "Statement": [ { "Sid": "AllowInvokeApiDestination", "Effect": "Allow", "Action": [ "events:InvokeApiDestination" ], "Resource": [ "arn:aws:events:{us-west-2}:{123456789012}:api-destination/AmazonSecurityLake*/*" ] } ] }

Attach the following trust policy to your IAM role to permit EventBridge to assume the role:

{ "Version": "2012-10-17", "Statement": [ { "Sid": "AllowEventBridgeToAssume", "Effect": "Allow", "Principal": { "Service": "events.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }

Security Lake automatically creates an IAM role that permits the subscriber to read data from the data lake (or poll events from an Amazon SQS queue if that's the preferred method of notification). This role is protected with an AWS managed policy called AmazonSecurityLakePermissionsBoundary.

Creating a subscriber with data access

Choose one of the following access methods to create a subscriber with access to data in the current AWS Region.

Console
  1. Open the Security Lake console at https://console.aws.amazon.com/securitylake/.

  2. By using the AWS Region selector in the upper-right corner of the page, select the Region where you want to create the subscriber.

  3. In the navigation pane, choose Subscribers.

  4. On the Subscribers page, choose Create subscriber.

  5. For Subscriber details, enter Subscriber name and an optional Description.

    The Region is auto-populated as your currently selected AWS Region and can't be modified.

  6. For Log and event sources, choose which sources the subscriber is authorized to consume.

  7. For Data access method, choose S3 to set up data access for the subscriber.

  8. For Subscriber credentials, provide the subscriber's AWS account ID and external ID.

  9. (Optional) For Notification details, if you want Security Lake to create an Amazon SQS queue that the subscriber can poll for object notifications, select SQS queue. If you want Security Lake to send notifications through EventBridge to an HTTPS endpoint, select Subscription endpoint.

    If you select Subscription endpoint, also do the following:

    1. Enter the Subscription endpoint. Examples of valid endpoint formats include http://example.com. Optionally, you can also provide an HTTPS key name and HTTPS key value.

    2. For Service Access, create a new IAM role or use an existing IAM role that gives EventBridge permission to invoke API destinations and send object notifications to the correct endpoints.

      For information about creating a new IAM role, see Create IAM role to invoke EventBridge API destinations.

  10. (Optional) For Tags, enter as many as 50 tags to assign to the subscriber.

    A tag is a label that you can define and assign to certain types of AWS resources. Each tag consists of a required tag key and an optional tag value. Tags can help you identify, categorize, and manage resources in different ways. To learn more, see Tagging Amazon Security Lake resources.

  11. Choose Create.

API

To create a subscriber with data access programmatically, use the CreateSubscriber operation of the Security Lake API. If you're using the AWS Command Line Interface (AWS CLI), run the create-subscriber command.

In your request, use these parameters to specify the following settings for the subscriber:

  • For sources, specify each source that you want the subscriber to access.

  • For subscriberIdentity, specify the AWS account ID and external ID that the subscriber will use to access source data.

  • For subscriber-name, specify the name of the subscriber.

  • For accessTypes, specify S3.

Example 1

The following example creates a subscriber with access to data in the current AWS Region for the specified subscriber identity for an AWS source.

$ aws securitylake create-subscriber \ --subscriber-identity {"accountID": 1293456789123,"externalId": 123456789012} \ --sources [{awsLogSource: {sourceName: VPC_FLOW, sourceVersion: 1.0}}] \ --subscriber-name subscriber name \ --access-types S3

Example 2

The following example creates a subscriber with access to data in the current AWS Region for the specified subscriber identity for a custom source.

$ aws securitylake create-subscriber \ --subscriber-identity {"accountID": 1293456789123,"externalId": 123456789012} \ --sources [{customLogSource: {sourceName: custom-source-name, sourceVersion: 1.0}}] \ --subscriber-name subscriber name --access-types S3

The preceding examples are formatted for Linux, macOS, or Unix, and they use the backslash (\) line-continuation character to improve readability.

(Optional) After you create a subscriber, use the CreateSubscriberNotification operation to specify how to notify the subscriber when new data is written to the data lake for the sources that you want the subscriber to access. If you're using the AWS Command Line Interface (AWS CLI), run the create-subscriber-notification command.

  • To override the default notification method (HTTPS endpoint) and create an Amazon SQS queue, specify values for the sqsNotificationConfiguration parameters.

  • If you prefer notification with an HTTPS endpoint, specify values for the httpsNotificationConfiguration parameters.

  • For the targetRoleArn field, specify the ARN of the IAM role that you created to invoke EventBridge API destinations.

$ aws securitylake create-subscriber-notification \ --subscriber-id "12345ab8-1a34-1c34-1bd4-12345ab9012" \ --configuration httpsNotificationConfiguration={"targetRoleArn"="arn:aws:iam::XXX:role/service-role/RoleName", "endpoint"="https://account-management.$3.$2.securitylake.aws.dev/v1/datalake"}

To get the subscriberID, use the ListSubscribers operation of the Security Lake API. If you're using the AWS Command Line Interface (AWS CLI), run the list-subscriber command.

$ aws securitylake list-subscribers

To subsequently change the notification method (Amazon SQS queue or HTTPS endpoint) for the subscriber, use the UpdateSubscriberNotification operation or, if you're using the AWS CLI, run the update-subscriber-notification command. You can also change the notification method by using the Security Lake console: select the subscriber on the Subscribers page, and then choose Edit.

Sample object notification message

{ "source": "aws.s3", "time": "2021-11-12T00:00:00Z", "account": "123456789012", "region": "ca-central-1", "resources": [ "arn:aws:s3:::example-bucket" ], "detail": { "bucket": { "name": "example-bucket" }, "object": { "key": "example-key", "size": 5, "etag": "b57f9512698f4b09e608f4f2a65852e5" }, "request-id": "N4N7GDK58NMKJ12R", "requester": "securitylake.amazonaws.com" } }

Updating a data subscriber

You can update a subscriber by changing the sources from which the subscriber consumes. You can also assign or edit the tags for a subscriber. A tag is a label that you can define and assign to certain types of AWS resources, including subscribers. To learn more, see Tagging Amazon Security Lake resources.

Choose one of the access methods, and follow these steps to define new sources for an existing subscription.

Console
  1. Open the Security Lake console at https://console.aws.amazon.com/securitylake/.

  2. In the navigation pane, choose Subscribers.

  3. Select the subscriber.

  4. Choose Edit, and then do any of the following:

    • To update the sources for the subscriber, enter the new settings in the Log and event sources section.

    • To assign or edit tags for the subscriber, change the tags as necessary in the Tags section.

  5. When you finish, choose Save.

API

To update data access sources for a subscriber programmatically, use the UpdateSubscriber operation of the Security Lake API. If you're using the AWS Command Line Interface (AWS CLI), run the update-subscriber command. In your request, use the sources parameters to specify each source that you want the subscriber to access.

$ aws securitylake update-subscriber --subscriber-id subscriber ID

For a list of subscribers associated with a specific AWS account or organization, use the ListSubscribers operation. If you're using the AWS Command Line Interface (AWS CLI), run the list-subscribers command.

$ aws securitylake list-subscribers

To review the current settings for a particular subscriber, use the GetSubscriber operation. run the get-subscriber command. Security Lake then returns the subscriber's name and description, external ID, and additional information. If you're using the AWS Command Line Interface (AWS CLI), run the get-subscriber command.

To update the notification method for a subscriber, use the UpdateSubscriberNotification operation. If you're using the AWS Command Line Interface (AWS CLI), run the update-subscriber-notification command. For example, you can specify a new HTTPS endpoint for the subscriber or switch from an HTTPS endpoint to an Amazon SQS queue.

Removing a data subscriber

If you no longer want a subscriber to consume data from Security Lake, you can remove the subscriber by following these steps.

Console
  1. Open the Security Lake console at https://console.aws.amazon.com/securitylake/.

  2. In the navigation pane, choose Subscribers.

  3. Select the subscriber that you want to remove.

  4. Choose Delete and confirm the action. This will delete the subscriber and all the associated notification settings.

API

Based on your scenario, do one of the following:

  • To delete the subscriber and all associated notification settings, use the DeleteSubscriber operation of the Security Lake API. If you're using the AWS Command Line Interface (AWS CLI), run the delete-subscriber command.

  • To retain the subscriber but stop future notifications to the subscriber, use the DeleteSubscriberNotification operation of the Security Lake API. If you're using the AWS Command Line Interface (AWS CLI), run the run the delete-subscriber-notification command.