Publish flow logs to Amazon Data Firehose - Amazon Virtual Private Cloud

Publish flow logs to Amazon Data Firehose

Flow logs can publish flow log data directly to Amazon Data Firehose.

When publishing to Amazon Data Firehose, flow log data is published to a Amazon Data Firehose delivery stream, in plain text format.

Pricing

Standard ingestion and delivery charges apply. For more information, open Amazon CloudWatch Pricing, select Logs and find Vended Logs.

IAM roles for cross account delivery

When you publish to Amazon Data Firehose, you can choose a delivery stream that's in the same account as the resource to monitor (the source account), or in a different account (the destination account). To enable cross account delivery of flow logs to Amazon Data Firehose, you must create an IAM role in the source account and an IAM role in the destination account.

Source account role

In the source account, create a role that grants the following permissions. In this example, the name of the role is mySourceRole, but you can choose a different name for this role. The last statement allows the role in the destination account to assume this role. The condition statements ensure that this role is passed only to the log delivery service, and only when monitoring the specified resource. When you create your policy, specify the VPCs, network interfaces, or subnets that you're monitoring with the condition key iam:AssociatedResourceARN.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::source-account:role/mySourceRole", "Condition": { "StringEquals": { "iam:PassedToService": "delivery.logs.amazonaws.com" }, "StringLike": { "iam:AssociatedResourceARN": [ "arn:aws:ec2:region:source-account:vpc/vpc-00112233344556677" ] } } }, { "Effect": "Allow", "Action": [ "logs:CreateLogDelivery", "logs:DeleteLogDelivery", "logs:ListLogDeliveries", "logs:GetLogDelivery" ], "Resource": "*" }, { "Effect": "Allow", "Action": "sts:AssumeRole", "Resource": "arn:aws:iam::destination-account:role/AWSLogDeliveryFirehoseCrossAccountRole" } ] }

Ensure that this role has the following trust policy, which allows the log delivery service to assume the role.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "delivery.logs.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }

From the source account, use the following procedure to create the role.

To create the source account role
  1. Open the IAM console at https://console.aws.amazon.com/iam/.

  2. In the navigation pane, choose Policies.

  3. Choose Create policy.

  4. On the Create policy page, do the following:

    1. Choose JSON.

    2. Replace the contents of this window with the permissions policy at the start of this section.

    3. Choose Next.

    4. Enter a name for your policy and an optional description and tags, and then choose Create policy.

  5. In the navigation pane, choose Roles.

  6. Choose Create role.

  7. For Trusted entity type, choose Custom trust policy. For Custom trust policy, replace "Principal": {}, with the following, which specifies the log delivery service. Choose Next.

    "Principal": { "Service": "delivery.logs.amazonaws.com" },
  8. On the Add permissions page, select the checkbox for the policy that you created earlier in this procedure, and then choose Next.

  9. Enter a name for your role and optionally provide a description.

  10. Choose Create role.

Destination account role

In the destination account, create a role with a name that starts with AWSLogDeliveryFirehoseCrossAccountRole. This role must grant the following permissions.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iam:CreateServiceLinkedRole", "firehose:TagDeliveryStream" ], "Resource": "*" } ] }

Ensure that this role has the following trust policy, which allows the role that you created in the source account to assume this role.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::source-account:role/mySourceRole" }, "Action": "sts:AssumeRole" } ] }

From the destination account, use the following procedure to create the role.

To create the destination account role
  1. Open the IAM console at https://console.aws.amazon.com/iam/.

  2. In the navigation pane, choose Policies.

  3. Choose Create policy.

  4. On the Create policy page, do the following:

    1. Choose JSON.

    2. Replace the contents of this window with the permissions policy at the start of this section.

    3. Choose Next.

    4. Enter a name for your policy that starts with AWSLogDeliveryFirehoseCrossAccountRole, and then choose Create policy.

  5. In the navigation pane, choose Roles.

  6. Choose Create role.

  7. For Trusted entity type, choose Custom trust policy. For Custom trust policy, replace "Principal": {}, with the following, which specifies the source account role. Choose Next.

    "Principal": { "AWS": "arn:aws:iam::source-account:role/mySourceRole" },
  8. On the Add permissions page, select the checkbox for the policy that you created earlier in this procedure, and then choose Next.

  9. Enter a name for your role and optionally provide a description.

  10. Choose Create role.

Create a flow log that publishes to Amazon Data Firehose

You can create flow logs for your VPCs, subnets, or network interfaces.

Prerequisites
To create a flow log that publishes to Amazon Data Firehose
  1. Do one of the following:

  2. Choose Actions, Create flow log.

  3. For Filter, specify the type of traffic to log.

    • Accept – Log only accepted traffic

    • Reject – Log only rejected traffic

    • All – Log accepted and rejected traffic

  4. For Maximum aggregation interval, choose the maximum period of time during which a flow is captured and aggregated into one flow log record.

  5. For Destination, choose either of the following options:

    • Send to Amazon Data Firehose in the same account – The delivery stream and the resource to monitor are in the same account.

    • Send to Amazon Data Firehose in a different account – The delivery stream and the resource to monitor are in different accounts.

  6. For Amazon Data Firehose stream name, choose the delivery stream that you created.

  7. [Cross account delivery only] For IAM roles, specify the required roles (see IAM roles for cross account delivery).

  8. For Log record format, specify the format for the flow log record.

    • To use the default flow log record format, choose AWS default format.

    • To create a custom format, choose Custom format. For Log format, choose the fields to include in the flow log record.

  9. For Additional metadata, select if you want to include metadata from Amazon ECS in the log format.

  10. (Optional) Choose Add tag to apply tags to the flow log.

  11. Choose Create flow log.

To create a flow log that publishes to Amazon Data Firehose using a command line tool

Use one of the following commands:

The following AWS CLI example creates a flow log that captures all traffic for the specified VPC and delivers the flow logs to the specified Amazon Data Firehose delivery stream in the same account.

aws ec2 create-flow-logs --traffic-type ALL \ --resource-type VPC \ --resource-ids vpc-00112233344556677 \ --log-destination-type kinesis-data-firehose \ --log-destination arn:aws:firehose:us-east-1:123456789012:deliverystream:flowlogs_stream

The following AWS CLI example creates a flow log that captures all traffic for the specified VPC and delivers the flow logs to the specified Amazon Data Firehose delivery stream in a different account.

aws ec2 create-flow-logs --traffic-type ALL \ --resource-type VPC \ --resource-ids vpc-00112233344556677 \ --log-destination-type kinesis-data-firehose \ --log-destination arn:aws:firehose:us-east-1:123456789012:deliverystream:flowlogs_stream \ --deliver-logs-permission-arn arn:aws:iam::source-account:role/mySourceRole \ --deliver-cross-account-role arn:aws:iam::destination-account:role/AWSLogDeliveryFirehoseCrossAccountRole

Process flow log records in Amazon Data Firehose

You can get the flow log data from the destination that you configured for the delivery stream.