Amazon Data Firehose was previously known as Amazon Kinesis Data Firehose
Controlling Access with Amazon Data Firehose
The following sections cover how to control access to and from your Amazon Data Firehose resources. The information they cover includes how to grant your application access so it can send data to your Firehose stream. They also describe how you can grant Amazon Data Firehose access to your Amazon Simple Storage Service (Amazon S3) bucket, Amazon Redshift cluster, or Amazon OpenSearch Service cluster, as well as the access permissions you need if you use Datadog, Dynatrace, LogicMonitor, MongoDB, New Relic, Splunk, or Sumo Logic as your destination. Finally, you'll find in this topic guidance on how to configure Amazon Data Firehose so it can deliver data to a destination that belongs to a different AWS account. The technology for managing all these forms of access is AWS Identity and Access Management (IAM). For more information about IAM, see What is IAM?.
Contents
- Grant Your Application Access to Your Amazon Data Firehose Resources
- Grant Amazon Data Firehose Access to your Private Amazon MSK Cluster
- Allow Amazon Data Firehose to Assume an IAM Role
- Grant Amazon Data Firehose Access to AWS Glue for Data Format Conversion
- Grant Amazon Data Firehose Access to an Amazon S3 Destination
- Grant Amazon Data Firehose Access to an Amazon Redshift Destination
- Grant Amazon Data Firehose Access to a Public OpenSearch Service Destination
- Grant Amazon Data Firehose Access to an OpenSearch Service Destination in a VPC
- Grant Amazon Data Firehose Access to a Public OpenSearch Serverless Destination
- Grant Amazon Data Firehose Access to an OpenSearch Serverless Destination in a VPC
- Grant Amazon Data Firehose Access to a Splunk Destination
- Access to Splunk in VPC
- Grant Firehose Access to a Snowflake Destination
- Grant Amazon Data Firehose Access to an HTTP Endpoint Destination
- Cross-Account Delivery from Amazon MSK
- Cross-Account Delivery to an Amazon S3 Destination
- Cross-Account Delivery to an OpenSearch Service Destination
- Using Tags to Control Access
Grant Your Application Access to Your Amazon Data Firehose Resources
To give your application access to your Firehose stream, use a policy similar to
this example. You can adjust the individual API operations to which you grant access by
modifying the Action
section, or grant access to all operations with
"firehose:*"
.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "firehose:DeleteDeliveryStream", "firehose:PutRecord", "firehose:PutRecordBatch", "firehose:UpdateDestination" ], "Resource": [ "arn:aws:firehose:
region
:account-id
:deliverystream/delivery-stream-name
" ] } ] }
Grant Amazon Data Firehose Access to your Private Amazon MSK Cluster
If the source of your Firehose stream is a private Amazon MSK cluster, then use a policy similar to this example.
{ "Version": "2012-10-17", "Statement": [ { "Principal": { "Service": [ "firehose.amazonaws.com" ] }, "Effect": "Allow", "Action": [ "kafka:CreateVpcConnection" ], "Resource": "cluster-arn" } ] }
Allow Amazon Data Firehose to Assume an IAM Role
This section describes the permissions and policies that grant Amazon Data Firehose access to ingest, process, and deliver data from source to destination.
Note
If you use the console to create a Firehose stream and choose the option to create
a new role, AWS attaches the required trust policy to the role. If you want Amazon Data Firehose
to use an existing IAM role or if you create a role on your own, attach the
following trust policies to that role so that Amazon Data Firehose can assume it. Edit the policies
to replace account-id
with your AWS account ID. For
information about how to modify the trust relationship of a role, see Modifying a Role.
Amazon Data Firehose uses an IAM role for all the permissions that the delivery stream needs to process and deliver data. Make sure that the following trust policies are attached to that role so that Amazon Data Firehose can assume it.
{ "Version": "2012-10-17", "Statement": [{ "Sid": "", "Effect": "Allow", "Principal": { "Service": "firehose.amazonaws.com" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "account-id" } } }] }
This policy uses the sts:ExternalId
condition context key to ensure that
only Amazon Data Firehose activity originating from your AWS account can assume this IAM role. For
more information about preventing unauthorized use of IAM roles, see The confused deputy
problem in the IAM User Guide.
If you choose Amazon MSK as the source for your Firehose stream, you must specify another IAM role that grants Amazon Data Firehose permissions to ingest source data from the specified Amazon MSK cluster. Make sure that the following trust policies are attached to that role so that Amazon Data Firehose can assume it.
{ "Version": "2012-10-17", "Statement": [ { "Principal": { "Service": [ "firehose.amazonaws.com" ] }, "Effect": "Allow", "Action": "sts:AssumeRole" } ] }
Make sure that this role that grants Amazon Data Firehose permissions to ingest source data from the specified Amazon MSK cluster grants the following permissions:
{ "Version": "2012-10-17", "Statement": [{ "Effect":"Allow", "Action": [ "kafka:GetBootstrapBrokers", "kafka:DescribeCluster", "kafka:DescribeClusterV2", "kafka-cluster:Connect" ], "Resource": "CLUSTER-ARN" }, { "Effect":"Allow", "Action": [ "kafka-cluster:DescribeTopic", "kafka-cluster:DescribeTopicDynamicConfiguration", "kafka-cluster:ReadData" ], "Resource": "TOPIC-ARN" }] }
Grant Amazon Data Firehose Access to AWS Glue for Data Format Conversion
If your Firehose stream performs data-format conversion, Amazon Data Firehose references table definitions stored in AWS Glue. To give Amazon Data Firehose the necessary access to AWS Glue, add the following statement to your policy. For information on how to find the ARN of the table, see Specifying AWS Glue Resource ARNs.
{ "Effect": "Allow", "Action": [ "glue:GetTable", "glue:GetTableVersion", "glue:GetTableVersions" ], "Resource": "
table-arn
" }
Note
Currently, AWS Glue is not supported in the Israel (Tel Aviv), Asia Pacific (Jakarta) or Middle East (UAE) Regions. If you are working with Amazon Data Firehose in the Asia Pacific (Jakarta) Region or Middle East (UAE) Region, make sure to give your Amazon Data Firehose access to AWS Glue in one of the Regions where AWS Glue is currently supported. Cross-region interoperability between Data Firehose and AWS Glue is supported. For more information on regions where AWS Glue is supported, see https://docs.aws.amazon.com/general/latest/gr/glue.html
Grant Amazon Data Firehose Access to an Amazon S3 Destination
When you're using an Amazon S3 destination, Amazon Data Firehose delivers data to your S3 bucket and can optionally use an AWS KMS key that you own for data encryption. If error logging is enabled, Amazon Data Firehose also sends data delivery errors to your CloudWatch log group and streams. You are required to have an IAM role when creating a Firehose stream. Amazon Data Firehose assumes that IAM role and gains access to the specified bucket, key, and CloudWatch log group and streams.
Use the following access policy to enable Amazon Data Firehose to access your S3 bucket and AWS KMS
key. If you don't own the S3 bucket, add s3:PutObjectAcl
to the list of
Amazon S3 actions. This grants the bucket owner full access to the objects delivered by
Amazon Data Firehose.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::
bucket-name
", "arn:aws:s3:::bucket-name
/*" ] }, { "Effect": "Allow", "Action": [ "kinesis:DescribeStream", "kinesis:GetShardIterator", "kinesis:GetRecords", "kinesis:ListShards" ], "Resource": "arn:aws:kinesis:region
:account-id
:stream/stream-name
" }, { "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:GenerateDataKey" ], "Resource": [ "arn:aws:kms:region
:account-id
:key/key-id
" ], "Condition": { "StringEquals": { "kms:ViaService": "s3.region
.amazonaws.com" }, "StringLike": { "kms:EncryptionContext:aws:s3:arn": "arn:aws:s3:::bucket-name/prefix
*" } } }, { "Effect": "Allow", "Action": [ "logs:PutLogEvents" ], "Resource": [ "arn:aws:logs:region
:account-id
:log-group:log-group-name
:log-stream:log-stream-name
" ] }, { "Effect": "Allow", "Action": [ "lambda:InvokeFunction", "lambda:GetFunctionConfiguration" ], "Resource": [ "arn:aws:lambda:region
:account-id
:function:function-name
:function-version
" ] } ] }
The policy above also has a statement that allows access to Amazon Kinesis Data Streams. If you don't use Kinesis Data Streams as your data source, you can remove that statement. If you use Amazon MSK as your source, then you can substiute that statement with the following:
{ "Sid":"", "Effect":"Allow", "Action":[ "kafka:GetBootstrapBrokers", "kafka:DescribeCluster", "kafka:DescribeClusterV2", "kafka-cluster:Connect" ], "Resource":"arn:aws:kafka:{{mskClusterRegion}}:{{mskClusterAccount}}:cluster/{{mskClusterName}}/{{clusterUUID}}" }, { "Sid":"", "Effect":"Allow", "Action":[ "kafka-cluster:DescribeTopic", "kafka-cluster:DescribeTopicDynamicConfiguration", "kafka-cluster:ReadData" ], "Resource":"arn:aws:kafka:{{mskClusterRegion}}:{{mskClusterAccount}}:topic/{{mskClusterName}}/{{clusterUUID}}/{{mskTopicName}}" }, { "Sid":"", "Effect":"Allow", "Action":[ "kafka-cluster:DescribeGroup" ], "Resource":"arn:aws:kafka:{{mskClusterRegion}}:{{mskClusterAccount}}:group/{{mskClusterName}}/{{clusterUUID}}/*" }
For more information about allowing other AWS services to access your AWS resources, see Creating a Role to Delegate Permissions to an AWS Service in the IAM User Guide.
To learn how to grant Amazon Data Firehose access to an Amazon S3 destination in another account, see Cross-Account Delivery to an Amazon S3 Destination.
Grant Amazon Data Firehose Access to an Amazon Redshift Destination
Refer to the following when you are granting access to Amazon Data Firehose when using an Amazon Redshift destination.
Topics
IAM Role and Access Policy
When you're using an Amazon Redshift destination, Amazon Data Firehose delivers data to your S3 bucket as an intermediate location. It can optionally use an AWS KMS key you own for data encryption. Amazon Data Firehose then loads the data from the S3 bucket to your Amazon Redshift provisioned cluster or Amazon Redshift Serverless workgroup. If error logging is enabled, Amazon Data Firehose also sends data delivery errors to your CloudWatch log group and streams. Amazon Data Firehose uses the specified Amazon Redshift user name and password to access your provisioned cluster or Amazon Redshift Serverless workgroup, and uses an IAM role to access the specified bucket, key, CloudWatch log group, and streams. You are required to have an IAM role when creating a Firehose stream.
Use the following access policy to enable Amazon Data Firehose to access your S3 bucket and AWS KMS
key. If you don't own the S3 bucket, add s3:PutObjectAcl
to the list of
Amazon S3 actions, which grants the bucket owner full access to the objects delivered by
Amazon Data Firehose. This policy also has a statement that allows access to Amazon Kinesis Data Streams. If you
don't use Kinesis Data Streams as your data source, you can remove that statement.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::
bucket-name
", "arn:aws:s3:::bucket-name
/*" ] }, { "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:GenerateDataKey" ], "Resource": [ "arn:aws:kms:region
:account-id
:key/key-id
" ], "Condition": { "StringEquals": { "kms:ViaService": "s3.region
.amazonaws.com" }, "StringLike": { "kms:EncryptionContext:aws:s3:arn": "arn:aws:s3:::bucket-name/prefix
*" } } }, { "Effect": "Allow", "Action": [ "kinesis:DescribeStream", "kinesis:GetShardIterator", "kinesis:GetRecords", "kinesis:ListShards" ], "Resource": "arn:aws:kinesis:region
:account-id
:stream/stream-name
" }, { "Effect": "Allow", "Action": [ "logs:PutLogEvents" ], "Resource": [ "arn:aws:logs:region
:account-id
:log-group:log-group-name
:log-stream:log-stream-name
" ] }, { "Effect": "Allow", "Action": [ "lambda:InvokeFunction", "lambda:GetFunctionConfiguration" ], "Resource": [ "arn:aws:lambda:region
:account-id
:function:function-name
:function-version
" ] } ] }
For more information about allowing other AWS services to access your AWS resources, see Creating a Role to Delegate Permissions to an AWS Service in the IAM User Guide.
VPC Access to an Amazon Redshift Provisioned Cluster or Amazon Redshift Serverless Workgroup
If your Amazon Redshift provisioned cluster or Amazon Redshift Serverless workgroup is in a virtual private cloud (VPC), it must be publicly accessible with a public IP address. Also, grant Amazon Data Firehose access to your Amazon Redshift provisioned cluster or Amazon Redshift Serverless workgroup by unblocking the Amazon Data Firehose IP addresses. Amazon Data Firehose currently uses one CIDR block for each available Region:
-
13.58.135.96/27
for US East (Ohio) -
52.70.63.192/27
for US East (N. Virginia) -
13.57.135.192/27
for US West (N. California) -
52.89.255.224/27
for US West (Oregon) -
18.253.138.96/27
for AWS GovCloud (US-East) -
52.61.204.160/27
for AWS GovCloud (US-West) -
35.183.92.128/27
for Canada (Central) -
40.176.98.192/27
for Canada West (Calgary) -
18.162.221.32/27
for Asia Pacific (Hong Kong) -
13.232.67.32/27
for Asia Pacific (Mumbai) -
18.60.192.128/27
for Asia Pacific (Hyderabad) -
13.209.1.64/27
for Asia Pacific (Seoul) -
13.228.64.192/27
for Asia Pacific (Singapore) -
13.210.67.224/27
for Asia Pacific (Sydney) -
108.136.221.64/27
for Asia Pacific (Jakarta) -
13.113.196.224/27
for Asia Pacific (Tokyo) -
13.208.177.192/27
for Asia Pacific (Osaka) -
52.81.151.32/27
for China (Beijing) -
161.189.23.64/27
for China (Ningxia) -
16.62.183.32/27
for Europe (Zurich) -
35.158.127.160/27
for Europe (Frankfurt) -
52.19.239.192/27
for Europe (Ireland) -
18.130.1.96/27
for Europe (London) -
35.180.1.96/27
for Europe (Paris) -
13.53.63.224/27
for Europe (Stockholm) -
15.185.91.0/27
for Middle East (Bahrain) -
18.228.1.128/27
for South America (São Paulo) -
15.161.135.128/27
for Europe (Milan) -
13.244.121.224/27
for Africa (Cape Town) -
13.208.177.192/27
for Asia Pacific (Osaka) -
108.136.221.64/27
for Asia Pacific (Jakarta) -
3.28.159.32/27
for Middle East (UAE) -
51.16.102.0/27
for Israel (Tel Aviv) -
16.50.161.128/27
for Asia Pacific (Melbourne)
For more information about how to unblock IP addresses, see the step Authorize Access to the Cluster in the Amazon Redshift Getting Started Guide guide.
Grant Amazon Data Firehose Access to a Public OpenSearch Service Destination
When you're using an OpenSearch Service destination, Amazon Data Firehose delivers data to your OpenSearch Service cluster, and concurrently backs up failed or all documents to your S3 bucket. If error logging is enabled, Amazon Data Firehose also sends data delivery errors to your CloudWatch log group and streams. Amazon Data Firehose uses an IAM role to access the specified OpenSearch Service domain, S3 bucket, AWS KMS key, and CloudWatch log group and streams. You are required to have an IAM role when creating a Firehose stream.
Use the following access policy to enable Amazon Data Firehose to access your S3 bucket, OpenSearch
Service domain, and AWS KMS key. If you do not own the S3 bucket, add
s3:PutObjectAcl
to the list of Amazon S3 actions, which grants the bucket
owner full access to the objects delivered by Amazon Data Firehose. This policy also has a statement
that allows access to Amazon Kinesis Data Streams. If you don't use Kinesis Data Streams as your data source, you can
remove that statement.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::
bucket-name
", "arn:aws:s3:::bucket-name
/*" ] }, { "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:GenerateDataKey" ], "Resource": [ "arn:aws:kms:region
:account-id
:key/key-id
" ], "Condition": { "StringEquals": { "kms:ViaService": "s3.region
.amazonaws.com" }, "StringLike": { "kms:EncryptionContext:aws:s3:arn": "arn:aws:s3:::bucket-name/prefix
*" } } }, { "Effect": "Allow", "Action": [ "es:DescribeDomain", "es:DescribeDomains", "es:DescribeDomainConfig", "es:ESHttpPost", "es:ESHttpPut" ], "Resource": [ "arn:aws:es:region
:account-id
:domain/domain-name
", "arn:aws:es:region
:account-id
:domain/domain-name
" ] }, { "Effect": "Allow", "Action": [ "es:ESHttpGet" ], "Resource": [ "arn:aws:es:region
:account-id
:domain/domain-name
/_all/_settings", "arn:aws:es:region
:account-id
:domain/domain-name
/_cluster/stats", "arn:aws:es:region
:account-id
:domain/domain-name
/index-name
*/_mapping/type-name
", "arn:aws:es:region
:account-id
:domain/domain-name
/_nodes", "arn:aws:es:region
:account-id
:domain/domain-name
/_nodes/stats", "arn:aws:es:region
:account-id
:domain/domain-name
/_nodes/*/stats", "arn:aws:es:region
:account-id
:domain/domain-name
/_stats", "arn:aws:es:region
:account-id
:domain/domain-name
/index-name
*/_stats", "arn:aws:es:region
:account-id
:domain/domain-name
/" ] }, { "Effect": "Allow", "Action": [ "kinesis:DescribeStream", "kinesis:GetShardIterator", "kinesis:GetRecords", "kinesis:ListShards" ], "Resource": "arn:aws:kinesis:region
:account-id
:stream/stream-name
" }, { "Effect": "Allow", "Action": [ "logs:PutLogEvents" ], "Resource": [ "arn:aws:logs:region
:account-id
:log-group:log-group-name
:log-stream:log-stream-name
" ] }, { "Effect": "Allow", "Action": [ "lambda:InvokeFunction", "lambda:GetFunctionConfiguration" ], "Resource": [ "arn:aws:lambda:region
:account-id
:function:function-name
:function-version
" ] } ] }
For more information about allowing other AWS services to access your AWS resources, see Creating a Role to Delegate Permissions to an AWS Service in the IAM User Guide.
To learn how to grant Amazon Data Firehose access to an OpenSearch Service cluster in another account, see Cross-Account Delivery to an OpenSearch Service Destination.
Grant Amazon Data Firehose Access to an OpenSearch Service Destination in a VPC
If your OpenSearch Service domain is in a VPC, make sure you give Amazon Data Firehose the permissions that are described in the previous section. In addition, you need to give Amazon Data Firehose the following permissions to enable it to access your OpenSearch Service domain's VPC.
-
ec2:DescribeVpcs
-
ec2:DescribeVpcAttribute
-
ec2:DescribeSubnets
-
ec2:DescribeSecurityGroups
-
ec2:DescribeNetworkInterfaces
-
ec2:CreateNetworkInterface
-
ec2:CreateNetworkInterfacePermission
-
ec2:DeleteNetworkInterface
Important
Do not revoke these permissions after you create the delivery stream. If you revoke these permissions, your Firehose stream will be degraded or stop delivering data to your OpenSearch service domain whenever the service attempts to query or update ENIs.
Important
When you specify subnets for delivering data to the destination in a private VPC, make sure you have enough number of free IP addresses in chosen subnets. If there is no available free IP address in a specified subnet, Firehose cannot create or add ENIs for the data delivery in the private VPC, and the delivery will be degraded or fail.
When you create or update your delivery stream, you specify a security group for Firehose to use when it sends data to your OpenSearch Service domain. You can use the same security group that the OpenSearch Service domain uses or a different one. If you specify a different security group, ensure that it allows outbound HTTPS traffic to the OpenSearch Service domain's security group. Also ensure that the OpenSearch Service domain's security group allows HTTPS traffic from the security group you specified when you configured your Firehose stream. If you use the same security group for both your Firehose stream and the OpenSearch Service domain, make sure the security group inbound rule allows HTTPS traffic. For more information about security group rules, see Security group rules in the Amazon VPC documentation.
Grant Amazon Data Firehose Access to a Public OpenSearch Serverless Destination
When you're using an OpenSearch Serverless destination, Amazon Data Firehose delivers data to your OpenSearch Serverless collection, and concurrently backs up failed or all documents to your S3 bucket. If error logging is enabled, Amazon Data Firehose also sends data delivery errors to your CloudWatch log group and streams. Amazon Data Firehose uses an IAM role to access the specified OpenSearch Serverless collection, S3 bucket, AWS KMS key, and CloudWatch log group and streams. You are required to have an IAM role when creating a Firehose stream.
Use the following access policy to enable Amazon Data Firehose to access your S3 bucket, OpenSearch
Serverless domain, and AWS KMS key. If you do not own the S3 bucket, add
s3:PutObjectAcl
to the list of Amazon S3 actions, which grants the bucket
owner full access to the objects delivered by Amazon Data Firehose. This policy also has a statement
that allows access to Amazon Kinesis Data Streams. If you don't use Kinesis Data Streams as your data source, you can
remove that statement.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::
bucket-name
", "arn:aws:s3:::bucket-name
/*" ] }, { "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:GenerateDataKey" ], "Resource": [ "arn:aws:kms:region
:account-id
:key/key-id
" ], "Condition": { "StringEquals": { "kms:ViaService": "s3.region
.amazonaws.com" }, "StringLike": { "kms:EncryptionContext:aws:s3:arn": "arn:aws:s3:::bucket-name/prefix
*" } } }, { "Effect": "Allow", "Action": [ "kinesis:DescribeStream", "kinesis:GetShardIterator", "kinesis:GetRecords", "kinesis:ListShards" ], "Resource": "arn:aws:kinesis:region
:account-id
:stream/stream-name
" }, { "Effect": "Allow", "Action": [ "logs:PutLogEvents" ], "Resource": [ "arn:aws:logs:region
:account-id
:log-group:log-group-name
:log-stream:log-stream-name
" ] }, { "Effect": "Allow", "Action": [ "lambda:InvokeFunction", "lambda:GetFunctionConfiguration" ], "Resource": [ "arn:aws:lambda:region
:account-id
:function:function-name
:function-version
" ] }, { "Effect": "Allow", "Action": "aoss:APIAccessAll", "Resource": "arn:aws:aoss:region
:account-id
:collection/collection-id
" } ] }
In addition to the policy above, you must also configure Amazon Data Firehose to have the following minimum permissions assigned in a data access policy:
[ { "Rules":[ { "ResourceType":"index", "Resource":[ "index/target-collection/target-index" ], "Permission":[ "aoss:WriteDocument", "aoss:UpdateIndex", "aoss:CreateIndex" ] } ], "Principal":[ "arn:aws:sts::
account-id
:assumed-role/firehose-delivery-role-name
/*" ] } ]
For more information about allowing other AWS services to access your AWS resources, see Creating a Role to Delegate Permissions to an AWS Service in the IAM User Guide.
Grant Amazon Data Firehose Access to an OpenSearch Serverless Destination in a VPC
If your OpenSearch Serverless collection is in a VPC, make sure you give Amazon Data Firehose the permissions that are described in the previous section. In addition, you need to give Amazon Data Firehose the following permissions to enable it to access your OpenSearch Serverless collection's VPC.
-
ec2:DescribeVpcs
-
ec2:DescribeVpcAttribute
-
ec2:DescribeSubnets
-
ec2:DescribeSecurityGroups
-
ec2:DescribeNetworkInterfaces
-
ec2:CreateNetworkInterface
-
ec2:CreateNetworkInterfacePermission
-
ec2:DeleteNetworkInterface
Important
Do not revoke these permissions after you create the delivery stream. If you revoke these permissions, your Firehose stream will be degraded or stop delivering data to your OpenSearch service domain whenever the service attempts to query or update ENIs.
Important
When you specify subnets for delivering data to the destination in a private VPC, make sure you have enough number of free IP addresses in chosen subnets. If there is no available free IP address in a specified subnet, Firehose cannot create or add ENIs for the data delivery in the private VPC, and the delivery will be degraded or fail.
When you create or update your delivery stream, you specify a security group for Firehose to use when it sends data to your OpenSearch Serverless collection. You can use the same security group that the OpenSearch Serverless collection uses or a different one. If you specify a different security group, ensure that it allows outbound HTTPS traffic to the OpenSearch Serverless collection's security group. Also ensure that the OpenSearch Serverless collection's security group allows HTTPS traffic from the security group you specified when you configured your Firehose stream. If you use the same security group for both your Firehose stream and the OpenSearch Serverless collection, make sure the security group inbound rule allows HTTPS traffic. For more information about security group rules, see Security group rules in the Amazon VPC documentation.
Grant Amazon Data Firehose Access to a Splunk Destination
When you're using a Splunk destination, Amazon Data Firehose delivers data to your Splunk HTTP Event Collector (HEC) endpoint. It also backs up that data to the Amazon S3 bucket that you specify, and you can optionally use an AWS KMS key that you own for Amazon S3 server-side encryption. If error logging is enabled, Firehose sends data delivery errors to your CloudWatch log streams. You can also use AWS Lambda for data transformation.
If you use an AWS load balancer, make sure that it is a Classic Load Balancer or an Application Load Balancer. Also, enable duration-based sticky sessions with cookie expiration disabled for Classic Load Balancer and expiration is set to the maximum (7 days) for Application Load Balancer. For information about how to do this, see Duration-Based Session Stickiness for Classic Load Balancer or an Application Load Balancer.
You are required to have an IAM role when creating a delivery stream. Firehose assumes that IAM role and gains access to the specified bucket, key, and CloudWatch log group and streams.
Use the following access policy to enable Amazon Data Firehose to access your S3 bucket. If you don't
own the S3 bucket, add s3:PutObjectAcl
to the list of Amazon S3 actions, which
grants the bucket owner full access to the objects delivered by Amazon Data Firehose. This policy also
grants Amazon Data Firehose access to CloudWatch for error logging and to AWS Lambda for data transformation.
The policy also has a statement that allows access to Amazon Kinesis Data Streams. If you don't use Kinesis Data Streams
as your data source, you can remove that statement. Amazon Data Firehose doesn't use IAM to access
Splunk. For accessing Splunk, it uses your HEC token.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::
bucket-name
", "arn:aws:s3:::bucket-name
/*" ] }, { "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:GenerateDataKey" ], "Resource": [ "arn:aws:kms:region
:account-id
:key/key-id
" ], "Condition": { "StringEquals": { "kms:ViaService": "s3.region
.amazonaws.com" }, "StringLike": { "kms:EncryptionContext:aws:s3:arn": "arn:aws:s3:::bucket-name/prefix
*" } } }, { "Effect": "Allow", "Action": [ "kinesis:DescribeStream", "kinesis:GetShardIterator", "kinesis:GetRecords", "kinesis:ListShards" ], "Resource": "arn:aws:kinesis:region
:account-id
:stream/stream-name
" }, { "Effect": "Allow", "Action": [ "logs:PutLogEvents" ], "Resource": [ "arn:aws:logs:region
:account-id
:log-group:log-group-name
:log-stream:*" ] }, { "Effect": "Allow", "Action": [ "lambda:InvokeFunction", "lambda:GetFunctionConfiguration" ], "Resource": [ "arn:aws:lambda:region
:account-id
:function:function-name
:function-version
" ] } ] }
For more information about allowing other AWS services to access your AWS resources, see Creating a Role to Delegate Permissions to an AWS Service in the IAM User Guide.
Access to Splunk in VPC
If your Splunk platform is in a VPC, it must be publicly accessible with a public IP address. Also, grant Amazon Data Firehose access to your Splunk platform by unblocking the Amazon Data Firehose IP addresses. Amazon Data Firehose currently uses the following CIDR blocks.
-
18.216.68.160/27, 18.216.170.64/27, 18.216.170.96/27
for US East (Ohio) -
34.238.188.128/26, 34.238.188.192/26, 34.238.195.0/26
for US East (N. Virginia) -
13.57.180.0/26
for US West (N. California) -
34.216.24.32/27, 34.216.24.192/27, 34.216.24.224/27
for US West (Oregon) -
18.253.138.192/26
for AWS GovCloud (US-East) -
52.61.204.192/26
for AWS GovCloud (US-West) -
18.162.221.64/26
for Asia Pacific (Hong Kong) -
13.232.67.64/26
for Asia Pacific (Mumbai) -
13.209.71.0/26
for Asia Pacific (Seoul) -
13.229.187.128/26
for Asia Pacific (Singapore) -
13.211.12.0/26
for Asia Pacific (Sydney) -
13.230.21.0/27, 13.230.21.32/27
for Asia Pacific (Tokyo) -
51.16.102.64/26
for Israel (Tel Aviv) -
35.183.92.64/26
for Canada (Central) -
40.176.98.128/26
for Canada West (Calgary) -
18.194.95.192/27, 18.194.95.224/27, 18.195.48.0/27
for Europe (Frankfurt) -
34.241.197.32/27, 34.241.197.64/27, 34.241.197.96/27
for Europe (Ireland) -
18.130.91.0/26
for Europe (London) -
35.180.112.0/26
for Europe (Paris) -
13.53.191.0/26
for Europe (Stockholm) -
15.185.91.64/26
for Middle East (Bahrain) -
18.228.1.192/26
for South America (São Paulo) -
15.161.135.192/26
for Europe (Milan) -
13.244.165.128/26
for Africa (Cape Town) -
13.208.217.0/26
for Asia Pacific (Osaka) -
52.81.151.64/26
for China (Beijing) -
161.189.23.128/26
for China (Ningxia) -
108.136.221.128/26
for Asia Pacific (Jakarta) -
3.28.159.64/26
for Middle East (UAE) -
51.16.102.64/26
for Israel (Tel Aviv) -
16.62.183.64/26
for Europe (Zurich) -
18.60.192.192/26
for Asia Pacific (Hyderabad) -
16.50.161.192/26
for Asia Pacific (Melbourne)
Grant Firehose Access to a Snowflake Destination
When you're using Snowflake as a destination, Firehose delivers data to a Snowflake account using your Snowflake account URL. It also backs up error data to the Amazon Simple Storage Service bucket that you specify, and you can optionally use an AWS Key Management Service key that you own for Amazon S3 server-side encryption. If error logging is enabled, Firehose sends data delivery errors to your CloudWatch Logs streams.
You are required to have an IAM role when creating a delivery stream. Firehose assumes that IAM role and gains access to the specified bucket, key, and
CloudWatch Logs group and streams.
Use the following access policy to enable Firehose to access your S3 bucket. If you don't own the S3 bucket, add s3:PutObjectAcl
to the list
of Amazon Simple Storage Service actions, which grants the bucket owner full access to the objects delivered by Firehose. This policy also grants Firehose access to CloudWatch
for error logging. The policy also has a statement that allows access to Amazon Kinesis Data Streams. If you don't use Kinesis Data Streams as your data source,
you can remove that statement. Firehose doesn't use IAM to access Snowflake. For accessing Snowflake, it uses your Snowflake account Url and PrivateLink Vpce Id in the case of a private cluster.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::bucket-name", "arn:aws:s3:::bucket-name/*" ] }, { "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:GenerateDataKey" ], "Resource": [ "arn:aws:kms:region:account-id:key/key-id" ], "Condition": { "StringEquals": { "kms:ViaService": "s3.region.amazonaws.com" }, "StringLike": { "kms:EncryptionContext:aws:s3:arn": "arn:aws:s3:::bucket-name/prefix*" } } }, { "Effect": "Allow", "Action": [ "kinesis:DescribeStream", "kinesis:GetShardIterator", "kinesis:GetRecords", "kinesis:ListShards" ], "Resource": "arn:aws:kinesis:region:account-id:stream/stream-name" }, { "Effect": "Allow", "Action": [ "logs:PutLogEvents" ], "Resource": [ "arn:aws:logs:region:account-id:log-group:log-group-name:log-stream:*" ] } ] }
For more information about allowing other AWS services to access your AWS resources, see Creating a Role to Delegate Permissions to an AWS Service in the IAM User Guide.
Grant Amazon Data Firehose Access to an HTTP Endpoint Destination
You can use Amazon Data Firehose to deliver data to any HTTP endpoint destination. Amazon Data Firehose also backs up that data to the Amazon S3 bucket that you specify, and you can optionally use an AWS KMS key that you own for Amazon S3 server-side encryption. If error logging is enabled, Amazon Data Firehose sends data delivery errors to your CloudWatch log streams. You can also use AWS Lambda for data transformation.
You are required to have an IAM role when creating a Firehose stream. Amazon Data Firehose assumes that IAM role and gains access to the specified bucket, key, and CloudWatch log group and streams.
Use the following access policy to enable Amazon Data Firehose to access the S3 bucket that you
specified for data backup. If you don't own the S3 bucket, add
s3:PutObjectAcl
to the list of Amazon S3 actions, which grants the bucket
owner full access to the objects delivered by Amazon Data Firehose. This policy also grants Amazon Data Firehose
access to CloudWatch for error logging and to AWS Lambda for data transformation. The policy
also has a statement that allows access to Amazon Kinesis Data Streams. If you don't use Kinesis Data Streams as your
data source, you can remove that statement.
Important
Amazon Data Firehose doesn't use IAM to access HTTP endpoint destinations owned by supported third-party service providers, including Datadog, Dynatrace, LogicMonitor, MongoDB, New Relic, Splunk, or Sumo Logic. For accessing a specified HTTP endpoint destination owned by a supported third-party service provider, contact that service provider to obtain the API key or the access key that is required to enable data delivery to that service from Amazon Data Firehose.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::
bucket-name
", "arn:aws:s3:::bucket-name
/*" ] }, { "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:GenerateDataKey" ], "Resource": [ "arn:aws:kms:region
:account-id
:key/key-id
" ], "Condition": { "StringEquals": { "kms:ViaService": "s3.region
.amazonaws.com" }, "StringLike": { "kms:EncryptionContext:aws:s3:arn": "arn:aws:s3:::bucket-name/prefix
*" } } }, { "Effect": "Allow", "Action": [ "kinesis:DescribeStream", "kinesis:GetShardIterator", "kinesis:GetRecords", "kinesis:ListShards" ], "Resource": "arn:aws:kinesis:region
:account-id
:stream/stream-name
" }, { "Effect": "Allow", "Action": [ "logs:PutLogEvents" ], "Resource": [ "arn:aws:logs:region
:account-id
:log-group:log-group-name
:log-stream:*" ] }, { "Effect": "Allow", "Action": [ "lambda:InvokeFunction", "lambda:GetFunctionConfiguration" ], "Resource": [ "arn:aws:lambda:region
:account-id
:function:function-name
:function-version
" ] } ] }
For more information about allowing other AWS services to access your AWS resources, see Creating a Role to Delegate Permissions to an AWS Service in the IAM User Guide.
Important
Currently Amazon Data Firehose does NOT support data delivery to HTTP endpoints in a VPC.
Cross-Account Delivery from Amazon MSK
If yours is a cross-account scenario where you're creating a delivery stream from your Firehose account (for example, Account B) and your source is an MSK cluster in another AWS account (Account A), you must have the following configurations in place:
Account A:
In the Amazon MSK console, choose the provisioned cluster and then choose Properties.
Under Network settings, choose Edit and turn on Multi-VPC connectivity.
Under Security settings choose Edit cluster policy.
If the cluster does not already have a policy configured, check Include Firehose service principal and Enable Firehose cross-account S3 delivery. The AWS Management Console will automatically generate a policy with the appropriate permissions.
-
If the cluster already has a policy configured, add the following permissions to the existing policy:
{ "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::
arn
:role/mskaasTestDeliveryRole" }, "Action": [ "kafka:GetBootstrapBrokers", "kafka:DescribeCluster", "kafka:DescribeClusterV2", "kafka-cluster:Connect" ], "Resource": "arn:aws:kafka:us-east-1:arn
:cluster/DO-NOT-TOUCH-mskaas-provisioned-privateLink/xxxxxxxxx-2f3a-462a-ba09-xxxxxxxxxx-20" // ARN of the cluster }, { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::arn
:role/mskaasTestDeliveryRole" }, "Action": [ "kafka-cluster:DescribeTopic", "kafka-cluster:DescribeTopicDynamicConfiguration", "kafka-cluster:ReadData" ], "Resource": "arn:aws:kafka:us-east-1:arn
:topic/DO-NOT-TOUCH-mskaas-provisioned-privateLink/xxxxxxxxx-2f3a-462a-ba09-xxxxxxxxxx-20/*"//topic of the cluster }, { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::233450236687:role/mskaasTestDeliveryRole" }, "Action": "kafka-cluster:DescribeGroup", "Resource": "arn:aws:kafka:us-east-1:arn
:group/DO-NOT-TOUCH-mskaas-provisioned-privateLink/xxxxxxxxx-2f3a-462a-ba09-xxxxxxxxxx-20/*" //topic of the cluster }, }
Under AWS principal, enter the principal ID from Account B.
Under Topic, specify the Apache Kafka topic from which you want your delivery stream to ingest data. Once the delivery stream is created, you cannot update this topic.
Choose Save changes
Account B:
In the Firehose console, choose Create delivery stream using Account B.
Under Source, choose Amazon Managed Streaming for Apache Kafka.
Under Source settings, for the Amazon Managed Streaming for Apache Kafka cluster, enter the ARN of the Amazon MSK cluster in Account A.
Under Topic, specify the Apache Kafka topic from which you want your delivery stream to ingest data. Once the delivery stream is created, you cannot update this topic.
-
In Delivery stream name specify the name for your delivery stream.
In Account B when you're creating your delivery stream, you must have an IAM role (created by default when using the AWS Management Console) that grants the delivery stream 'read' access to the cross-account Amazon MSK cluster for the configured topic.
The following is what gets configured by the AWS Management Console:
{ "Sid": "", "Effect": "Allow", "Action": [ "kafka:GetBootstrapBrokers", "kafka:DescribeCluster", "kafka:DescribeClusterV2", "kafka-cluster:Connect" ], "Resource": "arn:aws:kafka:us-east-1:
arn
:cluster/DO-NOT-TOUCH-mskaas-provisioned-privateLink/xxxxxxxxx-2f3a-462a-ba09-xxxxxxxxxx-20/*" //topic of the cluster }, { "Sid": "", "Effect": "Allow", "Action": [ "kafka-cluster:DescribeTopic", "kafka-cluster:DescribeTopicDynamicConfiguration", "kafka-cluster:ReadData" ], "Resource": "arn:aws:kafka:us-east-1:arn
:topic/DO-NOT-TOUCH-mskaas-provisioned-privateLink/xxxxxxxxx-2f3a-462a-ba09-xxxxxxxxxx-20/mskaas_test_topic" //topic of the cluster }, { "Sid": "", "Effect": "Allow", "Action": [ "kafka-cluster:DescribeGroup" ], "Resource": "arn:aws:kafka:us-east-1:arn
:group/DO-NOT-TOUCH-mskaas-provisioned-privateLink/xxxxxxxxx-2f3a-462a-ba09-xxxxxxxxxx-20/*" //topic of the cluster }, }
Next, you can complete the optional step of configuring record transformation and record format conversion. For more information, see Record Transformation and Format Conversion.
Cross-Account Delivery to an Amazon S3 Destination
You can use the AWS CLI or the Amazon Data Firehose APIs to create a Firehose stream in one AWS account with an Amazon S3 destination in a different account. The following procedure shows an example of configuring a Firehose stream owned by account A to deliver data to an Amazon S3 bucket owned by account B.
-
Create an IAM role under account A using steps described in Grant Firehose Access to an Amazon S3 Destination.
Note
The Amazon S3 bucket specified in the access policy is owned by account B in this case. Make sure you add
s3:PutObjectAcl
to the list of Amazon S3 actions in the access policy, which grants account B full access to the objects delivered by Amazon Amazon Data Firehose. This permission is required for cross account delivery. Amazon Data Firehose sets the "x-amz-acl" header on the request to "bucket-owner-full-control". -
To allow access from the IAM role previously created, create an S3 bucket policy under account B. The following code is an example of the bucket policy. For more information, see Using Bucket Policies and User Policies.
{ "Version": "2012-10-17", "Id": "PolicyID", "Statement": [ { "Sid": "StmtID", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::
accountA-id
:role/iam-role-name
" }, "Action": [ "s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:PutObject", "s3:PutObjectAcl" ], "Resource": [ "arn:aws:s3:::bucket-name
", "arn:aws:s3:::bucket-name
/*" ] } ] } -
Create a Firehose stream under account A using the IAM role that you created in step 1.
Cross-Account Delivery to an OpenSearch Service Destination
You can use the AWS CLI or the Amazon Data Firehose APIs to create a Firehose stream in one AWS account with an OpenSearch Service destination in a different account. The following procedure shows an example of how you can create a Firehose stream under account A and configure it to deliver data to an OpenSearch Service destination owned by account B.
-
Create an IAM role under account A using the steps described in Grant Amazon Data Firehose Access to a Public OpenSearch Service Destination.
-
To allow access from the IAM role that you created in the previous step, create an OpenSearch Service policy under account B. The following JSON is an example.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::
Account-A-ID
:role/firehose_delivery_role " }, "Action": "es:ESHttpGet", "Resource": [ "arn:aws:es:us-east-1:Account-B-ID
:domain/cross-account-cluster/_all/_settings", "arn:aws:es:us-east-1:Account-B-ID
:domain/cross-account-cluster/_cluster/stats", "arn:aws:es:us-east-1:Account-B-ID
:domain/cross-account-cluster/roletest*/_mapping/roletest", "arn:aws:es:us-east-1:Account-B-ID
:domain/cross-account-cluster/_nodes", "arn:aws:es:us-east-1:Account-B-ID
:domain/cross-account-cluster/_nodes/stats", "arn:aws:es:us-east-1:Account-B-ID
:domain/cross-account-cluster/_nodes/*/stats", "arn:aws:es:us-east-1:Account-B-ID
:domain/cross-account-cluster/_stats", "arn:aws:es:us-east-1:Account-B-ID
:domain/cross-account-cluster/roletest*/_stats", "arn:aws:es:us-east-1:Account-B-ID
:domain/cross-account-cluster/" ] } ] } -
Create a Firehose stream under account A using the IAM role that you created in step 1. When you create the Firehose stream, use the AWS CLI or the Amazon Data Firehose APIs and specify the
ClusterEndpoint
field instead ofDomainARN
for OpenSearch Service.
Note
To create a Firehose stream in one AWS account with an OpenSearch Service destination in a different account, you must use the AWS CLI or the Amazon Data Firehose APIs. You can't use the AWS Management Console to create this kind of cross-account configuration.
Using Tags to Control Access
You can use the optional Condition
element (or Condition
block) in an IAM policy to fine-tune access to Amazon Data Firehose operations
based on tag keys and values. The following subsections describe how to do this for the
different Amazon Data Firehose operations. For more on the use of the Condition
element
and the operators that you can use within it, see IAM JSON Policy Elements: Condition.
CreateDeliveryStream
For the CreateDeliveryStream
operation, use the
aws:RequestTag
condition key. In the following example,
MyKey
and MyValue
represent the key and corresponding
value for a tag. For more information, see Tag Basics
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "firehose:CreateDeliveryStream", "Resource": "*", "Condition": { "StringEquals": { "aws:RequestTag/MyKey": "MyValue" } } } ] }
TagDeliveryStream
For the TagDeliveryStream
operation, use the aws:TagKeys
condition key. In the following example, MyKey
is an example tag
key.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "firehose:TagDeliveryStream", "Resource": "*", "Condition": { "ForAnyValue:StringEquals": { "aws:TagKeys": "MyKey" } } } ] }
UntagDeliveryStream
For the UntagDeliveryStream
operation, use the
aws:TagKeys
condition key. In the following example,
MyKey
is an example tag key.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "firehose:UntagDeliveryStream", "Resource": "*", "Condition": { "ForAnyValue:StringEquals": { "aws:TagKeys": "MyKey" } } } ] }
ListDeliveryStreams
You can't use tag-based access control with
ListDeliveryStreams
.
Other Amazon Data Firehose Operations
For all Amazon Data Firehose operations other than CreateDeliveryStream
,
TagDeliveryStream
, UntagDeliveryStream
, and
ListDeliveryStreams
, use the aws:RequestTag
condition
key. In the following example, MyKey
and MyValue
represent
the key and corresponding value for a tag.
ListDeliveryStreams
, use the firehose:ResourceTag
condition key to control access based on the tags on that Firehose stream.
In the following example, MyKey
and MyValue
represent
the key and corresponding value for a tag. The policy would only apply to Data
Firehose Firehose streams having a tag named MyKey
with a value of
MyValue
. For more information about controlling access based on
resource tags, see Controlling access to AWS resources using tags in the
IAM User Guide.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Deny", "Action": "firehose:DescribeDeliveryStream", "Resource": "*", "Condition": { "StringEquals": { "firehose:ResourceTag/MyKey": "MyValue" } } } ] }