Authenticate with AWS Secrets Manager in Amazon Data Firehose

Amazon Data Firehose integrates with AWS Secrets Manager to provide secure access to your secrets and automate credential rotation. This integration allows Firehose to retrieve a secret from Secrets Manager at runtime to connect to previously mentioned streaming destinations and deliver your data streams. With this, your secrets are not visible in plain text during stream creation workflow either in AWS Management Console or API parameters. It provides a secure practice to manage your secrets and relieves you from complex credential management activities such as setting up custom Lambda functions to manage password rotations.

For more information, see the AWS Secrets Manager User Guide.

Understand secrets

A secret can be a password, a set of credentials such as a user name and password, an OAuth token, or other secret information that you store in an encrypted form in Secrets Manager.

For each destination, you must specify the secret key-value pair in the correct JSON format as shown in the following section. Amazon Data Firehose will fail to connect to your destination if your secret doesn't have the correct JSON format as per the destination.

Format of secret for Amazon Redshift Provisioned cluster and Amazon Redshift Serverless workgroup

{
    "username":  "<username>",
    "password":  "<password>"
}

Format of secret for Splunk

{
    "hec_token":  "<hec token>"
}

Format of secret for Snowflake

{
    "user":  "<user>",
    "private_key":  "<private_key>",
    "key_passphrase":  "<passphrase>" // optional
}

Format of secret for HTTP endpoint, Coralogix, Datadog, Dynatrace, Elastic, Honeycomb, LogicMonitor, Logz.io, MongoDB Cloud, and New Relic

{
    "api_key":  "<apikey>"
}

Create a secret

To create a secret, follow the steps in Create an AWS Secrets Manager secret in the AWS Secrets Manager User Guide.

Use the secret

We recommend that you use AWS Secrets Manager to store your credentials or keys to connect to streaming destinations such as Amazon Redshift, HTTP endpoint, Snowflake, Splunk, Coralogix, Datadog, Dynatrace, Elastic, Honeycomb, LogicMonitor, Logz.io, MongoDB Cloud, and New Relic.

You can configure authentication with Secrets Manager for these destinations through the AWS Management Console at the time of Firehose stream creation. For more information, see Configure destination settings. Alternatively, you can also use the CreateDeliveryStream and UpdateDestination API operations to configure authentication with Secrets Manager.

Firehose caches the secrets with an encryption and uses them for every connection to destinations. It refreshes the cache every 10 minutes to ensure that the latest credentials are used.

You can choose to turn off the capability of retrieving secrets from Secrets Manager at any time during the lifecycle of the stream. If you don’t want to use Secrets Manager to retrieve secrets, you can use the username/password or API key instead.

Note

Although, there is no additional cost for this feature in Firehose, you are billed for access and maintenance of Secrets Manager. For more information, see AWS Secrets Manager pricing page.

Grant access to Firehose to retrieve the secret

For Firehose to retrieve a secret from AWS Secrets Manager, you must provide Firehose the required permissions to access the secret and the key that encrypts your secret.

When using AWS Secrets Manager to store and retrieve secrets, there are a few different configuration options depending on where the secret is stored and how it is encrypted.

• If the secret is stored in the same AWS account as your IAM role and it's encrypted with the default AWS managed key (aws/secretsmanager), the IAM role that Firehose assumes only needs secretsmanager:GetSecretValue permission on the secret.

  // secret role policy
  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": "secretsmanager:GetSecretValue",
              "Resource": "Secret ARN"
          }
      ]
  }

  For more information on IAM policies, see Permissions policy examples for AWS Secrets Manager.

• If the secret is stored in the same account as the role but encrypted with a customer managed key (CMK), the role needs both secretsmanager:GetSecretValue and kms:Decrypt permissions. The CMK policy also needs to allow the IAM role to perform kms:Decrypt.

  {
      "Version": "2012-10-17",
      "Statement": [{
              "Effect": "Allow",
              "Action": "secretsmanager:GetSecretValue",
              "Resource": "Secret ARN"
          },
          {
              "Effect": "Allow",
              "Action": "kms:Decrypt",
              "Resource": "KMSKeyARN"
          }
      ]
  }

• If the secret is stored in a different AWS account than your role, and it is encrypted with the default AWS managed key, this configuration is not possible as Secrets Manager does not allow cross-account access when secret is encrypted with AWS managed key.

• If the secret is stored in a different account and encrypted with a CMK, IAM role needs secretsmanager:GetSecretValue permission on the secret and kms:Decrypt permission on the CMK. The secret's resource policy and the CMK policy in the other account also need to allow the IAM role the necessary permissions. For more information, see Cross-account access.

Rotate the secret

Rotation is when you periodically update a secret. You can configure AWS Secrets Manager to automatically rotate the secret for you on a schedule that you specify. This way, you can replace long-term secrets with short-term ones. This helps to reduce the risk of compromise. For more information, see Rotate AWS Secrets Manager secrets in the AWS Secrets Manager User Guide.