Configuring an Apache Airflow connection using a Secrets Manager secret
AWS Secrets Manager is a supported alternative Apache Airflow backend on an Amazon Managed Workflows for Apache Airflow (MWAA) environment. This guide shows how to use AWS Secrets Manager to securely store secrets for Apache Airflow variables and an Apache Airflow connection on Amazon Managed Workflows for Apache Airflow (MWAA).
-
You will be charged for the secrets you create. For more information on Secrets Manager pricing, see AWS Pricing
. -
Amazon MWAA does not support AWS Systems Manager Parameter Store as a supported backend.
Contents
- Step one: Provide Amazon MWAA with permission to access Secrets Manager secret keys
- Step two: Create the Secrets Manager backend as an Apache Airflow configuration option
- Step three: Generate an Apache Airflow AWS connection URI string
- Step four: Add the variables in Secrets Manager
- Step five: Add the connection in Secrets Manager
- Sample code
- Resources
- What's next?
Step one: Provide Amazon MWAA with permission to access Secrets Manager secret keys
The execution role for your Amazon MWAA environment needs read access to the secret key in AWS Secrets Manager.
The following IAM policy allows read-write access using the AWS managed
SecretsManagerReadWrite
To attach the policy to your execution role
-
Open the Environments page
on the Amazon MWAA console. -
Choose an environment.
-
Choose your execution role on the Permissions pane.
-
Choose Attach policies.
-
Type
SecretsManagerReadWrite
in the Filter policies text field. -
Choose Attach policy.
If you do not want to use an AWS managed permission policy, you can directly update your environment's execution role to allow any level of access to your Secrets Manager resources. For example, the following policy statement grants read access to all secrets you create in a specific AWS Region in Secrets Manager.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:GetResourcePolicy", "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret", "secretsmanager:ListSecretVersionIds" ], "Resource": "arn:aws:secretsmanager:us-west-2:012345678910:secret:*" }, { "Effect": "Allow", "Action": "secretsmanager:ListSecrets", "Resource": "*" } ] }
Step two: Create the Secrets Manager backend as an Apache Airflow configuration option
The following section describes how to create an Apache Airflow configuration option on the Amazon MWAA console for the AWS Secrets Manager backend. If you're using a configuration setting of the same name in airflow.cfg
, the configuration you create in the following steps will take precedence and override the configuration settings.
Step three: Generate an Apache Airflow AWS connection URI string
The key to creating a connection URI string is to use the "tab" key on your keyboard to indent the key-value pairs in the Connectionextra
object in your shell session. The following section walks you through the steps to generate an Apache Airflow connection URI
Step four: Add the variables in Secrets Manager
The following section describes how to create the secret for a variable in Secrets Manager.
To create the secret
-
Open the AWS Secrets Manager console
. -
Choose Store a new secret.
-
Choose Other type of secret.
-
On the Specify the key/value pairs to be stored in this secret pane, choose Plaintext.
-
Add the variable value as Plaintext in the following format.
"
YOUR_VARIABLE_VALUE
"For example, to specify an integer:
14
For example, to specify a string:
"mystring"
-
For Encryption key, choose an AWS KMS key option from the dropdown list.
-
Enter a name in the text field for Secret name in the following format.
airflow/variables/
YOUR_VARIABLE_NAME
For example:
airflow/variables/test-variable
-
Choose Next.
-
On the Configure secret page, on the Secret name and description pane, do the following.
-
For Secret name, provide a name for your secret.
-
(Optional) For Description, provide a description for your secret.
Choose Next.
-
-
On the Configure rotation - optional leave the default options and choose Next.
-
Repeat these steps in Secrets Manager for any additional variables you want to add.
-
On the Review page, review your secret, then choose Store.
Step five: Add the connection in Secrets Manager
The following section describes how to create the secret for your connection string URI in Secrets Manager.
To create the secret
-
Open the AWS Secrets Manager console
. -
Choose Store a new secret.
-
Choose Other type of secret.
-
On the Specify the key/value pairs to be stored in this secret pane, choose Plaintext.
-
Add the connection URI string as Plaintext in the following format.
YOUR_CONNECTION_URI_STRING
For example:
mysql://288888a0-50a0-888-9a88-1a111aaa0000.a1.us-east-1.airflow.amazonaws.com%2Fhome?role_arn=arn%3Aaws%3Aiam%3A%3A001122332255%3Arole%2Fservice-role%2FAmazonMWAA-MyAirflowEnvironment-iAaaaA®ion_name=us-east-1
Warning Apache Airflow parses each of the values in the connection string. You must not use single nor double quotes, or it will parse the connection as a single string.
-
For Encryption key, choose an AWS KMS key option from the dropdown list.
-
Enter a name in the text field for Secret name in the following format.
airflow/connections/
YOUR_CONNECTION_NAME
For example:
airflow/connections/myconn
-
Choose Next.
-
On the Configure secret page, on the Secret name and description pane, do the following.
-
For Secret name, provide a name for your secret.
-
(Optional) For Description, provide a description for your secret.
Choose Next.
-
-
On the Configure rotation - optional leave the default options and choose Next.
-
Repeat these steps in Secrets Manager for any additional variables you want to add.
-
On the Review page, review your secret, then choose Store.
Sample code
-
Learn how to use the secret key for the Apache Airflow connection (
myconn
) on this page using the sample code at Using a secret key in AWS Secrets Manager for an Apache Airflow connection. -
Learn how to use the secret key for the Apache Airflow variable (
test-variable
) on this page using the sample code at Using a secret key in AWS Secrets Manager for an Apache Airflow variable.
Resources
-
For more information about configuring Secrets Manager secrets using the console and the AWS CLI, see Create a secret in the AWS Secrets Manager User Guide.
-
Use a Python script to migrate a large volume of Apache Airflow variables and connections to Secrets Manager in Move your Apache Airflow connections and variables to AWS Secrets Manager
.
What's next?
-
Learn how to generate a token to access the Apache Airflow UI in Accessing the Apache Airflow UI.