Step one: Permit access of the secret keys to Amazon MWAA Step two: Create the Secrets Manager backend as an Apache Airflow configuration option Step three: Generate an Apache Airflow AWS connection URI string Step four: Add the variables in Secrets Manager Step five: Add the connection in Secrets Manager Sample code Using AWS blogs and tutorials What's next?

Configuring an Apache Airflow connection using a Secrets Manager secret key

AWS Secrets Manager is a supported alternative Apache Airflow backend on an Amazon Managed Workflows for Apache Airflow (MWAA) environment. This guide shows how to use AWS Secrets Manager to securely store secrets for Apache Airflow variables and an Apache Airflow connection on Amazon Managed Workflows for Apache Airflow (MWAA).

Note

AWS Systems Manager Parameter Store is not a supported backend at this time on Amazon MWAA.

Contents

Step one: Permit access of the secret keys to Amazon MWAA

The execution role for your Amazon MWAA environment needs read access to the secret key in AWS Secrets Manager. The following IAM policy allows read-write access using the AWS managed SecretsManagerReadWrite policy.

To attach the policy to your execution role

Open the Environments page on the Amazon MWAA console.
Choose an environment.
Choose your execution role on the Permissions pane.
Choose Attach policies.
Type SecretsManagerReadWrite in the Filter policies text field.
Choose Attach policy.

Step two: Create the Secrets Manager backend as an Apache Airflow configuration option

The following section describes how to create an Apache Airflow configuration option on the Amazon MWAA console for the AWS Secrets Manager backend. If you're using a configuration setting of the same name in airflow.cfg, the configuration you create in the following steps will take precedence and override the configuration settings.

Airflow v2.0.2

Open the Environments page on the Amazon MWAA console.
Choose an environment.
Choose Edit.
Choose Next.
Choose Add custom configuration in the Airflow configuration options pane. Add the following key-value pairs:
1. secrets.backend : airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
2. secrets.backend_kwargs : {"connections_prefix" : "airflow/connections", "variables_prefix" : "airflow/variables"}
  
  This tells Apache Airflow to look for the secret at the airflow/connections/* and airflow/variables/* path.
Choose Save.

Airflow v1.10.12

Open the Environments page on the Amazon MWAA console.
Choose an environment.
Choose Edit.
Choose Next.
Choose Add custom configuration in the Airflow configuration options pane. Add the following key-value pairs:
1. secrets.backend : airflow.contrib.secrets.aws_secrets_manager.SecretsManagerBackend
2. secrets.backend_kwargs : {"connections_prefix" : "airflow/connections", "variables_prefix" : "airflow/variables"}
  
  This tells Apache Airflow to look for the secret at the airflow/connections/* and airflow/variables/* path.
Choose Save.

Step three: Generate an Apache Airflow AWS connection URI string

The key to creating a connection URI string is to use the "tab" key on your keyboard to indent the key-value pairs in the Connection object. We also recommend creating a variable for the extra object in your shell session. The following section walks you through the steps to generate an Apache Airflow connection URI string for an Amazon MWAA environment using Apache Airflow or a Python script.

Airflow CLI

The following shell session uses your local Airflow CLI to generate a connection string. If you don't have the CLI installed, we recommend using the Python script.

Open a Python shell session:
```
python3
```
Enter the following command:
```
>>> import json
```

Enter the following command:


>>> from airflow.models.connection import Connection

Create a variable in your shell session for the extra object. Substitute the sample values in YOUR_EXECUTION_ROLE_ARN with the execution role ARN, and the region in YOUR_REGION (such as us-east-1).
```
>>> extra=json.dumps({'role_arn': 'YOUR_EXECUTION_ROLE_ARN', 'region_name': 'YOUR_REGION'})
                        
```
Create the connection object. Substitute the sample value in myconn with the name of the Apache Airflow connection.
```
>>> myconn = Connection(
```
Use the "tab" key on your keyboard to indent each of the following key-value pairs in your connection object. Substitute the sample values in red.
1. Specify the AWS connection type:
```
... conn_id='aws',
```
2. Specify the Apache Airflow database option:
```
... conn_type='mysql',
```
3. Specify the Apache Airflow UI URL on Amazon MWAA:
```
... host='288888a0-50a0-888-9a88-1a111aaa0000.a1.us-east-1.airflow.amazonaws.com/home',
```
4. Specify the AWS access key ID (username) to login to Amazon MWAA:
```
... login='YOUR_AWS_ACCESS_KEY_ID',
```
5. Specify the AWS secret access key (password) to login to Amazon MWAA:
```
... password='YOUR_AWS_SECRET_ACCESS_KEY',
```
6. Specify the extra shell session variable:
```
... extra=extra
```
7. Close the connection object.
```
... )
```

Print the connection URI string:


>>> myconn.get_uri()

You should see the connection URI string in the response:


'mysql://288888a0-50a0-888-9a88-1a111aaa0000.a1.us-east-1.airflow.amazonaws.com%2Fhome?role_arn=arn%3Aaws%3Aiam%3A%3A001122332255%3Arole%2Fservice-role%2FAmazonMWAA-MyAirflowEnvironment-iAaaaA&region_name=us-east-1'

Python script

The following Python script does not require the Apache Airflow CLI.

Copy the contents of the following code sample and save locally as mwaa_connection.py.


import urllib.parse

conn_type = 'YOUR_DB_OPTION'
host = 'YOUR_MWAA_AIRFLOW_UI_URL'
port = 'YOUR_PORT'
login = 'YOUR_AWS_ACCESS_KEY_ID'
password = 'YOUR_AWS_SECRET_ACCESS_KEY'
role_arn = urllib.parse.quote_plus('YOUR_EXECUTION_ROLE_ARN')
region_name = 'YOUR_REGION'

conn_string = '{0}://{1}:{2}@{3}:{4}?role_arn={5}&region_name={6}'.format(conn_type, login, password, host, port, role_arn, region_name)
print(conn_string)

Substitute the placeholders in red.
Run the following script to generate a connection string.
```
python3 mwaa_connection.py
```

Step four: Add the variables in Secrets Manager

The following section describes how to create the secret for a variable in Secrets Manager.

To create the secret key

Open the AWS Secrets Manager console.
Choose Store a new secret.
Choose Other type of secrets.
Choose Plaintext on the Specify the key/value pairs to be stored in this secret pane.
Add the variable value as Plaintext in the following format.
```
"YOUR_VARIABLE_VALUE"
```
For example, to specify an integer:
```
14
```
For example, to specify a string:
```
"mystring"
```
Choose an AWS KMS key option from the dropdown list.
Enter a name in the text field for Secret name in the following format.
```
airflow/variables/YOUR_VARIABLE_NAME
```
For example:
```
airflow/variables/test-variable
```
Leave the remaining options blank, or set to their default values.
Choose Next, Next, Store.
Repeat these steps in Secrets Manager for any additional variables you want to add.

Step five: Add the connection in Secrets Manager

The following section describes how to create the secret for your connection string URI in Secrets Manager.

To create the secret key

Open the AWS Secrets Manager console.
Choose Store a new secret.
Choose Other type of secrets.
Choose Plaintext on the Specify the key/value pairs to be stored in this secret pane.
Add the connection URI string as Plaintext in the following format.
```
YOUR_CONNECTION_URI_STRING
```
For example:
```
mysql://288888a0-50a0-888-9a88-1a111aaa0000.a1.us-east-1.airflow.amazonaws.com%2Fhome?role_arn=arn%3Aaws%3Aiam%3A%3A001122332255%3Arole%2Fservice-role%2FAmazonMWAA-MyAirflowEnvironment-iAaaaA&region_name=us-east-1
```
Warning

Apache Airflow parses each of the values in the connection string. You must not use single nor double quotes, or it will parse the connection as a single string.
Choose an AWS KMS key option from the dropdown list.
Enter a name in the text field for Secret name in the following format.
```
airflow/connections/YOUR_CONNECTION_NAME
```
For example:
```
airflow/connections/myconn
```
Leave the remaining options blank, or set to their default values.
Choose Next, Next, Store.

Sample code

Learn how to use the secret key for the Apache Airflow connection (myconn) on this page using the sample code at Using a secret key in AWS Secrets Manager for an Apache Airflow connection.
Learn how to use the secret key for the Apache Airflow variable (test-variable) on this page using the sample code at Using a secret key in AWS Secrets Manager for an Apache Airflow variable.

Using AWS blogs and tutorials

Use a Python script to migrate a large volume of Apache Airflow variables and connections to Secrets Manager in Move your Apache Airflow connections and variables to AWS Secrets Manager.

What's next?

Learn how to generate a token to access the Apache Airflow UI in Accessing the Apache Airflow UI.

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

Connection types

Configuring environments