Lookup order Default connections Choosing a storage option Secrets Manager backend Execution role permissions Configuring the backend Environment variable connections Combining variables Example Limitations Troubleshooting

Managing connections and variables

Apache Airflow connections store the credentials and endpoint information that your tasks use to reach external systems, such as a database or an HTTP API. Apache Airflow variables store reusable configuration values as key-value pairs. Your task code retrieves a connection with BaseHook.get_connection and a variable with Variable.get. Operators that take a connection ID, such as http_conn_id, make the same lookup for you.

With Amazon MWAA Serverless, you use the native Apache Airflow mechanisms for connections and variables. For background on these concepts, see Managing connections, Variables, and Secrets backend in the Apache Airflow documentation.

You can store connections and variables in a secrets backend that you configure for the workflow, such as AWS Secrets Manager or AWS Systems Manager Parameter Store. You can also define connections and variables that don't contain secrets as environment variables.

Topics

How Amazon MWAA Serverless looks up connections and variables
Default connections
Choosing where to store connections and variables
Using the Secrets Manager backend
Execution role permissions
Configuring a secrets backend
Defining connections and variables with environment variables
Combining backend settings with other environment variables
Example – Reading a connection and a variable in a task
Limitations for connections and variables
Troubleshooting connections and variables

How Amazon MWAA Serverless looks up connections and variables

When a task looks up a connection or a variable, Apache Airflow checks the following locations in order and returns the first value it finds:

The secrets backend, if you configured one.
Environment variables: AIRFLOW_CONN_CONN_ID for connections and AIRFLOW_VAR_KEY for variables. For more information, see Defining connections and variables with environment variables.

Important

When you use the Secrets Manager backend, every connection or variable lookup calls the Secrets Manager API. This includes lookups for keys that don't exist in Secrets Manager. These calls can increase your Secrets Manager charges. To limit the lookups, use lookup patterns, as described in Option A: Secrets Manager with custom prefixes.

The same lookup applies when you reference a connection or variable through Jinja templating in your workflow definition, with {{ conn.conn_id }}, {{ var.value.key }}, or {{ var.json.key }}. In the following example, a BashOperator task reads the host of the my_http connection and the value of the app_stage variable:


templated_dag:
  start_date: "2024-01-01"
  schedule: null
  tasks:
    print_values:
      operator: airflow.providers.standard.operators.bash.BashOperator
      bash_command: 'echo "host={{ conn.my_http.host }} stage={{ var.value.app_stage }}"'

For the list of supported template variables, see Template variables, filters, and macros.

If neither location has a value, the lookup fails: BaseHook.get_connection raises AirflowNotFoundException, Variable.get raises KeyError, and a template that references the missing key fails to render.

Default connections

Amazon MWAA Serverless provides two default Apache Airflow connections to every task, so you don't need to define them:

aws_default (AIRFLOW_CONN_AWS_DEFAULT=aws://) – uses your workflow execution role credentials.
duckdb_default (AIRFLOW_CONN_DUCKDB_DEFAULT={"conn_type": "duckdb_aws"}) – the default DuckDB connection, of type duckdb_aws.

You can overwrite either default connection by setting the same key in Environment.Variables. For example, the following entry changes aws_default for every task so that it targets the eu-west-1 Region:


"AIRFLOW_CONN_AWS_DEFAULT": "aws://?region_name=eu-west-1"

Note

AWS operators use the aws_default connection unless a task sets aws_conn_id. Overriding aws_default affects every such task in the workflow.

Choosing where to store connections and variables

Use the following guidance to choose where to store a value:

Secrets backend – Use for credentials, tokens, and any other sensitive value. You can use AWS Secrets Manager, AWS Systems Manager Parameter Store, or your own backend class, with your own prefixes, lookup patterns, or secrets in another account. For more information, see Configuring a secrets backend.
Environment variables – Use only for non-secret values, such as a public endpoint or a stage name, because GetWorkflow returns them in plain text. For more information, see Defining connections and variables with environment variables.

The following table compares the storage options.

Storage option	Where values are stored	Configuration needed	Use for secrets?
Secrets Manager backend	AWS Secrets Manager secrets under prefixes that you choose, optionally in another account.	Secrets backend environment variables	Yes
AWS Systems Manager Parameter Store	Parameter Store parameters, stored as `SecureString` for sensitive values.	Secrets backend environment variables	Yes
Custom secrets backend class	Any store that your own backend class reads. You include the class in your code package.	Secrets backend environment variables and code package	Yes, depending on the store that you use.
Environment variables	The workflow's `Environment` block, which `GetWorkflow` returns in plain text.	None	No

Using the Secrets Manager backend

To read connections and variables from Secrets Manager, set the backend class airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend and its keyword arguments (connections_prefix, variables_prefix, and region_name) in the workflow's Environment block, as in the following example. Replace us-east-1 with the Region of your secrets.


--environment '{"Variables": {
    "AIRFLOW__WORKERS__SECRETS_BACKEND": "airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend",
    "AIRFLOW__WORKERS__SECRETS_BACKEND_KWARGS": "{\"connections_prefix\": \"airflow/connections\", \"variables_prefix\": \"airflow/variables\", \"region_name\": \"us-east-1\"}"
}}'

For more information about these variables, see Configuring a secrets backend. For more information about this backend class, see Secrets Manager backend in the Apache Airflow documentation.

To read secrets in another account, see Option D: Secrets Manager secrets in another account.

Apache Airflow builds the secret name from the prefix and the key that you pass to the lookup call. With the prefixes in the preceding example, the names are as shown in the following table.

Lookup call in your task	Secrets Manager secret name
`BaseHook.get_connection("my_db")`	`airflow/connections/my_db`
`Variable.get("my_variable")`	`airflow/variables/my_variable`

Store the secret value in the format that matches the lookup:

For a connection, store an Apache Airflow connection URI string, such as postgresql://my-user:my-password@mydb.example.com:5432/app. Don't wrap the URI in quotation marks, and URL-encode special characters in the user name and password. Apache Airflow also accepts a JSON representation of a connection.
For a variable, store the plaintext value.

You can both create new secrets and update existing ones. Grant your execution role permission to read them, as described in Execution role permissions.

Creating a connection or variable secret

Use the following AWS CLI create-secret commands to create a connection secret and a variable secret. Before you run them, replace the following values:

The connection URI in the first command – the Apache Airflow connection URI for your external system
my-value in the second command – the value of your variable
us-east-1 – the Region of your workflow


aws secretsmanager create-secret \
    --name airflow/connections/my_db \
    --secret-string 'postgresql://my-user:my-password@mydb.example.com:5432/app' \
    --region us-east-1


aws secretsmanager create-secret \
    --name airflow/variables/my_variable \
    --secret-string 'my-value' \
    --region us-east-1

Updating a connection or variable secret

To change the value of an existing connection or variable, store a new version of the secret with put-secret-value. Replace the values and the Region as in the create commands.


aws secretsmanager put-secret-value \
    --secret-id airflow/connections/my_db \
    --secret-string 'postgresql://my-user:my-new-password@mydb.example.com:5432/app' \
    --region us-east-1


aws secretsmanager put-secret-value \
    --secret-id airflow/variables/my_variable \
    --secret-string 'my-new-value' \
    --region us-east-1

After you change a secret, you don't need to update the workflow. Your tasks read the secret each time they look it up, so you get the new value in any task that starts after the change.

Important

Secrets Manager charges for the secrets that you store and for the API calls that Amazon MWAA Serverless makes to retrieve them. For more information about pricing, see AWS Secrets Manager pricing. To reduce the number of API calls, you can limit which keys Apache Airflow looks up in the backend with lookup patterns, as described in Option A: Secrets Manager with custom prefixes.

Execution role permissions

Your workflow execution role needs permission to read the secrets or parameters that hold your connections and variables. Add the statements that match the backend that you use. For more information about execution roles, see Execution roles.

The following policy grants the secretsmanager:GetSecretValue permission on secrets stored under the airflow/connections/ and airflow/variables/ prefixes in one account and Region. The Secrets Manager backend needs this permission when you use these prefixes.

Replace us-east-1 with your Region and 123456789012 with your account ID. If you use different prefixes, change airflow/connections/* and airflow/variables/* in the Resource element to match your prefixes.

If you use the AWS Systems Manager Parameter Store backend (Option B in Configuring a secrets backend), grant the ssm:GetParameter and ssm:GetParameters permissions on parameters stored under the /airflow/connections/ and /airflow/variables/ prefixes instead.

Replace us-east-1 with your Region and 123456789012 with your account ID. Adjust the parameter prefixes to match the prefixes that you configure in the backend.

If you encrypt your secrets or SecureString parameters with a customer managed AWS KMS key, also grant the kms:Decrypt permission on that key.

Replace us-east-1 with your Region, 123456789012 with your account ID, and the key ID 1234abcd-12ab-34cd-56ef-1234567890ab with the ID of your key.

Note

If you use a custom secrets backend class, grant your execution role the permissions for the API calls that the class makes. The required permissions depend on the store that your backend reads.

Configuring a secrets backend

To configure where Amazon MWAA Serverless reads connections and variables, set both of the following environment variables in the workflow's Environment block:

AIRFLOW__WORKERS__SECRETS_BACKEND – the full class path of the backend.
AIRFLOW__WORKERS__SECRETS_BACKEND_KWARGS – a JSON string of keyword arguments for the backend.

The AIRFLOW__WORKERS__SECRETS_BACKEND_KWARGS value is a JSON string inside the Environment JSON. Escape its inner quotation marks with a backslash, as shown in the following examples.

These variables map to the Apache Airflow 3 [workers] secrets_backend and secrets_backend_kwargs settings, which configure the backend that task workers use. Set them with the Environment parameter of CreateWorkflow or the Environment parameter of UpdateWorkflow.

Option A: Secrets Manager with custom prefixes

To keep your connections and variables under prefixes that you choose, use the Secrets Manager backend class with custom connections_prefix and variables_prefix values. The following AWS CLI command creates a workflow with this configuration. Replace my-workflow with your workflow name. Replace dags/my_dag.yaml with the Amazon S3 object key of your YAML workflow definition. Replace the role ARN with your execution role ARN, and replace us-east-1 with your Region.


aws mwaa-serverless create-workflow \
    --name my-workflow \
    --definition-s3-location '{"Bucket": "amzn-s3-demo-bucket", "ObjectKey": "dags/my_dag.yaml"}' \
    --environment '{"Variables": {"AIRFLOW__WORKERS__SECRETS_BACKEND": "airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend", "AIRFLOW__WORKERS__SECRETS_BACKEND_KWARGS": "{\"connections_prefix\": \"my-app/connections\", \"variables_prefix\": \"my-app/variables\", \"region_name\": \"us-east-1\"}"}}' \
    --role-arn arn:aws:iam::111122223333:role/MyMWAAServerlessRole \
    --region us-east-1

With this configuration, BaseHook.get_connection("my_db") reads the secret my-app/connections/my_db, and Variable.get("my_variable") reads my-app/variables/my_variable.

To reduce the number of Secrets Manager API calls, add the connections_lookup_pattern and variables_lookup_pattern keyword arguments. Each is a regular expression, and Apache Airflow queries the backend only for keys that match. In the following example, Apache Airflow looks up only connection IDs my_db and my_http, and only variables whose keys start with my_:


--environment '{"Variables": {"AIRFLOW__WORKERS__SECRETS_BACKEND": "airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend", "AIRFLOW__WORKERS__SECRETS_BACKEND_KWARGS": "{\"connections_prefix\": \"my-app/connections\", \"variables_prefix\": \"my-app/variables\", \"region_name\": \"us-east-1\", \"connections_lookup_pattern\": \"^(my_db|my_http)$\", \"variables_lookup_pattern\": \"^my_\"}"}}'

Option B: AWS Systems Manager Parameter Store

To read connections and variables from AWS Systems Manager Parameter Store, use the Parameter Store backend class. Parameter names begin with a forward slash, so get_connection("my_db") reads the parameter /airflow/connections/my_db:


--environment '{"Variables": {"AIRFLOW__WORKERS__SECRETS_BACKEND": "airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend", "AIRFLOW__WORKERS__SECRETS_BACKEND_KWARGS": "{\"connections_prefix\": \"/airflow/connections\", \"variables_prefix\": \"/airflow/variables\", \"region_name\": \"us-east-1\"}"}}'

Store sensitive values as a SecureString parameter. Use the following AWS CLI put-parameter commands to create a connection parameter and a variable parameter. Replace the connection URI with the URI for your system, my-value with your variable value, and us-east-1 with the Region of your workflow.


aws ssm put-parameter \
    --name /airflow/connections/my_db \
    --type SecureString \
    --value 'postgresql://my-user:my-password@mydb.example.com:5432/app' \
    --region us-east-1


aws ssm put-parameter \
    --name /airflow/variables/my_variable \
    --type SecureString \
    --value 'my-value' \
    --region us-east-1

To update an existing parameter, run the same put-parameter command with the new value and the --overwrite option:


aws ssm put-parameter \
    --name /airflow/connections/my_db \
    --type SecureString \
    --value 'postgresql://my-user:my-new-password@mydb.example.com:5432/app' \
    --overwrite \
    --region us-east-1

For more information about this backend class, see Parameter Store backend in the Apache Airflow documentation.

Grant your execution role the Parameter Store permissions shown in Execution role permissions, and kms:Decrypt if your SecureString parameters use a customer managed key.

Option C: Your own secrets backend class

To read connections and variables from a store that the built-in backends don't support, write your own backend class. Subclass airflow.secrets.BaseSecretsBackend and implement two methods:

get_conn_value(conn_id) – return a connection URI or JSON string, or None so that Apache Airflow continues to the next lookup location.
get_variable(key) – return the variable value as a string, or None.

Apache Airflow passes each key in the AIRFLOW__WORKERS__SECRETS_BACKEND_KWARGS JSON to your class's __init__ method as a keyword argument. The following example reads from a store of your choice. It returns None when a secret doesn't exist, and logs and re-raises any other error so that a failed lookup isn't silently treated as a missing value.


# my_secrets_backend.py
from airflow.secrets import BaseSecretsBackend


class MySecretsBackend(BaseSecretsBackend):
    """Reads Airflow connections and variables from your own secret store."""

    def __init__(self, prefix="airflow", **kwargs):
        super().__init__(**kwargs)
        self.prefix = prefix

    def _fetch_from_store(self, path):
        # Replace with a call to your secret store, using the workflow
        # execution role credentials. Raise LookupError if the secret
        # doesn't exist.
        raise LookupError(path)

    def _read_secret(self, path):
        try:
            return self._fetch_from_store(path)
        except LookupError:
            # The secret doesn't exist. Return None so that Airflow
            # continues to the next lookup location.
            return None
        except Exception:
            self.log.exception("Failed to read secret %s", path)
            raise

    def get_conn_value(self, conn_id):
        # Return a connection URI or JSON string, or None so that Airflow
        # continues to the next lookup location.
        return self._read_secret(f"{self.prefix}/connections/{conn_id}")

    def get_variable(self, key):
        # Return the variable value as a string, or None.
        return self._read_secret(f"{self.prefix}/variables/{key}")

Set the backend class path and its keyword arguments in the Environment block:


--environment '{"Variables": {"AIRFLOW__WORKERS__SECRETS_BACKEND": "my_secrets_backend.MySecretsBackend", "AIRFLOW__WORKERS__SECRETS_BACKEND_KWARGS": "{\"prefix\": \"my-app\"}"}}'

Include the module in your workflow code package, and pass the package in the --code parameter. You don't need a separate install or registration step. Amazon MWAA Serverless extracts your code onto the task's import path. The class path that you set in AIRFLOW__WORKERS__SECRETS_BACKEND is module.Class for a module at the root of the package, or package.module.Class for a module inside a folder. For more information about building a code package, see Preparing your code package.

Your backend class needs the IAM permissions for the API calls that its code makes. Grant those permissions to your execution role.

The following AWS CLI command creates a workflow that uses the custom backend. Replace my-workflow with your workflow name. Replace dags/my_dag.yaml and code/my_package.zip with the Amazon S3 object keys of your YAML workflow definition and your code package. Replace the role ARN with your execution role ARN, and replace us-east-1 with your Region.


aws mwaa-serverless create-workflow \
    --name my-workflow \
    --definition-s3-location '{"Bucket": "amzn-s3-demo-bucket", "ObjectKey": "dags/my_dag.yaml"}' \
    --code '{"S3Location": {"Bucket": "amzn-s3-demo-bucket", "ObjectKey": "code/my_package.zip"}}' \
    --environment '{"Variables": {"AIRFLOW__WORKERS__SECRETS_BACKEND": "my_secrets_backend.MySecretsBackend", "AIRFLOW__WORKERS__SECRETS_BACKEND_KWARGS": "{\"prefix\": \"my-app\"}"}}' \
    --role-arn arn:aws:iam::111122223333:role/MyMWAAServerlessRole \
    --region us-east-1

Option D: Secrets Manager secrets in another account

To read Secrets Manager secrets that are in another account, set connections_prefix and variables_prefix to the ARN prefix of the secrets in that account. In the following example, the workflow runs in account 111122223333 and the secrets are in account 444455556666:


--environment '{"Variables": {"AIRFLOW__WORKERS__SECRETS_BACKEND": "airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend", "AIRFLOW__WORKERS__SECRETS_BACKEND_KWARGS": "{\"connections_prefix\": \"arn:aws:secretsmanager:us-east-1:444455556666:secret:airflow/connections\", \"variables_prefix\": \"arn:aws:secretsmanager:us-east-1:444455556666:secret:airflow/variables\", \"region_name\": \"us-east-1\"}"}}'

With this configuration, BaseHook.get_connection("my_conn") reads the secret arn:aws:secretsmanager:us-east-1:444455556666:secret:airflow/connections/my_conn. The two prefixes can point at the same account or at different accounts.

Standard Secrets Manager cross-account requirements apply:

The secret's resource policy must allow the workflow execution role to read it.
The secret must be encrypted with a customer managed AWS KMS key whose key policy allows the execution role to use it. You can't use the AWS managed key aws/secretsmanager for cross-account access.
The execution role's identity policy must allow secretsmanager:GetSecretValue on the secrets and kms:Decrypt on the key.

In the account that owns the secrets (444455556666), attach a resource policy to each secret that allows your workflow execution role in account 111122223333 to call secretsmanager:GetSecretValue.

In the workflow account (111122223333), grant your execution role permission to call secretsmanager:GetSecretValue on the secrets and kms:Decrypt on the customer managed key in the other account (444455556666).

For more information about cross-account access to secrets, see Permissions to access secrets in a different account in the AWS Secrets Manager User Guide.

Defining connections and variables with environment variables

You can define an Apache Airflow connection without a secrets backend by setting an AIRFLOW_CONN_CONN_ID environment variable to an Apache Airflow connection URI. Apache Airflow parses the URI into connection fields.

For example, set AIRFLOW_CONN_MY_HTTP to http://api.example.com:8443/v1?timeout=30. BaseHook.get_connection("my_http") then returns a connection with the following fields:

Host: api.example.com
Port: 8443
Schema: v1
Extra: {"timeout": "30"}

Similarly, set an AIRFLOW_VAR_KEY environment variable to define a variable, which Variable.get("key") then returns.

For more information about the connection URI format, see Storing connections in environment variables and Storing variables in environment variables in the Apache Airflow documentation.

If you configured a secrets backend, Apache Airflow looks up a connection or variable there first, and checks environment variables only if the backend returns no value. For more information about the lookup order, see How Amazon MWAA Serverless looks up connections and variables. For instructions on setting environment variables and their limits, see Using environment variables in Amazon MWAA Serverless workflows.

Note

Use environment variables only for connections and variables that don't contain secrets. Keep credentials and other sensitive values in a secrets backend.

Combining backend settings with other environment variables

The secrets backend variables, AIRFLOW_CONN_CONN_ID and AIRFLOW_VAR_KEY entries, and your own application variables all go in the single Variables map in the workflow's Environment block. The following example combines backend settings, a connection, and two application variables. Replace the secrets prefixes my-app/connections and my-app/variables, the Region us-east-1, and the connection URI with your own values.


--environment '{"Variables": {"AIRFLOW__WORKERS__SECRETS_BACKEND": "airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend", "AIRFLOW__WORKERS__SECRETS_BACKEND_KWARGS": "{\"connections_prefix\": \"my-app/connections\", \"variables_prefix\": \"my-app/variables\", \"region_name\": \"us-east-1\"}", "AIRFLOW_CONN_MY_HTTP": "http://api.example.com:8443/v1?timeout=30", "APP_STAGE": "production", "FEATURE_X_ENABLED": "true"}}'

The example sets the following variables:

AIRFLOW__WORKERS__SECRETS_BACKEND – The secrets backend class. This example uses the Secrets Manager backend class.
AIRFLOW__WORKERS__SECRETS_BACKEND_KWARGS – A JSON string of keyword arguments for the backend class. This example sets the connections prefix, the variables prefix, and the Region.
AIRFLOW_CONN_MY_HTTP – An Apache Airflow connection URI that defines the my_http connection.
APP_STAGE and FEATURE_X_ENABLED – Your own application variables. Your task code can read them as environment variables.

Important

Define each connection in only one place: either the secrets backend or an AIRFLOW_CONN_CONN_ID environment variable, not both. If both exist, Apache Airflow uses the secrets backend value.

Example – Reading a connection and a variable in a task

This example reads one connection and one variable with the Secrets Manager backend. For a version that reads the same values from a custom operator, see Example: a custom operator that reads a connection and a variable.

First, create a connection secret and a variable secret in the workflow's account and Region:


aws secretsmanager create-secret \
    --name airflow/connections/my_db \
    --secret-string 'postgresql://my-user:my-password@mydb.example.com:5432/app' \
    --region us-east-1

aws secretsmanager create-secret \
    --name airflow/variables/my_variable \
    --secret-string 'my-value' \
    --region us-east-1

Next, write a Python callable that reads the connection and the variable. The callable logs only non-secret fields, such as the host, port, and schema of the connection, and the value of a non-secret variable.


# my_module.py
import logging

from airflow.exceptions import AirflowException, AirflowNotFoundException

try:
    from airflow.sdk import BaseHook, Variable  # Apache Airflow 3
except ImportError:
    from airflow.hooks.base import BaseHook
    from airflow.models import Variable

logger = logging.getLogger(__name__)


def read_connection_and_variable():
    # Look up a connection through the configured secrets backend.
    try:
        conn = BaseHook.get_connection("my_db")
    except AirflowNotFoundException as e:
        raise AirflowException("Connection my_db not found") from e
    # Log only non-secret fields. Never log the password or the full URI.
    logger.info("host=%s port=%s schema=%s", conn.host, conn.port, conn.schema)

    # Look up a non-secret variable.
    try:
        my_variable = Variable.get("my_variable")
    except KeyError as e:
        raise AirflowException("Variable my_variable not found") from e
    logger.info("my_variable=%s", my_variable)

Define a workflow that runs the callable as a PythonOperator task in the following YAML workflow definition:


my_dag:
  start_date: "2024-01-01"
  schedule: null
  tasks:
    read_task:
      operator: airflow.providers.standard.operators.python.PythonOperator
      python_callable: my_module.read_connection_and_variable

Package the module with zip, and then use the following AWS CLI commands to upload the code package and the YAML definition to your Amazon S3 bucket. Replace amzn-s3-demo-bucket with your bucket name.


zip my_package.zip my_module.py

aws s3 cp my_package.zip s3://amzn-s3-demo-bucket/code/my_package.zip
aws s3 cp my_dag.yaml s3://amzn-s3-demo-bucket/dags/my_dag.yaml

In the following command, replace my-workflow with your workflow name. Replace dags/my_dag.yaml and code/my_package.zip with the Amazon S3 object keys of your YAML definition and code package. Replace the role ARN with your execution role ARN, and replace us-east-1 with your Region. The following AWS CLI command creates the workflow with the code package, the YAML definition, and the Secrets Manager backend settings:


aws mwaa-serverless create-workflow \
    --name my-workflow \
    --definition-s3-location '{"Bucket": "amzn-s3-demo-bucket", "ObjectKey": "dags/my_dag.yaml"}' \
    --code '{"S3Location": {"Bucket": "amzn-s3-demo-bucket", "ObjectKey": "code/my_package.zip"}}' \
    --environment '{"Variables": {
        "AIRFLOW__WORKERS__SECRETS_BACKEND": "airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend",
        "AIRFLOW__WORKERS__SECRETS_BACKEND_KWARGS": "{\"connections_prefix\": \"airflow/connections\", \"variables_prefix\": \"airflow/variables\", \"region_name\": \"us-east-1\"}"
    }}' \
    --role-arn arn:aws:iam::111122223333:role/MyMWAAServerlessRole \
    --region us-east-1

Use the following AWS CLI command to start a workflow run. Replace the workflow ARN and the Region with your own values.


aws mwaa-serverless start-workflow-run \
    --workflow-arn arn:aws:airflow-serverless:us-east-1:111122223333:workflow/my-workflow-a1b2c3d4e5 \
    --region us-east-1

The task logs the non-secret connection fields and the variable value. To find these log messages, see Viewing task logs.

Limitations for connections and variables

The following limitations apply to connections and variables in Amazon MWAA Serverless:

To configure a secrets backend, set both AIRFLOW__WORKERS__SECRETS_BACKEND and AIRFLOW__WORKERS__SECRETS_BACKEND_KWARGS. Both count toward the limit of 50 environment variables per workflow. For more information, see Limits and reserved keys.
A workflow uses one secrets backend. You can't chain multiple backends.
To read Secrets Manager secrets across accounts, use a customer managed AWS KMS key. You can't use the AWS managed key aws/secretsmanager.
GetWorkflow returns connections and variables that you define as environment variables in plain text. Use them only for non-secret values.

Troubleshooting connections and variables

Errors from connection and variable lookups appear in the task logs. To find the logs, see Viewing task logs. The following table lists common problems, along with their causes and fixes.

Error or symptom	Cause	Fix
`AirflowNotFoundException` for a connection, or `KeyError` for a variable	You haven't configured a secrets backend, or no location in the lookup order has the key. For example, your secret name doesn't match the prefix and key, your secret is in a different Region or account, or your lookup pattern excludes the key.	Check that the secret name is the prefix followed by `/` and the exact key, in the workflow's Region. If you set lookup patterns, check that the key matches them. If you store the value in a secrets backend, set the backend variables as described in Configuring a secrets backend.
`AccessDeniedException` when reading a secret or parameter	The execution role lacks `secretsmanager:GetSecretValue`, or `ssm:GetParameter` and `ssm:GetParameters`, on the resource.	Add the policy for your backend, with `Resource` values that match your prefixes. For more information about the policy, see Execution role permissions.
AWS KMS `AccessDeniedException` or a decrypt error	The secret or `SecureString` parameter uses a customer managed key that the execution role can't use.	Grant `kms:Decrypt` on the key in the role's policy and allow the role in the key policy.
A connection has incorrect or empty fields	The URI is wrapped in quotation marks, or special characters in the user name or password aren't URL-encoded.	Store the URI without quotation marks and URL-encode special characters, or store the connection as JSON.
The secrets backend has no effect	Only one of the two backend variables is set, or the `AIRFLOW__WORKERS__SECRETS_BACKEND_KWARGS` JSON is malformed.	Set both variables, escape the inner quotation marks in the kwargs JSON, and run `update-workflow`.
`ModuleNotFoundError` for a custom backend class	The class path in `AIRFLOW__WORKERS__SECRETS_BACKEND` doesn't match the module's location in the code package, or the workflow has no `--code` package.	Match the class path to the package layout. For more information, see Preparing your code package.
Cross-account lookups fail	The secret's resource policy doesn't allow the execution role, or the secret uses the `aws/secretsmanager` key.	Meet the cross-account requirements. For more information about the requirements, see Option D: Secrets Manager secrets in another account.
An `AIRFLOW_CONN_` or `AIRFLOW_VAR_` value isn't used	A value with the same key exists in the secrets backend, which Apache Airflow checks first.	Define each connection and variable in only one place. For more information, see How Amazon MWAA Serverless looks up connections and variables.
Higher than expected Secrets Manager charges	Every lookup, including lookups for keys that don't exist in Secrets Manager, calls the Secrets Manager API.	Add lookup patterns to the backend keyword arguments so that Apache Airflow queries Secrets Manager only for keys that you store there.

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

Environment variables

Security