Apache Airflow versions on Amazon Managed Workflows for Apache Airflow (MWAA)

This page describes the Apache Airflow versions Amazon Managed Workflows for Apache Airflow (MWAA) supports and the strategies we recommend to upgrade to the latest version.

Contents

• About Amazon MWAA versions
• Latest version
• Apache Airflow versions
• Apache Airflow components
  • Schedulers
  • Workers
  • Features
• Apache Airflow features
• Upgrading Apache Airflow
  • Upgrading from Apache Airflow v2.0.2 to Apache Airflow v2.2.2
  • Upgrading from Apache Airflow v1 to Apache Airflow v2
• Changes between v1 and v2

About Amazon MWAA versions

Amazon MWAA builds container images that bundle Apache Airflow releases with other common binaries and Python libraries. The image uses the Apache Airflow base install for the version you specify. When you create an environment, you specify an image version to use. Once an environment is created, it keeps using the specified image version until you upgrade it to a later version.

Latest version

Amazon MWAA supports more than one Apache Airflow version. If you do not specify an image version when you create an environment, Amazon MWAA creates an environment using the latest supported version of Apache Airflow.

Apache Airflow versions

The following Apache Airflow versions are supported on Amazon Managed Workflows for Apache Airflow (MWAA).

Note

Beginning with Apache Airflow v2.2.2, Amazon MWAA supports installing Python requirements, provider packages, and custom plugins directly on the Apache Airflow web server.

Apache Airflow version	Apache Airflow guide	Apache Airflow constraints	Python version
v2.4.3	Apache Airflow v2.4.3 reference guide	Apache Airflow v2.4.3 constraints file	Python 3.10
v2.2.2	Apache Airflow v2.2.2 reference guide	Apache Airflow v2.2.2 constraints file	Python 3.7
v2.0.2	Apache Airflow v2.0.2 reference guide	Apache Airflow v2.0.2 constraints file	Python 3.7
v1.10.12	Apache Airflow v1.10.12 reference guide	Apache Airflow v1.10.12 constraints file	Python 3.7

Apache Airflow components

This section describes the number of Apache Airflow schedulers and workers available for each Apache Airflow version on Amazon MWAA, and provides a list of key Apache Airflow features, indicating the version that supports each feature.

Schedulers

Apache Airflow version	Scheduler (default)	Scheduler (min)	Scheduler (max)
Apache Airflow v2 and above	2	2	5
Apache Airflow v1	1	1	1

Workers

Airflow version	Workers (min)	Workers (max)	Workers (default)
Apache Airflow v2	1	25	10
Apache Airflow v1	1	25	10

Features

Apache Airflow features

The following table provides an overview of major features, and lists which features are supported in each Apache Airflow version on Amazon MWAA. Some features might not be supported in a minor version release, but later supported in a subsequent Amazon MWAA release.

Apache Airflow feature	Supported	Version	Description	Apache Airflow guide
High Availability	Yes	v2+	Allows you to run multiple Schedulers concurrently.	Running More Than One Scheduler
Full REST API	No	N/A	Allows you and third-parties to acccess CRUD (Create, Read, Update, Delete) operations on all Airflow resources using a Swagger/OpenAPI specification.	Airflow's REST API
Smart Sensors	Yes	v2+	Allows you to schedule a single and long running task, checks the status of a batch of Sensor tasks, and stores in your metadata database.	Smart Sensors
Deferrable operators	No	N/A	A deferrable operator suspents itself and frees up the worker as it waits for a trigger to resume working.	Deferrable Operators and Triggers
TaskFlow API	Yes	v2+	Allows you to organize tasks into hierarchical groups and pass/share data between tasks.	TaskFlow API
Task Groups	Yes	v2+	Allows you to view task groups in the Apache Airflow UI (a UI grouping concept which fulfills the primary purpose of SubDAGs).	TaskGroup
Independent Providers	Yes	v2+	Allows you to use Apache Airflow packages that have been separated from and independently versioned from the core Apache Airflow distribution. (Bash and Python Operators remain in the core distribution.)	Provider Packages
Simplified Kubernetes Executor	No	N/A	Allows you to use a re-architected Kubernetes Executor and KubernetesPodExecutor (which allow users to dynamically launch tasks as individual Kubernetes Pods).	Kubernetes Executor
UI/UX Improvements	Yes	v2+	Allows you to use a more intuitive front-end experience to the Apache Airflow UI.	Airflow UI
Apache Airflow connections on Amazon MWAA	Yes	v2+	Allows you to use the Apache Airflow connections add and connections delete CLI commands. To learn more, see Apache Airflow CLI command reference.	Command Line Interface
DAG decorators	Yes	v2+	Allows you to decorate a function with @dag or @task to to turn it into a DAG generator function or into a task instance using a PythonOperator.	The DAG decorator

Upgrading Apache Airflow

The following section contains links to tutorials in the Apache Airflow reference guide and the steps we recommend to upgrade your environment.

Important

Currently, Amazon MWAA does not support in-place upgrades of existing environments for any Apache Airflow versions. To upgrade your Amazon MWAA environment to the latest supported release, you have to create a new environment and migrate your resources to the new environment.

Topics

• Upgrading from Apache Airflow v2.0.2 to Apache Airflow v2.2.2
• Upgrading from Apache Airflow v1 to Apache Airflow v2

Upgrading from Apache Airflow v2.0.2 to Apache Airflow v2.2.2

The following section shows how you can upgrade your Apache Airflow v2.0.2 environment to Apache Airflow v2.2.2.

1. Create a new Apache Airflow v2.2.2 environment.

2. Copy your DAGs, custom plugins, and requirements.txt resources from your existing v2.0.2 Amazon S3 bucket to the new environment's Amazon S3 bucket.

   1. If you use requirements.txt in your environment, you'll need to update the --constraint to v2.2.2 constraints and verify that current libraries and packages are compatible with Apache Airflow v2.2.2

3. Starting with Apache Airflow v2.2.2, the list of provider packages Amazon MWAA installs by default for your environment has changed, and you can now install dependencies and plugin on the Apache Airflow web server. Compare the list of provider packages installed by default in Apache Airflow v2.2.2 and Apache Airflow v2.0.2, and configure any additional packages you might need for your new v2.2.2 environment.

4. Test your DAGs using the new Apache Airflow v2.2.2 environment.

5. Once you have confirmed that your tasks complete successfully, delete the v2.0.2 environment.

Upgrading from Apache Airflow v1 to Apache Airflow v2

The following section shows how you can upgrade your Apache Airflow v1 environment to Apache Airflow v2. For more information about migrating your self-managed Apache Airflow deployments, or migrating an existing Amazon MWAA environment, including instructions for backing up your metadata database, see the Amazon MWAA Migration Guide.

The steps in this section assume you are starting with an existing codebase in Apache Airflow v1.10.12 and are upgrading it to any of the supported Apache Airflow v2 minor version releases.To upgrade to Apache Airflow v2, create a new environment using the latest supported version with the required Apache Airflow configuration options from your existing environment, and follow the steps in this section to modify your DAGs, Python dependencies in requirements.txt, and custom plugins in plugins.zip for the new version. After you have successfully migrated your workflows,you can delete the older environment.

1. Follow the steps to upgrade to the Apache Airflow v1 bridge release, run upgrade check scripts, and convert and test custom plugins and DAGs to Apache Airflow v2 locally, in Upgrading from 1.10 to 2 in the Apache Airflow reference guide and Updating Airflow on GitHub.

   1. Upgrade import statements. Get started by reviewing the Apache Airflow v2 sample code in Using a secret key in AWS Secrets Manager for an Apache Airflow connection— the import statements and hooks from Apache Airflow v1 to Apache Airflow v2 have changed.

      1. Example Apache Airflow v1

         from airflow.operators.python_operator import PythonOperator
         from airflow.contrib.hooks.aws_hook import AwsHook
             ...
         hook = AwsHook()

      2. Example Apache Airflow v2

         from airflow.operators.python import PythonOperator
         from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
             ...
         hook = AwsBaseHook(client_type='secretsmanager')

   2. Upgrade Apache Airflow CLI scripts. Get started by reviewing the Apache Airflow v2 CLI commands in Using a bash script— Related commands are now grouped together as subcommands. For example, trigger_dag in Apache Airflow v1 is now dags trigger Apache Airflow v2.

      1. Example Apache Airflow v1

         # brew install jq
         aws mwaa create-cli-token --name YOUR_ENVIRONMENT_NAME | export CLI_TOKEN=$(jq -r .CliToken) && curl --request POST "https://YOUR_HOST_NAME/aws_mwaa/cli" \
             --header "Authorization: Bearer $CLI_TOKEN" \
             --header "Content-Type: text/plain" \
             --data-raw "trigger_dag YOUR_DAG_NAME"
         

      2. Example Apache Airflow v2

         # brew install jq
         aws mwaa create-cli-token --name YOUR_ENVIRONMENT_NAME | export CLI_TOKEN=$(jq -r .CliToken) && curl --request POST "https://YOUR_HOST_NAME/aws_mwaa/cli" \
             --header "Authorization: Bearer $CLI_TOKEN" \
             --header "Content-Type: text/plain" \
             --data-raw "dags trigger YOUR_DAG_NAME"
         

2. Create an Amazon Managed Workflows for Apache Airflow (MWAA) environment using the latest supported version and iteratively add DAGs, custom plugins in plugins.zip, and Python dependencies in requirements.txt to your new environment as you finish testing locally.

   1. Get started with the Apache Airflow v2 sample code in Aurora PostgreSQL database cleanup on an Amazon MWAA environment— The import statements in Apache Airflow v1 to Apache Airflow v2 and the Python dependencies in the requirements.txt have changed.

      1. Example Apache Airflow v1 dependencies

         apache-airflow[postgres]==1.10.12
         apache-airflow[mysql]==1.10.12

      2. Example Apache Airflow v2 dependencies

         apache-airflow[postgres]==2.0.2
         apache-airflow[mysql]==2.0.2

      3. Example Apache Airflow v1 imports

         from airflow.operators.python_operator import PythonOperator
         from airflow.jobs import BaseJob

      4. Example Apache Airflow v2 imports

         from airflow.operators.python import PythonOperator
         from airflow.jobs.base_job import BaseJob

3. After you've migrated and tested all DAGs, custom plugins in plugins.zip, and Python dependencies in requirements.txt to your new new environment, you can delete your old environment.

Changes between v1 and v2

The following is a summary of what's changed between Apache Airflow v1 and Apache Airflow v2 on Amazon MWAA.

• New: High availability by default. By default, an Amazon MWAA environment in Apache Airflow v2 uses 2 Airflow Schedulers, and accepts a value up to 5 Schedulers. To learn more about running more than one scheduler concurrently, see Airflow Scheduler in the Apache Airflow reference guide.

• New: Airflow package extras. The Python dependencies that you specify in a requirements.txt on Amazon MWAA have changed in Apache Airflow v2. For example, the core extras, provider extras, locally installed software extras, external service extras, "other" extras, bundle extras, doc extras, and software extras have changed. To view a list of the packages installed for Apache Airflow v2 on Amazon MWAA, see Amazon MWAA local runner requirements.txt on the GitHub website.

• New: Operators, Hooks, and Executors. The import statements in your DAGs, and the custom plugins you specify in a plugins.zip on Amazon MWAA have changed between Apache Airflow v1 and Apache Airflow v2. For example, from airflow.contrib.hooks.aws_hook import AwsHook in Apache Airflow v1 has changed to from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook in Apache Airflow v2. To learn more, see Python API Reference in the Apache Airflow reference guide.

• New: Imports in plugins. Importing operators, sensors, hooks added in plugins using airflow.{operators,sensors,hooks}.<plugin_name> is no longer supported. These extensions should be imported as regular Python modules. In v2 and above, the recommended approach is to place them in the DAGs directory and create and use an .airflowignore file to exclude them from being parsed as DAGs. To learn more, see Modules Management and Creating a custom Operator in the Apache Airflow reference guide.

• New: Airflow CLI command structure. The Apache Airflow v2 CLI is organized so that related commands are grouped together as subcommands, which means you need to update Apache Airflow v1 scripts if you want to upgrade to Apache Airflow v2. For example, unpause in Apache Airflow v1 is now dags unpause in Apache Airflow v2. To learn more, see Airflow CLI changes in 2 in the Apache Airflow reference guide.

• Changed: Airflow connection types. By default, the Airflow UI contains a subset of the connection types that were available in Apache Airflow v1. To view a list of the connection types available for Apache Airflow v2 on Amazon MWAA by default, see Apache Airflow v2 provider packages installed on Amazon MWAA environments.

• Existing: Airflow configuration options. The Apache Airflow v2 configuration options are the same as Apache Airflow v1. Although, in version 2 and above, Apache Airflow requires explicit specifications of configuration values in some cases, rather than defaulting to a generic value. To learn more, see Airflow configuration options in the Apache Airflow reference guide.