Apache Airflow versions on Amazon Managed Workflows for Apache Airflow (MWAA)

This page describes the Apache Airflow versions Amazon Managed Workflows for Apache Airflow (MWAA) supports and the strategies we recommend to upgrade to the latest version.

Contents

• About Amazon MWAA versions
• Latest version
• Apache Airflow versions
• Apache Airflow components
  • Schedulers
  • Workers
• Apache Airflow v2.0.2
  • What's supported for Apache Airflow v2.0.2
  • What's changed between v1.10.12 and v2.0.2
  • Installing Apache Airflow v2.0.2
  • Upgrading to Apache Airflow v2.0.2
• Apache Airflow v1.10.12
  • Installing Apache Airflow v1.10.12

About Amazon MWAA versions

Amazon MWAA builds container images that bundle Apache Airflow releases with other common binaries and Python libraries. The image uses the Airflow base install for the version you specify. When you create an environment, you specify an image version to use. Once an environment is created, it keeps using the specified image version until you upgrade it to a later version.

Latest version

Amazon MWAA supports more than one Airflow version. If you do not specify an image version when you create an environment, the latest version is used.

Apache Airflow versions

The following Apache Airflow versions are supported on Amazon Managed Workflows for Apache Airflow (MWAA).

Airflow version	Airflow guide	Airflow constraints	Python version
v2.0.2	Apache Airflow v2.0.2 reference guide	https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt	Python 3.7
v1.10.12	Apache Airflow v1.10.12 reference guide	https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt	Python 3.7

Apache Airflow components

This section describes the number of Apache Airflow Scheduler(s) and Workers available for each Apache Airflow version on Amazon MWAA.

Schedulers

Airflow version	Scheduler (default)	Scheduler (min)	Scheduler (max)
Apache Airflow v2.0.2	2	2	5
Apache Airflow v1.10.12	1	1	1

Workers

Airflow version	Workers (min)	Workers (max)	Workers (default)
Apache Airflow v2.0.2	1	25	10
Apache Airflow v1.10.12	1	25	10

Apache Airflow v2.0.2

This section provides an overview of what's supported on Amazon MWAA, what's changed between Apache Airflow v1.10.12 and Apache Airflow v2.0.2, and links to install or upgrade to Apache Airflow v2.0.2 in the Apache Airflow reference guide.

What's supported for Apache Airflow v2.0.2

Airflow feature	Supported	Description	Apache Airflow guide
High Availability	Yes	Allows you to run multiple Schedulers concurrently.	Running More Than One Scheduler
Full REST API	No	Allows you and third-parties to acccess CRUD (Create, Read, Update, Delete) operations on all Airflow resources using a Swagger/OpenAPI specification.	Airflow's REST API
Smart Sensors	Yes	Allows you to schedule a single and long running task, checks the status of a batch of Sensor tasks, and stores in your metadata database.	Smart Sensors
TaskFlow API	Yes	Allows you to organize tasks into hierarchical groups and pass/share data between tasks.	TaskFlow API
Task Groups	Yes	Allows you to view task groups in the Apache Airflow UI (a UI grouping concept which fulfills the primary purpose of SubDAGs).	TaskGroup
Independent Providers	Yes	Allows you to use Airflow packages that have been separated from and independently versioned from the core Apache Airflow distribution. (Bash and Python Operators remain in the core distribution.)	Provider Packages
Simplified Kubernetes Executor	No	Allows you to use a re-architected Kubernetes Executor and KubernetesPodExecutor (which allow users to dynamically launch tasks as individual Kubernetes Pods).	Kubernetes Executor
UI/UX Improvements	Yes	Allows you to use a more intuitive front-end experience to the Apache Airflow UI.	Airflow UI
Airflow connections on Amazon MWAA	Yes	Allows you to use the Apache Airflow connections add and connections delete CLI commands. To learn more, see Apache Airflow CLI command reference.	Command Line Interface

What's changed between v1.10.12 and v2.0.2

The following section provides a summary of what's changed between Apache Airflow v1.10.12 and Apache Airflow v2.0.2 on Amazon MWAA.

• New: High availability by default. By default, an Amazon MWAA environment in Apache Airflow v2.0.2 uses 2 Airflow Schedulers, and accepts a value up to 5 Schedulers. To learn more about running more than one scheduler concurrently, see Airflow Scheduler in the Apache Airflow reference guide.

• New: Airflow package extras. The Python dependencies that you specify in a requirements.txt on Amazon MWAA have changed. A large number of Apache Airflow core extras, provider extras, locally installed software extras, external service extras, "other" extras, bundle extras, doc extras, and software extras are available for Apache Airflow v2.0.2 at https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt.

• New: Operators, Hooks, and Executors. The import statements in your Apache Airflow v1.10.12 DAGs, and the custom plugins you specify in a plugins.zip on Amazon MWAA have changed. For example, from airflow.contrib.hooks.aws_hook import AwsHook in Apache Airflow v1.10.12 has changed to from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook in Apache Airflow v2.0.2. To learn more, see Python API Reference in the Apache Airflow reference guide.

• New: Airflow CLI command structure. The Apache Airflow v2.0.2 CLI is organized so that related commands are grouped together as subcommands, which means you need to update Apache Airflow v1.10.12 scripts if you want to upgrade to Apache Airflow v2.0.2. For example, unpause in Apache Airflow v1.10.12 is now dags unpause in Apache Airflow v2.0.2. To learn more, see Airflow CLI changes in 2.0 in the Apache Airflow reference guide.

• Changed: Airflow connection types. By default, the Airflow UI contains a subset of the connection types that were available in Apache Airflow v1.10.12. If a connection template is not available in the Apache Airflow UI, an alternate connection template can be used. To learn more, see Overview of connection types.

• Existing: Airflow configuration options. The Apache Airflow v2.0.2 configuration options are the same as Apache Airflow v1.10.12. Although, Airflow 2.0+ requires explicit specifications of configuration values in some cases, rather than defaulting to a generic value. To learn more, see Airflow configuration options in 2.0 in the Apache Airflow reference guide.

Installing Apache Airflow v2.0.2

The following section contains links to tutorials in the Apache Airflow reference guide to install and run Apache Airflow v2.0.2. The steps assume you are starting from scratch and have the Docker Engine and Docker Compose installed locally.

To install Apache Airflow v2.0.2 in Docker, see Running Airflow in Docker in the Apache Airflow reference guide.

Upgrading to Apache Airflow v2.0.2

The following section contains links to tutorials in the Apache Airflow reference guide and the steps we recommend to upgrade to Apache Airflow v2.0.2. We provide before and after examples in this guide to demonstrate what's changed. The steps assume you are starting with an existing codebase in Apache Airflow v1.10.12 and are upgrading it to Apache Airflow v2.0.2.

Amazon MWAA doesn't support upgrading an existing environment from Apache Airflow v1.10.12 to v2.0.2. To upgrade to Apache Airflow v2.0.2, create a new environment in Apache Airflow v2.0.2 with the required Apache Airflow configuration options from your v1.10.12 environment, and follow the steps in this section to modify your DAGs, Python dependencies in requirements.txt, and custom plugins in plugins.zip for the new version. After you have successfully migrated your workflows, you can delete the older environment.

1. Follow the steps to upgrade to the Apache Airflow v1.10.15 "bridge" release, run upgrade check scripts, and convert and test custom plugins and DAGs to Apache Airflow v2.0.2 locally, in Upgrading to Airflow 2.0+ in the Apache Airflow reference guide and Updating Airflow on GitHub.

   1. Upgrade import statements. Get started by reviewing the Apache Airflow v2.0.2 sample code in Using a secret key in AWS Secrets Manager for an Apache Airflow connection—the import statements and hooks from Apache Airflow v1.10.12 to Apache Airflow v2.0.2 have changed.

      1. Example Apache Airflow v1.10.12

         from airflow.operators.python_operator import PythonOperator
         from airflow.contrib.hooks.aws_hook import AwsHook
             ...
         hook = AwsHook()

      2. Example Apache Airflow v2.0.2

         from airflow.operators.python import PythonOperator
         from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
             ...
         hook = AwsBaseHook(client_type='secretsmanager')

   2. Upgrade Airflow CLI scripts. Get started by reviewing the Apache Airflow v2.0.2 CLI commands in Using a bash script—related commands are now grouped together as subcommands. For example, trigger_dag in Apache Airflow v1.10.12 is now dags trigger Apache Airflow v2.0.2.

      1. Example Apache Airflow v1.10.12

         # brew install jq
         aws mwaa create-cli-token --name YOUR_ENVIRONMENT_NAME | export CLI_TOKEN=$(jq -r .CliToken) && curl --request POST "https://YOUR_HOST_NAME/aws_mwaa/cli" \
             --header "Authorization: Bearer $CLI_TOKEN" \
             --header "Content-Type: text/plain" \
             --data-raw "trigger_dag YOUR_DAG_NAME"
         

      2. Example Apache Airflow v2.0.2

         # brew install jq
         aws mwaa create-cli-token --name YOUR_ENVIRONMENT_NAME | export CLI_TOKEN=$(jq -r .CliToken) && curl --request POST "https://YOUR_HOST_NAME/aws_mwaa/cli" \
             --header "Authorization: Bearer $CLI_TOKEN" \
             --header "Content-Type: text/plain" \
             --data-raw "dags trigger YOUR_DAG_NAME"
         

2. Create an Amazon Managed Workflows for Apache Airflow (MWAA) environment in Apache Airflow v2.0.2 and iteratively add DAGs, custom plugins in plugins.zip, and Python dependencies in requirements.txt to your new environment as you finish testing locally.

   1. Get started with the Apache Airflow v2.0.2 sample code in Aurora PostgreSQL database cleanup on an Amazon MWAA environment—note the import statements in Apache Airflow v1.10.12 to Apache Airflow v2.0.2 and the Python dependencies in the requirements.txt have changed.

      1. Example Apache Airflow v1.10.12 dependencies

         apache-airflow[postgres]==1.10.12
         apache-airflow[mysql]==1.10.12

      2. Example Apache Airflow v2.0.2 dependencies

         apache-airflow[postgres]==2.0.2
         apache-airflow[mysql]==2.0.2

      3. Example Apache Airflow v1.10.12 imports

         from airflow.operators.python_operator import PythonOperator
         from airflow.jobs import BaseJob

      4. Example Apache Airflow v2.0.2 imports

         from airflow.operators.python import PythonOperator
         from airflow.jobs.base_job import BaseJob

3. After you've migrated and tested all DAGs, custom plugins in plugins.zip, and Python dependencies in requirements.txt to your new Apache Airflow v2.0.2 environment, delete your Apache Airflow v1.10.12 environment.

Apache Airflow v1.10.12

This section provides an overview of how to get started with Apache Airflow v1.10.12, and links to install Apache Airflow v1.10.12 in the Apache Airflow reference guide.

Installing Apache Airflow v1.10.12

The following section contains links to tutorials in the Apache Airflow reference guide to install and run Apache Airflow v1.10.12. To install Apache Airflow v1.10.12, see Installation for Apache Airflow v1.10.12 in the Apache Airflow reference guide.