Apache Airflow versions on Amazon Managed Workflows for Apache Airflow (MWAA)
This page describes the Apache Airflow versions Amazon Managed Workflows for Apache Airflow (MWAA) supports and the strategies we recommend to upgrade to the latest version.
Contents
About Amazon MWAA versions
Amazon MWAA builds container images that bundle Apache Airflow releases with other common binaries and Python libraries. The image uses the Apache Airflow base install for the version you specify. When you create an environment, you specify an image version to use. Once an environment is created, it keeps using the specified image version until you upgrade it to a later version.
Latest version
Amazon MWAA supports more than one Apache Airflow version. If you do not specify an image version when you create an environment, Amazon MWAA creates an environment using the latest supported version of Apache Airflow.
Apache Airflow versions
The following Apache Airflow versions are supported on Amazon Managed Workflows for Apache Airflow (MWAA).
Beginning with Apache Airflow v2.2.2, Amazon MWAA supports installing Python requirements, provider packages, and custom plugins directly on the Apache Airflow web server.
Apache Airflow version | Apache Airflow guide | Apache Airflow constraints | Python version |
---|---|---|---|
v2.4.3 |
|||
v2.2.2 |
|||
v2.0.2 |
|||
v1.10.12 |
Apache Airflow components
This section describes the number of Apache Airflow schedulers and workers available for each Apache Airflow version on Amazon MWAA, and provides a list of key Apache Airflow features, indicating the version that supports each feature.
Schedulers
Apache Airflow version | Scheduler (default) | Scheduler (min) | Scheduler (max) |
---|---|---|---|
Apache Airflow v2 and above |
2 |
2 |
5 |
Apache Airflow v1 |
1 |
1 |
1 |
Workers
Airflow version | Workers (min) | Workers (max) | Workers (default) |
---|---|---|---|
Apache Airflow v2 |
1 |
25 |
10 |
Apache Airflow v1 |
1 |
25 |
10 |
Features
Apache Airflow features
The following table provides an overview of major features, and lists which features are supported in each Apache Airflow version on Amazon MWAA. Some features might not be supported in a minor version release, but later supported in a subsequent Amazon MWAA release.
Apache Airflow feature | Supported | Version | Description | Apache Airflow guide |
---|---|---|---|---|
High Availability |
Yes |
v2+ |
Allows you to run multiple Schedulers concurrently. |
|
Full REST API |
No |
N/A |
Allows you and third-parties to acccess CRUD (Create, Read, Update, Delete) operations on all Airflow resources using a Swagger/OpenAPI specification. |
|
Smart Sensors |
Yes |
v2+ |
Allows you to schedule a single and long running task, checks the status of a batch of Sensor tasks, and stores in your metadata database. |
|
Deferrable operators |
No |
N/A |
A deferrable operator suspents itself and frees up the worker as it waits for a trigger to resume working. |
|
TaskFlow API |
Yes |
v2+ |
Allows you to organize tasks into hierarchical groups and pass/share data between tasks. |
|
Task Groups |
Yes |
v2+ |
Allows you to view task groups in the Apache Airflow UI (a UI grouping concept which fulfills the primary purpose of SubDAGs). |
|
Independent Providers |
Yes |
v2+ |
Allows you to use Apache Airflow packages that have been separated from and independently versioned from the core Apache Airflow distribution. (Bash and Python Operators remain in the core distribution.) |
|
Simplified Kubernetes Executor |
No |
N/A |
Allows you to use a re-architected Kubernetes Executor and |
|
UI/UX Improvements |
Yes |
v2+ |
Allows you to use a more intuitive front-end experience to the Apache Airflow UI. |
|
Apache Airflow connections on Amazon MWAA |
Yes |
v2+ |
Allows you to use the Apache Airflow |
|
DAG decorators |
Yes |
v2+ |
Allows you to decorate a function with |
Upgrading Apache Airflow
The following section contains links to tutorials in the Apache Airflow reference guide and the steps we recommend to upgrade your environment.
Currently, Amazon MWAA does not support in-place upgrades of existing environments for any Apache Airflow versions. To upgrade your Amazon MWAA environment to the latest supported release, you have to create a new environment and migrate your resources to the new environment.
Topics
Upgrading from Apache Airflow v2.0.2 to Apache Airflow v2.2.2
The following section shows how you can upgrade your Apache Airflow v2.0.2 environment to Apache Airflow v2.2.2.
-
Create a new Apache Airflow v2.2.2 environment.
-
Copy your DAGs, custom plugins, and
requirements.txt
resources from your existing v2.0.2 Amazon S3 bucket to the new environment's Amazon S3 bucket.-
If you use
requirements.txt
in your environment, you'll need to update the--constraint
to v2.2.2 constraintsand verify that current libraries and packages are compatible with Apache Airflow v2.2.2
-
-
Starting with Apache Airflow v2.2.2, the list of provider packages Amazon MWAA installs by default for your environment has changed, and you can now install dependencies and plugin on the Apache Airflow web server. Compare the list of provider packages installed by default in Apache Airflow v2.2.2 and Apache Airflow v2.0.2, and configure any additional packages you might need for your new v2.2.2 environment.
-
Test your DAGs using the new Apache Airflow v2.2.2 environment.
-
Once you have confirmed that your tasks complete successfully, delete the v2.0.2 environment.
Upgrading from Apache Airflow v1 to Apache Airflow v2
The following section shows how you can upgrade your Apache Airflow v1 environment to Apache Airflow v2. For more information about migrating your self-managed Apache Airflow deployments, or migrating an existing Amazon MWAA environment, including instructions for backing up your metadata database, see the Amazon MWAA Migration Guide.
The steps in this section assume you are starting with an existing codebase in Apache Airflow v1.10.12 and are upgrading it to any of the supported Apache Airflow v2 minor version releases.To upgrade to Apache Airflow v2,
create a new environment using the latest supported version with the required Apache Airflow configuration options from your existing environment,
and follow the steps in this section to modify your DAGs, Python dependencies in requirements.txt
, and custom plugins in plugins.zip
for the new version. After you have successfully migrated your workflows,you can delete the older environment.
-
Follow the steps to upgrade to the Apache Airflow v1 bridge release, run upgrade check scripts, and convert and test custom plugins and DAGs to Apache Airflow v2 locally, in Upgrading from 1.10 to 2
in the Apache Airflow reference guide and Updating Airflow on GitHub. -
Upgrade import statements. Get started by reviewing the Apache Airflow v2 sample code in Using a secret key in AWS Secrets Manager for an Apache Airflow connection— the import statements and hooks
from Apache Airflow v1 to Apache Airflow v2 have changed. -
Example Apache Airflow v1
from airflow.operators.
python_operator
import PythonOperator from airflow.contrib.hooks.aws_hook import AwsHook
... hook =AwsHook()
-
Example Apache Airflow v2
from airflow.operators.
python
import PythonOperator from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
... hook =AwsBaseHook(client_type='secretsmanager')
-
-
Upgrade Apache Airflow CLI scripts. Get started by reviewing the Apache Airflow v2 CLI commands in Using a bash script— Related commands are now grouped together as subcommands. For example,
trigger_dag
in Apache Airflow v1 is nowdags trigger
Apache Airflow v2.-
Example Apache Airflow v1
# brew install jq aws mwaa create-cli-token --name YOUR_ENVIRONMENT_NAME | export CLI_TOKEN=$(jq -r .CliToken) && curl --request POST "https://YOUR_HOST_NAME/aws_mwaa/cli" \ --header "Authorization: Bearer $CLI_TOKEN" \ --header "Content-Type: text/plain" \ --data-raw "
trigger_dag
YOUR_DAG_NAME" -
Example Apache Airflow v2
# brew install jq aws mwaa create-cli-token --name YOUR_ENVIRONMENT_NAME | export CLI_TOKEN=$(jq -r .CliToken) && curl --request POST "https://YOUR_HOST_NAME/aws_mwaa/cli" \ --header "Authorization: Bearer $CLI_TOKEN" \ --header "Content-Type: text/plain" \ --data-raw "
dags trigger
YOUR_DAG_NAME"
-
-
-
Create an Amazon Managed Workflows for Apache Airflow (MWAA) environment using the latest supported version and iteratively add DAGs, custom plugins in
plugins.zip
, and Python dependencies inrequirements.txt
to your new environment as you finish testing locally.-
Get started with the Apache Airflow v2 sample code in Aurora PostgreSQL database cleanup on an Amazon MWAA environment— The import statements in Apache Airflow v1 to Apache Airflow v2 and the Python dependencies in the
requirements.txt
have changed.-
Example Apache Airflow v1 dependencies
apache-airflow[postgres]==
1.10.12
apache-airflow[mysql]==1.10.12
-
Example Apache Airflow v2 dependencies
apache-airflow[postgres]==
2.0.2
apache-airflow[mysql]==2.0.2
-
Example Apache Airflow v1 imports
from airflow.operators.
python_operator
import PythonOperator from airflow.jobsimport BaseJob
-
Example Apache Airflow v2 imports
from airflow.operators.
python
import PythonOperator from airflow.jobs.base_job import BaseJob
-
-
-
After you've migrated and tested all DAGs, custom plugins in
plugins.zip
, and Python dependencies inrequirements.txt
to your new new environment, you can delete your old environment.
Changes between v1 and v2
The following is a summary of what's changed between Apache Airflow v1 and Apache Airflow v2 on Amazon MWAA.
-
New: High availability by default. By default, an Amazon MWAA environment in Apache Airflow v2 uses 2 Airflow Schedulers, and accepts a value up to 5 Schedulers. To learn more about running more than one scheduler concurrently, see Airflow Scheduler
in the Apache Airflow reference guide. -
New: Airflow package extras. The Python dependencies that you specify in a
requirements.txt
on Amazon MWAA have changed in Apache Airflow v2. For example, the core extras, provider extras , locally installed software extras , external service extras , "other" extras , bundle extras , doc extras , and software extras have changed. To view a list of the packages installed for Apache Airflow v2 on Amazon MWAA, see Amazon MWAA local runner requirements.txt
on the GitHub website. -
New: Operators, Hooks, and Executors. The import statements in your DAGs, and the custom plugins you specify in a
plugins.zip
on Amazon MWAA have changed between Apache Airflow v1 and Apache Airflow v2. For example,from airflow.contrib.hooks.aws_hook import AwsHook
in Apache Airflow v1 has changed tofrom airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
in Apache Airflow v2. To learn more, see Python API Referencein the Apache Airflow reference guide. -
New: Imports in plugins. Importing operators, sensors, hooks added in plugins using
airflow.{operators,sensors,hooks}.<plugin_name>
is no longer supported. These extensions should be imported as regular Python modules. In v2 and above, the recommended approach is to place them in the DAGs directory and create and use an .airflowignore file to exclude them from being parsed as DAGs. To learn more, see Modules Managementand Creating a custom Operator in the Apache Airflow reference guide. -
New: Airflow CLI command structure. The Apache Airflow v2 CLI is organized so that related commands are grouped together as subcommands, which means you need to update Apache Airflow v1 scripts if you want to upgrade to Apache Airflow v2. For example,
unpause
in Apache Airflow v1 is nowdags unpause
in Apache Airflow v2. To learn more, see Airflow CLI changes in 2in the Apache Airflow reference guide. -
Changed: Airflow connection types. By default, the Airflow UI contains a subset of the connection types that were available in Apache Airflow v1. To view a list of the connection types available for Apache Airflow v2 on Amazon MWAA by default, see Apache Airflow v2 provider packages installed on Amazon MWAA environments.
-
Existing: Airflow configuration options. The Apache Airflow v2 configuration options are the same as Apache Airflow v1. Although, in version 2 and above, Apache Airflow requires explicit specifications of configuration values in some cases, rather than defaulting to a generic value. To learn more, see Airflow configuration options
in the Apache Airflow reference guide.