Amazon MWAA environment class - Amazon Managed Workflows for Apache Airflow

Amazon MWAA environment class

The environment class you choose for your Amazon MWAA environment determines the size of the AWS-managed AWS Fargate containers where the Celery Executor runs, and the AWS-managed Amazon Aurora PostgreSQL metadata database where the Apache Airflow scheduler(s) creates task instances. This page describes each Amazon MWAA environment class, and steps to update the environment class on the Amazon MWAA console.

Environment class

The following image shows where you can update the Environment class on the Amazon MWAA console.

Airflow v2.0.2

                This image shows the Environment class on the Amazon MWAA console.
Airflow v1.10.12

                This image shows the Environment class on the Amazon MWAA console.

Environment capabilities

The following section contains the default concurrent Apache Airflow tasks, Random Access Memory (RAM), and the virtual centralized processing units (vCPUs) for each environment class. The concurrent tasks listed assume that task concurrency does not exceed the Apache Airflow Worker capacity in the environment.

mw1.small
  • 5 concurrent tasks (by default)

  • 1 vCPUs

  • 2 GB RAM

mw1.medium
  • 10 concurrent tasks (by default)

  • 2 vCPUs

  • 4 GB RAM

mw1.large
  • 20 concurrent tasks (by default)

  • 4 vCPUs

  • 8 GB RAM

You can use celery.worker_autoscale to increase tasks per worker. To learn more, see the Example high performance use case.

Airflow Schedulers

The following section contains the Apache Airflow Scheduler options available on the Amazon MWAA console.

Airflow v2.0.2
  • v2.0.2 - Accepts between 2 to 5. Defaults to 2.

Airflow v1.10.12
  • v1.10.12 - Accepts 1.

Updating the environment class on the Amazon MWAA console

You can update your environment class on the Amazon MWAA console at any time.

To configure the environment size

  1. Open the Environments page on the Amazon MWAA console.

  2. Choose an environment.

  3. Choose Edit.

  4. Choose Next.

  5. On the Environment class pane, choose an option.

  6. Choose Save.

Note

It can take up to 30 minutes for changes to take effect.

Updating Airflow Schedulers on the Amazon MWAA console

If you're using Apache Airflow v2.0.2, you can update the number of Apache Airflow Schedulers on the Amazon MWAA console at any time.

To configure the environment size

  1. Open the Environments page on the Amazon MWAA console.

  2. Choose an environment.

  3. Choose Edit.

  4. Choose Next.

  5. On the Environment class pane, enter a value up to 5 Schedulers.

  6. Choose Save.

Note

It can take up to 30 minutes for changes to take effect.

Troubleshooting environment updates

  • If you update your environment to a different environment class (such as changing an mw1.medium to an mw1.small), and the request to update your environment failed, the environment status goes into an UPDATE_FAILED state and the environment is rolled back to, and is billed according to, the previous stable version of an environment. To learn more, see I tried changing the environment class but the update failed.