Installing custom plugins - Amazon Managed Workflows for Apache Airflow

Installing custom plugins

Amazon Managed Workflows for Apache Airflow (MWAA) supports Apache Airflow's built-in plugin manager, allowing you to use custom Apache Airflow operators, hooks, sensors, or interfaces. This page describes the steps to install Apache Airflow custom plugins on your Amazon MWAA environment using a plugins.zip file.

Prerequisites

To use the steps on this page, you'll need:

  1. The required AWS resources configured for your environment as defined in Get started with Amazon Managed Workflows for Apache Airflow (MWAA).

  2. An execution role with a permissions policy that grants Amazon MWAA access to the AWS resources used by your environment as defined in Amazon MWAA Execution role.

  3. An AWS account with access in AWS Identity and Access Management (IAM) to the Amazon S3 console, or the AWS Command Line Interface (AWS CLI) as defined in Accessing an Amazon MWAA environment.

How it works

To run custom plugins on your environment, you must do three things:

  1. Create a plugins.zip file locally.

  2. Upload the local plugins.zip file to your Amazon S3 storage bucket.

  3. Specify the version of this file in the Plugins file field on the Amazon MWAA console.

Note

If this is the first time you're uploading a plugins.zip to your Amazon S3 bucket, you'll also need to specify the path to the file on the Amazon MWAA console. You only need to complete this step once.

Apache Airflow's built-in plugin manager can integrate external features to its core by simply dropping files in an $AIRFLOW_HOME/plugins folder. It allows you to use custom Apache Airflow operators, hooks, sensors, or interfaces.

Amazon MWAA automatically detects and syncs changes from your Amazon S3 bucket to Apache Airflow every 30 seconds. To run an Apache Airflow platform that uses custom plugins on an Amazon MWAA environment, you need to create and copy a plugins.zip file to your Amazon S3 storage bucket. For example, the contents of the plugins.zip file in your storage bucket may look like this:

Example plugins.zip

__init__.py my_airflow_plugin.py hooks/ |-- __init__.py |-- my_airflow_hook.py operators/ |-- __init__.py |-- my_airflow_operator.py sensors/ |-- __init__.py |-- my_airflow_sensor.py

This assumes that your DAG definition in dag_def.py of your DAG folder uses an import statement for each file. For example:

Example dag_def.py

from airflow.operators.my_airflow_operator import MyOperator from airflow.sensors.my_airflow_sensor import MySensor from airflow.hooks.my_airflow_hook import MyHook ...

It also assumes that my_airflow_plugin.py uses the AirflowPlugin class:

Example my_airflow_plugin.py

from airflow.plugins_manager import AirflowPlugin from hooks.my_airflow_hook import * from operators.my_airflow_operator import * from utils.my_utils import * class PluginName(AirflowPlugin): name = 'my_airflow_plugin' hooks = [MyHook] operators = [MyOperator] sensors = [MySensor] ...

To refer to a given plugin from your DAG, import into your Python file. For example, the Python code for these files may look like this:

Example my_airflow_hook.py

from hooks.my_airflow_hook import BaseHook class MyHook(BaseHook): ...

Example my_airflow_sensor.py

from sensors.my_airflow_sensor import BaseSensorOperator from airflow.utils.decorators import apply_defaults class MySensor(BaseSensorOperator): ...

Example my_airflow_operator.py

from airflow.operators.bash_operator import BaseOperator from airflow.utils.decorators import apply_defaults from hooks.my_airflow_hook import MyHook class MyOperator(BaseOperator): ...

Creating a plugins.zip file

You can use a built-in ZIP archive utility, or any other ZIP utility (such as 7zip) to create a .zip file.

Note

The built-in zip utility for Windows OS may add subfolders when you create a .zip file. We recommend verifying the contents of the plugins.zip file before uploading to your Amazon S3 bucket to ensure no additional directories were added.

  1. Change directories to your $AIRFLOW_HOME/plugins folder.

    myproject$ cd $AIRFLOW_HOME/plugins
  2. Run the following command to ensure that the contents have executable permissions (macOS and Linux only).

    $AIRFLOW_HOME/plugins$ chmod -R 755 .
  3. Zip the contents within your $AIRFLOW_HOME/plugins folder.

    $AIRFLOW_HOME/plugins$ zip -r plugins.zip .
  4. The directory structure of your plugins.zip may look like this:

    __init__.py my_airflow_plugin.py hooks/ |-- __init__.py |-- my_airflow_hook.py operators/ |-- __init__.py |-- my_airflow_operator.py sensors/ |-- __init__.py |-- my_airflow_sensor.py

Uploading plugins.zip to Amazon S3

You can use the Amazon S3 console or the AWS Command Line Interface (AWS CLI) to upload a plugins.zip file to your Amazon S3 bucket.

Using the AWS CLI

The AWS Command Line Interface (AWS CLI) is an open source tool that enables you to interact with AWS services using commands in your command-line shell. To complete the steps in this section, you need the following:

To upload using the AWS CLI

  1. The following command lists all Amazon S3 buckets.

    aws s3 ls
  2. The following command lists the files and folders in an Amazon S3 bucket.

    aws s3 ls s3://your-s3-bucket-any-name
  3. The following command uploads a plugins.zip file to an Amazon S3 bucket.

    aws s3 cp plugins.zip s3://your-s3-bucket-any-name/plugins.zip

Using the Amazon S3 console

The Amazon S3 console is a web-based user interface that allows you to create and manage the resources in your Amazon S3 bucket.

To upload using the Amazon S3 console

  1. Open the Environments page on the Amazon MWAA console.

  2. Choose an environment.

  3. Select the S3 bucket link in the DAG code in S3 pane to open your storage bucket on the Amazon S3 console.

  4. Choose Upload.

  5. Choose Add file.

  6. Select the local copy of your plugins.zip, choose Upload.

Specifying the path to plugins.zip on the Amazon MWAA console (the first time)

If this is the first time you're uploading a plugins.zip to your Amazon S3 bucket, you'll also need to specify the path to the file on the Amazon MWAA console. You only need to complete this step once.

  1. Open the Environments page on the Amazon MWAA console.

  2. Choose an environment.

  3. Choose Edit.

  4. On the DAG code in Amazon S3 pane, choose Browse S3 next to the Plugins file - optional field.

  5. Select the plugins.zip file on your Amazon S3 bucket.

  6. Choose Choose.

  7. Choose Next, Update environment.

You can begin using the new plugins immediately after your environment finishes updating.

Specifying the plugins.zip version on the Amazon MWAA console

You need to specify the version of your plugins.zip file on the Amazon MWAA console each time you upload a new version of your plugins.zip in your Amazon S3 bucket.

  1. Open the Environments page on the Amazon MWAA console.

  2. Choose an environment.

  3. Choose Edit.

  4. On the DAG code in Amazon S3 pane, choose a plugins.zip version in the dropdown list.

  5. Choose Next, Update environment.

You can begin using the new plugins immediately after your environment finishes updating.

Viewing changes on your Apache Airflow UI

To access your Apache Airflow UI

  1. Open the Environments page on the Amazon MWAA console.

  2. Choose an environment.

  3. Choose Open Airflow UI to view your Apache Airflow UI.

Note

You may need to ask your account administrator to add AmazonMWAAWebServerAccess permissions for your account to view your Apache Airflow UI. For more information, see Managing access.