Creating a custom plugin that generates runtime environment variables - Amazon Managed Workflows for Apache Airflow
VersionPrerequisitesPermissionsRequirementsCustom pluginPlugins.zipAirflow configuration optionsWhat's next?

Creating a custom plugin that generates runtime environment variables

The following sample walks you through the steps to create a custom plugin that generates environment variables at runtime on an Amazon Managed Workflows for Apache Airflow environment.

Version

  • The sample code on this page can be used with Apache Airflow v1 in Python 3.7.

Prerequisites

To use the sample code on this page, you'll need the following:

Permissions

  • No additional permissions are required to use the code example on this page.

Requirements

  • To use this code example with Apache Airflow v1, no additional dependencies are required. The code uses the Apache Airflow v1 base install on your environment.

Custom plugin

Apache Airflow will execute the contents of Python files in the plugins folder at startup. This is used to set and modify environment variables. The following steps describe the sample code for the custom plugin.

  1. In your command prompt, navigate to the directory where your plugins are stored. For example:

    cd plugins

  2. Copy the contents of the following code sample and save locally as env_var_plugin.py in the above folder.

    from airflow.plugins_manager import AirflowPlugin
import os

os.environ["PATH"] = os.getenv("PATH") + ":/usr/local/airflow/.local/lib/python3.7/site-packages" 
os.environ["JAVA_HOME"]="/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-1.amzn2.0.1.x86_64"

class EnvVarPlugin(AirflowPlugin):                
     name = 'env_var_plugin'

Plugins.zip

The following steps show how to create plugins.zip. The contents of this example can be combined with other plugins and binaries into a single plugins.zip file.

  1. In your command prompt, navigate to the hive_plugin directory from the previous step. For example:

    cd plugins

  2. Zip the contents within your plugins folder.

    zip -r ../plugins.zip ./

Airflow configuration options

If you're using Apache Airflow v2, add core.lazy_load_plugins : False as an Apache Airflow configuration option. To learn more, see Using configuration options to load plugins in 2.

What's next?