Creating a custom plugin with Oracle - Amazon Managed Workflows for Apache Airflow

Creating a custom plugin with Oracle

The following sample walks you through the steps to create a custom plugin using Oracle for Amazon MWAA and can be combined with other custom plugins and binaries in your file.


  • The sample code on this page can be used with Apache Airflow v1 in Python 3.7.

  • You can use the code example on this page with Apache Airflow v2 and above in Python 3.10.


To use the sample code on this page, you'll need the following:


  • No additional permissions are required to use the code example on this page.


To use the sample code on this page, add the following dependencies to your requirements.txt. To learn more, see Installing Python dependencies.

Apache Airflow v2
-c cx_Oracle apache-airflow-providers-oracle
Apache Airflow v1
cx_Oracle==8.1.0 apache-airflow[oracle]==1.10.12

Code sample

The following steps describe how to create the DAG code that will test the custom plugin.

  1. In your command prompt, navigate to the directory where your DAG code is stored. For example:

    cd dags
  2. Copy the contents of the following code sample and save locally as

    from airflow import DAG from airflow.operators.python_operator import PythonOperator from airflow.utils.dates import days_ago import os import cx_Oracle DAG_ID = os.path.basename(__file__).replace(".py", "") def testHook(**kwargs): cx_Oracle.init_oracle_client() version = cx_Oracle.clientversion() print("cx_Oracle.clientversion",version) return version with DAG(dag_id=DAG_ID, schedule_interval=None, catchup=False, start_date=days_ago(1)) as dag: hook_test = PythonOperator( task_id="hook_test", python_callable=testHook, provide_context=True )

Create the custom plugin

This section describes how to download the dependencies, create the custom plugin and the

Download dependencies

Amazon MWAA will extract the contents of into /usr/local/airflow/plugins on each Amazon MWAA scheduler and worker container. This is used to add binaries to your environment. The following steps describe how to assemble the files needed for the custom plugin.

Pull the Amazon Linux container image
  1. In your command prompt, pull the Amazon Linux container image, and run the container locally. For example:

    docker pull amazonlinux docker run -it amazonlinux:latest /bin/bash

    Your command prompt should invoke a bash command line. For example:

  2. Install the Linux-native asynchronous I/O facility (libaio).

    yum -y install libaio
  3. Keep this window open for subsequent steps. We'll be copying the following files locally: lib64/, lib64/, lib64/

Download client folder
  1. Install the unzip package locally. For example:

    sudo yum install unzip
  2. Create an oracle_plugin directory. For example:

    mkdir oracle_plugin cd oracle_plugin
  3. Use the following curl command to download the from Oracle Instant Client Downloads for Linux x86-64 (64-bit).

    curl >
  4. Unzip the file. For example:

    unzip *.zip
Extract files from Docker
  1. In a new command prompt, display and write down your Docker container ID. For example:

    docker container ls

    Your command prompt should return all containers and their IDs. For example:

  2. In your oracle_plugin directory, extract the lib64/, lib64/, lib64/ files to the local instantclient_18_5 folder. For example:

    docker cp debc16fd6970:/lib64/ instantclient_18_5/ docker cp debc16fd6970:/lib64/ instantclient_18_5/ docker cp debc16fd6970:/lib64/ instantclient_18_5/

Custom plugin

Apache Airflow will execute the contents of Python files in the plugins folder at startup. This is used to set and modify environment variables. The following steps describe the sample code for the custom plugin.

  • Copy the contents of the following code sample and save locally as

    from airflow.plugins_manager import AirflowPlugin import os os.environ["LD_LIBRARY_PATH"]='/usr/local/airflow/plugins/instantclient_18_5' os.environ["DPI_DEBUG_LEVEL"]="64" class EnvVarPlugin(AirflowPlugin): name = 'env_var_plugin'

The following steps show how to create the The contents of this example can be combined with your other plugins and binaries into a single file.

Zip the contents of the plugin directory
  1. In your command prompt, navigate to the oracle_plugin directory. For example:

    cd oracle_plugin
  2. Zip the instantclient_18_5 directory in For example:

    zip -r ../ ./
  3. You should see the following in your command prompt:

    oracle_plugin$ ls instantclient_18_5
  4. Remove the file. For example:

Zip the file
  1. Add the file to the root of the For example:

  2. Your should now include the following: instantclient_18_5/

Airflow configuration options

If you're using Apache Airflow v2, add core.lazy_load_plugins : False as an Apache Airflow configuration option. To learn more, see Using configuration options to load plugins in 2.

What's next?