Troubleshooting Amazon Managed Workflows for Apache Airflow

This topic describes common issues and errors you may encounter when using Apache Airflow on Amazon Managed Workflows for Apache Airflow and recommended steps to resolve these errors.

Contents

• Troubleshooting: DAGs, Operators, Connections, and other issues in Apache Airflow v2
  • Connections
    • I can't connect to Secrets Manager
    • How do I configure secretsmanager:ResourceTag/<tag-key> secrets manager conditions or a resource restriction in my execution role policy?
    • I can't connect to Snowflake
    • I can't see my connection in the Airflow UI
  • Web server
    • I see a 5xx error accessing the web server
    • I see a 'The scheduler does not appear to be running' error
  • Tasks
    • I see my tasks stuck or not completing
  • CLI
    • I see a '503' error when triggering a DAG in the CLI
    • Why does the dags backfill Apache Airflow CLI command fail? Is there a workaround?
  • Operators
    • I received a PermissionError: [Errno 13] Permission denied error using the S3Transform operator
• Troubleshooting: DAGs, Operators, Connections, and other issues in Apache Airflow v1
  • Updating requirements.txt
    • Adding apache-airflow-providers-amazon causes my environment to fail
  • Broken DAG
    • I received a 'Broken DAG' error when using Amazon DynamoDB operators
    • I received 'Broken DAG: No module named psycopg2' error
    • I received a 'Broken DAG' error when using the Slack operators
    • I received various errors installing Google/GCP/BigQuery
    • I received 'Broken DAG: No module named Cython' error
  • Operators
    • I received an error using the BigQuery operator
  • Connections
    • I can't connect to Snowflake
    • I can't connect to Secrets Manager
    • I can't connect to my MySQL server on '<DB-identifier-name>.cluster-id.<region>.rds.amazonaws.com'
  • Web server
    • I'm using the BigQueryOperator and it's causing my web server to crash
    • I see a 5xx error accessing the web server
    • I see a 'The scheduler does not appear to be running' error
  • Tasks
    • I see my tasks stuck or not completing
  • CLI
    • I see a '503' error when triggering a DAG in the CLI
• Troubleshooting: Creating and updating an Amazon MWAA environment
  • Updating requirements.txt
    • I specified a new version of my requirements.txt and it's taking more than 20 minutes to update my environment
  • Plugins
    • Does Amazon MWAA support implementing custom UI?
    • I am able to implement custom UI changes on the Amazon MWAA local runner via plugins, yet when I try to do the same on Amazon MWAA, I do not see my changes nor any errors. Why is this happening?
  • Create bucket
    • I can't select the option for S3 Block Public Access settings
  • Create environment
    • I tried to create an environment and it's stuck in the "Creating" state
    • I tried to create an environment but it shows the status as "Create failed"
    • I tried to select a VPC and received a "Network Failure" error
    • I tried to create an environment and received a service, partition, or resource "must be passed" error
    • I tried to create an environment and it shows the status as "Available" but when I try to access the Airflow UI an "Empty Reply from Server" or "502 Bad Gateway" error is shown
    • I tried to create an environment and my user name is a bunch of random character names
  • Update environment
    • I tried changing the environment class but the update failed
  • Access environment
    • I can't access the Apache Airflow UI
• Troubleshooting: CloudWatch Logs and CloudTrail errors
  • Logs
    • I can't see my task logs, or I received a 'Reading remote log from Cloudwatch log_group' error
    • Tasks are failing without any logs
    • I see a 'ResourceAlreadyExistsException' error in CloudTrail
    • I see an 'Invalid request' error in CloudTrail
    • I see a 'Cannot locate a 64-bit Oracle Client library: "libclntsh.so: cannot open shared object file: No such file or directory' in Apache Airflow logs
    • I see psycopg2 'server closed the connection unexpectedly' in my Scheduler logs
    • I see 'Executor reports task instance %s finished (%s) although the task says its %s' in my DAG processing logs
    • I see 'Could not read remote logs from log_group: airflow-*{*environmentName}-Task log_stream:* {*DAG_ID}/*{*TASK_ID}/*{*time}/*{*n}.log.' in my task logs