Menu
AWS Glue
Developer Guide

Troubleshooting Errors in AWS Glue

If you encounter errors in AWS Glue, use the following solutions to help you find the source of the problems and fix them.

Note

The AWS Glue GitHub repository contains additional troubleshooting guidance in AWS Glue Frequently Asked Questions.

Error: Resource Unavailable

If AWS Glue returns a resource unavailable message, you can view error messages or logs to help you learn more about the issue. The following tasks describe general methods for troubleshooting.

  • A custom DNS configuration without reverse lookup can cause AWS Glue to fail. Check your DNS configuration. If you are using Amazon Route 53 or Microsoft Active Directory, make sure that there are forward and reverse lookups. For more information, see Setting Up DNS in Your VPC.

  • For any connections and development endpoints that you use, check that your cluster has not run out of elastic network interfaces.

Error: Could Not Find S3 Endpoint or NAT Gateway for subnetId in VPC

Check the subnet ID and VPC ID in the message to help you diagnose the issue.

  • Check that you have an Amazon S3 VPC endpoint set up, which is required with AWS Glue. In addition, check your NAT gateway if that's part of your configuration. For more information, see Amazon VPC Endpoints for Amazon S3.

Error: Inbound Rule in Security Group Required

At least one security group must open all ingress ports. To limit traffic, the source security group in your inbound rule can be restricted to the same security group.

Error: Outbound Rule in Security Group Required

At least one security group must open all egress ports. To limit traffic, the source security group in your outbound rule can be restricted to the same security group.

Error: DescribeVpcEndpoints Action Is Unauthorized. Unable to Validate VPC ID vpc-id

  • Check the policy passed to AWS Glue for the ec2:DescribeVpcEndpoints permission.

Error: DescribeRouteTables Action Is Unauthorized. Unable to Validate Subnet Id: subnet-id in VPC id: vpc-id

  • Check the policy passed to AWS Glue for the ec2:DescribeRouteTables permission.

Error: Failed to Call ec2:DescribeSubnets

  • Check the policy passed to AWS Glue for the ec2:DescribeSubnets permission.

Error: Failed to Call ec2:DescribeSecurityGroups

  • Check the policy passed to AWS Glue for the ec2:DescribeSecurityGroups permission.

Error: Could Not Find Subnet for AZ

  • The Availability Zone might not be available to AWS Glue. Create and use a new subnet in a different Availability Zone from the one specified in the message.

Error: Amazon S3 Timeout

If AWS Glue returns a connect timed out error, it might be because it is trying to access an Amazon S3 bucket in another AWS Region.

  • An Amazon S3 VPC endpoint can only route traffic to buckets within an AWS Region. If you need to connect to buckets in other Regions, a possible workaround is to use a NAT gateway. For more information, see NAT Gateways.

Error: No Private DNS For Network Interface Found

If a job fails or a development endpoint fails to provision, it might be because of a problem in the network setup.

  • If you are using the Amazon-provided DNS, the value of enableDnsHostnames must be set to true. For more information, see DNS.

Error: Development Endpoint Provisioning Failed

If AWS Glue fails to successfully provision a development endpoint, it might be because of a problem in the network setup.

  • When you define a development endpoint, the VPC, subnet, and security groups are validated to confirm that they meet certain requirements.

  • If you provided the optional SSH public key, check that it is a valid SSH public key.

  • Check in the VPC console that your VPC uses a valid DHCP option set. For more information, see DHCP option sets.

  • If the cluster remains in the PROVISIONING state, contact AWS Support.

Error: Notebook Server CREATE_FAILED

If AWS Glue fails to create the notebook server for a development endpoint, it might be because of one of the following problems:

  • AWS Glue passes an IAM role to Amazon EC2 when it is setting up the notebook server. The IAM role must have a trust relationship to Amazon EC2.

  • The IAM role must have an instance profile of the same name. When you create the role with the IAM console, the instance profile with the same name is automatically created. Check for an error in the log regarding an invalid instance profile name iamInstanceProfile.name. For more information, see Using Instance Profiles.

  • Check that your role has permission to access aws-glue* buckets in the policy that you pass to create the notebook server.

Error: Notebook Usage Errors

When using an Apache Zeppelin notebook, you might encounter errors due to your setup or environment.

  • You provide an IAM role with an attached policy when you created the notebook server. If the policy does not include all the required permissions, you might get an error such as assumed-role/name-of-role/i-0bf0fa9d038087062 is not authorized to perform some-action AccessDeniedException. Check the policy that is passed to your notebook server in the IAM console.

  • If the Zeppelin notebook does not render correctly in your web browser, check the Zeppelin requirements for browser support. For example, there might be specific versions and setup required for the Safari browser. You might need to update your browser or use a different browser.

Error: Running Crawler Failed

If AWS Glue fails to successfully run a crawler to catalog your data, it might be because of one of the following reasons. First check if an error is listed in the AWS Glue console crawlers list. Check if there is an exclamation icon next to the crawler name and hover over the icon to see any associated messages.

  • Check the logs for the crawler run in CloudWatch Logs under /aws-glue/crawlers. The URL link to the crawler logs on the AWS Glue console contains both the crawler name and the crawler ID. Also, the first record of the crawler log in CloudWatch Logs contains the crawler ID for that run.

Error: Upgrading Athena Data Catalog

If you encounter errors while upgrading your Athena Data Catalog to the AWS Glue Data Catalog, see the Amazon Athena User Guide topic Upgrading to the AWS Glue Data Catalog Step-by-Step.