Menu
AWS Glue
Developer Guide

Using Development Endpoints for Developing Scripts

AWS Glue can create an environment for you to iteratively develop and test your extract, transform, and load (ETL) scripts. You can develop your script in a notebook and point to an AWS Glue endpoint to test it. When you're satisfied with the results of your development process, you can create an ETL job that runs your script. With this process, you can add functions and debug your script in an interactive manner.

Managing Your Development Environment

With AWS Glue, you can create, edit, and delete development endpoints. You provide configuration values to provision the development environments. These values tell AWS Glue how to set up the network so that you can access your development endpoint securely, and your endpoint can access your data stores. Then, create a notebook that connects to the development endpoint, and use your notebook to author and test your ETL script.

For more information about managing a development endpoint using the AWS Glue console, see Working with Development Endpoints on the AWS Glue Console.

How to Use a Development Endpoint

To use a development endpoint, you can follow this workflow.

  1. Create an AWS Glue development endpoint through the console or API. This endpoint is launched in your virtual private cloud (VPC) with your defined security groups.

  2. The console or API can poll the development endpoint until it is provisioned and ready for work. When it's ready, you can connect to the development endpoint to create and test AWS Glue scripts.

    • You can create an Apache Zeppelin notebook using the AWS Glue console. Then, with your web browser, connect to the notebook server in your account. While you develop your scripts, you can discover, view, visualize, analyze, and validate your data.

  3. When testing is completed, you can delete the development endpoint.