Developing scripts using development endpoints

AWS Glue can create an environment—known as a development endpoint—that you can use to iteratively develop and test your extract, transform, and load (ETL) scripts. You can create, edit, and delete development endpoints using the AWS Glue console or API.

Note

Development endpoints are not supported for use with AWS Glue version 2.0 jobs. For more information, see Running Spark ETL Jobs with Reduced Startup Times.

Managing your development environment

When you create a development endpoint, you provide configuration values to provision the development environment. These values tell AWS Glue how to set up the network so that you can access the endpoint securely and the endpoint can access your data stores.

You can then create a notebook that connects to the endpoint, and use your notebook to author and test your ETL script. When you're satisfied with the results of your development process, you can create an ETL job that runs your script. With this process, you can add functions and debug your scripts in an interactive manner.

Follow the tutorials in this section to learn how to use your development endpoint with notebooks.

Topics

• Development endpoint workflow
• How AWS Glue development endpoints work with SageMaker notebooks
• Adding a development endpoint
• Viewing development endpoint properties
• Accessing your development endpoint
• Creating a notebook server hosted on Amazon EC2
• Tutorial setup: prerequisites for the development endpoint tutorials
• Tutorial: Set up a local Apache Zeppelin notebook to test and debug ETL scripts
• Tutorial: Set up an Apache Zeppelin notebook server on Amazon EC2
• Tutorial: Set up a Jupyter notebook in JupyterLab to test and debug ETL scripts
• Tutorial: Use a SageMaker notebook with your development endpoint
• Tutorial: Use a REPL shell with your development endpoint
• Tutorial: Set up PyCharm professional with a development endpoint
• Advanced configuration: sharing development endpoints among multiple users