Developing scripts using development endpoints

Note

Development Endpoints are only supported for versions of AWS Glue prior to 2.0. For an interactive environment where you can author and test ETL scripts, use Notebooks on AWS Glue Studio.

AWS Glue can create an environment—known as a development endpoint—that you can use to iteratively develop and test your extract, transform, and load (ETL) scripts. You can create, edit, and delete development endpoints using the AWS Glue console or API.

Managing your development environment

When you create a development endpoint, you provide configuration values to provision the development environment. These values tell AWS Glue how to set up the network so that you can access the endpoint securely and the endpoint can access your data stores.

You can then create a notebook that connects to the endpoint, and use your notebook to author and test your ETL script. When you're satisfied with the results of your development process, you can create an ETL job that runs your script. With this process, you can add functions and debug your scripts in an interactive manner.

Follow the tutorials in this section to learn how to use your development endpoint with notebooks.

Topics

• Development endpoint workflow
• How AWS Glue development endpoints work with SageMaker notebooks
• Adding a development endpoint
• Accessing your development endpoint
• Tutorial: Set up a Jupyter notebook in JupyterLab to test and debug ETL scripts
• Tutorial: Use a SageMaker notebook with your development endpoint
• Tutorial: Use a REPL shell with your development endpoint
• Tutorial: Set up PyCharm professional with a development endpoint
• Advanced configuration: sharing development endpoints among multiple users