Enabling the Apache Spark Web UI for Development Endpoints - AWS Glue

Enabling the Apache Spark Web UI for Development Endpoints

Use the Apache Spark web UI to monitor and debug Spark applications running on AWS Glue development endpoints. You can configure the development endpoints with the Spark Web UI using the AWS Glue console or the AWS Command Line Interface (AWS CLI).

Enabling the Spark UI (Console)

Follow these steps to configure the Spark UI using the AWS Management Console.

To create a development endpoint with the Spark UI enabled

  1. Sign in to the AWS Management Console and open the AWS Glue console at https://console.aws.amazon.com/glue/.

  2. In the navigation pane, choose Dev endpoints.

  3. Choose Add endpoint.

  4. In Configuration, open the Spark UI options.

  5. In the Spark UI tab, choose Enable.

  6. Specify an Amazon S3 path for storing the Spark event logs.

Enabling the Spark UI (AWS CLI)

To create a development endpoint with the Spark UI enabled using the AWS CLI, pass in the following arguments in JSON syntax.

{ "EndpointName": "Name", "RoleArn": "role_ARN", "PublicKey": "public_key_contents", "NumberOfNodes": 2, "Arguments": { "--enable-spark-ui": "true", "--spark-event-logs-path": "s3://s3-event-log-path" } }