Monitoring jobs using the Apache Spark web UI - AWS Glue

Monitoring jobs using the Apache Spark web UI

You can use the Apache Spark web UI to monitor and debug AWS Glue ETL jobs running on the AWS Glue job system, and also Spark applications running on AWS Glue development endpoints. The Spark UI enables you to check the following for each job:

  • The event timeline of each Spark stage

  • A directed acyclic graph (DAG) of the job

  • Physical and logical plans for SparkSQL queries

  • The underlying Spark environmental variables for each job

For more information about using the Spark Web UI, see Web UI in the Spark documentation.

You can see the Spark UI in the AWS Glue console for newer jobs. This is available when a AWS Glue job runs on AWS Glue 3.0 or later versions with logs generated in the Standard (rather than legacy) format, which is the default for newer jobs. For more information on finding the Spark UI in the console, see View information for recent job runs. For other versions of AWS Glue, you will need to provision your own history server. For more information, see Launching the Spark history server.

You can enable the Spark UI using the AWS Glue console or the AWS Command Line Interface (AWS CLI). When you enable the Spark UI, AWS Glue ETL jobs and Spark applications on AWS Glue development endpoints can persist Spark event logs to a location that you specify in Amazon Simple Storage Service (Amazon S3). The persisted event logs in Amazon S3 can be used with the Spark UI both in real time as the job is executing and after the job is complete. As long as the logs remain in Amazon S3, the Spark UI in the AWS Glue console will be able to view them.

In order to use the Spark UI in the AWS Glue console, your console role must have the glue:UseGlueStudio permission. For more information about this permission, see Creating Custom IAM Policies for AWS Glue Studio.


  • Spark UI in the AWS Glue console is not available for job runs that occurred before 20 Nov 2023, as they are in the legacy log format.

  • Spark UI in the AWS Glue console does not support rolling logs, such as those generated by default in streaming jobs.

    You can turn off rolling logs for a streaming job by passing in additional configuration. Be aware that very large log files may cost a lot to maintain.

    To turn off rolling logs, provide the following configuration:

    '--spark-ui-event-logs-path': 'true', '--conf': 'spark.eventLog.rolling.enabled=false'

Example: Apache Spark web UI

This example shows how to use the Spark UI to understand your job performance. Screenshots show the Spark web UI provided by a self-managed Spark history server, Spark UI in the AWS Glue console will provide similar views. For more information about using the Spark Web UI, see Web UI in the Spark documentation.

The following is an example of a Spark application which reads from two data sources, performs a join transform, and writes it out to Amazon S3 in Parquet format.

import sys from awsglue.transforms import * from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job from pyspark.sql.functions import count, when, expr, col, sum, isnull from pyspark.sql.functions import countDistinct from awsglue.dynamicframe import DynamicFrame args = getResolvedOptions(sys.argv, ['JOB_NAME']) sc = SparkContext() glueContext = GlueContext(sc) spark = glueContext.spark_session job = Job(glueContext) job.init(args['JOB_NAME']) df_persons ="s3://awsglue-datasets/examples/us-legislators/all/persons.json") df_memberships ="s3://awsglue-datasets/examples/us-legislators/all/memberships.json") df_joined = df_persons.join(df_memberships, == df_memberships.person_id, 'fullouter') df_joined.write.parquet("s3://aws-glue-demo-sparkui/output/") job.commit()

The following DAG visualization shows the different stages in this Spark job.

      Screenshot of Spark UI showing 2 completed stages for job 0.

The following event timeline for a job shows the start, execution, and termination of different Spark executors.

      Screenshot of Spark UI showing the completed, failed, and active stages of different
        Spark executors.

The following screen shows the details of the SparkSQL query plans:

  • Parsed logical plan

  • Analyzed logical plan

  • Optimized logical plan

  • Physical plan for execution

      SparkSQL query plans: parsed, analyzed, and optimized logical plan and physical plans
        for execution.