Managing job runs with the AWS CLI - Amazon EMR
Configure a job runUse S3 logsUse CloudWatch LogsList job runsDescribe a job runCancel a job run

Managing job runs with the AWS CLI

This page covers how to manage job runs with the AWS Command Line Interface (AWS CLI).

Options for configuring a job run

Use the following options to configure job run parameters:

  • --execution-role-arn: You must provide an IAM role that is used for running jobs. For more information, see Using job execution roles with Amazon EMR on EKS.

  • --release-label: You can deploy Amazon EMR on EKS with Amazon EMR versions 5.32.0 and 6.2.0 and later. Amazon EMR on EKS is not supported in previous Amazon EMR release versions. For more information, see Amazon EMR on EKS releases.

  • --job-driver: Job driver is used to provide input on the main job. This is a union type field where you can only pass one of the values for the job type that you want to run. Supported job types include:

    • Spark submit jobs - Used to run a command through Spark submit. You can use this job type to run Scala, PySpark, SparkR, SparkSQL and any other supported jobs through Spark Submit. This job type has the following parameters:

      • Entrypoint - This is the HCFS (Hadoop compatible file system) reference to the main jar/py file you want to run.

      • EntryPointArguments - This is an array of arguments you want to pass to your main jar/py file. You should handle reading these parameters using your entrypoint code. Each argument in the array should be separated by a comma. EntryPointArguments cannot contain brackets or parentheses, such as (), {}, or [].

      • SparkSubmitParameters - These are the additional spark parameters you want to send to the job. Use this parameter to override default Spark properties such as driver memory or number of executors like —conf or —class. For additional information, see Launching Applications with spark-submit.

    • Spark SQL jobs - Used to run a SQL query file through Spark SQL. You can use this job type to run SparkSQL jobs. This job type has the following parameters:

  • --configuration-overrides: You can override the default configurations for applications by supplying a configuration object. You can use a shorthand syntax to provide the configuration or you can reference the configuration object in a JSON file. Configuration objects consist of a classification, properties, and optional nested configurations. Properties consist of the settings you want to override in that file. You can specify multiple classifications for multiple applications in a single JSON object. The configuration classifications that are available vary by Amazon EMR release version. For a list of configuration classifications that are available for each release version of Amazon EMR, see Amazon EMR on EKS releases.

    If you pass the same configuration in an application override and in Spark submit parameters, the Spark submit parameters take precedence. The complete configuration priority list follows, in order of highest priority to lowest priority.

    • Configuration supplied when creating SparkSession.

    • Configuration supplied as part of sparkSubmitParameters using —conf.

    • Configuration provided as part of application overrides.

    • Optimized configurations chosen by Amazon EMR for the release.

    • Default open source configurations for the application.

    To monitor job runs using Amazon CloudWatch or Amazon S3, you must provide the configuration details for CloudWatch. For more information, see Configure a job run to use Amazon S3 logs and Configure a job run to use Amazon CloudWatch Logs. If the S3 bucket or CloudWatch log group does not exist, then Amazon EMR creates it before uploading logs to the bucket.

  • For an additional list of Kubernetes configuration options, see Spark Properties on Kubernetes.

    The following Spark configurations are not supported.

    • spark.kubernetes.authenticate.driver.serviceAccountName

    • spark.kubernetes.authenticate.executor.serviceAccountName

    • spark.kubernetes.namespace

    • spark.kubernetes.driver.pod.name

    • spark.kubernetes.container.image.pullPolicy

    • spark.kubernetes.container.image

      Note

      You can use spark.kubernetes.container.image for customized Docker images. For more information, see Customizing Docker images for Amazon EMR on EKS.

Configure a job run to use Amazon S3 logs

To be able to monitor the job progress and to troubleshoot failures, you must configure your jobs to send log information to Amazon S3, Amazon CloudWatch Logs, or both. This topic helps you get started publishing application logs to Amazon S3 on your jobs that are launched with Amazon EMR on EKS.

S3 logs IAM policy

Before your jobs can send log data to Amazon S3, the following permissions must be included in the permissions policy for the job execution role. Replace DOC-EXAMPLE-BUCKET-LOGGING with the name of your logging bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::DOC-EXAMPLE-BUCKET-LOGGING",
                "arn:aws:s3:::DOC-EXAMPLE-BUCKET-LOGGING/*",
            ]
        }
    ]
}
Note

Amazon EMR on EKS can also create an Amazon S3 bucket. If an Amazon S3 bucket is not available, include the “s3:CreateBucket” permission in the IAM policy.

After you've given your execution role the proper permissions to send logs to Amazon S3, your log data are sent to the following Amazon S3 locations when s3MonitoringConfiguration is passed in the monitoringConfiguration section of a start-job-run request, as shown in Managing job runs with the AWS CLI.

  • Controller Logs - /logUri/virtual-cluster-id/jobs/job-id/containers/pod-name/(stderr.gz/stdout.gz)

  • Driver Logs - /logUri/virtual-cluster-id/jobs/job-id/containers/spark-application-id/spark-job-id-driver/(stderr.gz/stdout.gz)

  • Executor Logs - /logUri/virtual-cluster-id/jobs/job-id/containers/spark-application-id/executor-pod-name/(stderr.gz/stdout.gz)

Configure a job run to use Amazon CloudWatch Logs

To monitor job progress and to troubleshoot failures, you must configure your jobs to send log information to Amazon S3, Amazon CloudWatch Logs, or both. This topic helps you get started using CloudWatch Logs on your jobs that are launched with Amazon EMR on EKS. For more information about CloudWatch Logs, see Monitoring Log Files in the Amazon CloudWatch User Guide.

CloudWatch Logs IAM policy

For your jobs to send log data to CloudWatch Logs, the following permissions must be included in the permissions policy for the job execution role. Replace my_log_group_name and my_log_stream_prefix with names of your CloudWatch log group and log stream names, respectively. Amazon EMR on EKS creates the log group and log stream if they do not exist as long as the execution role ARN has appropriate permissions. 

{
     "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams"
            ],
            "Resource": [
                "arn:aws:logs:*:*:*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:my_log_group_name:log-stream:my_log_stream_prefix/*"
            ]
        }
    ]
}
Note

Amazon EMR on EKS can also create a log stream. If a log stream does not exist, the IAM policy should include the"logs:CreateLogGroup" permission.

After you've given your execution role the proper permissions, your application sends its log data to CloudWatch Logs when cloudWatchMonitoringConfiguration is passed in the monitoringConfiguration section of a start-job-run request, as shown in Managing job runs with the AWS CLI.

In the StartJobRun API, log_group_name is the log group name for CloudWatch, and log_stream_prefix is the log stream name prefix for CloudWatch. You can view and search these logs in the AWS Management Console.

  • Controller logs - logGroup/logStreamPrefix/virtual-cluster-id/jobs/job-id/containers/pod-name/(stderr/stdout)

  • Driver logs - logGroup/logStreamPrefix/virtual-cluster-id/jobs/job-id/containers/spark-application-id/spark-job-id-driver/(stderrstdout)

  • Executor logs - logGroup/logStreamPrefix/virtual-cluster-id/jobs/job-id/containers/spark-application-id/executor-pod-name/(stderr/stdout)

List job runs

You can run list-job-run to show the states of job runs, as the following example demonstrates. 

aws emr-containers list-job-runs --virtual-cluster-id <cluster-id>

Describe a job run

You can run describe-job-run to get more details about the job, such as job state, state details, and job name, as the following example demonstrates. 

aws emr-containers describe-job-run --virtual-cluster-id cluster-id --id job-run-id

Cancel a job run

You can run cancel-job-run to cancel running jobs, as the following example demonstrates.

aws emr-containers cancel-job-run --virtual-cluster-id cluster-id --id job-run-id