Managing job runs using the AWS CLI - Amazon EMR

Managing job runs using the AWS CLI

This topic covers how to manage job runs using the AWS Command Line Interface (AWS CLI).

Submit a job run

To submit a job run using a JSON file with specified parameters

  1. Create a start-job-run-request.json file and specify the required parameters for your job run, as the following example JSON file demonstrates. For more information about the parameters, see Options for configuring a job run.

    { "name": "myjob", "virtualClusterId": "123456", "executionRoleArn": "iam_role_name_for_job_execution", "releaseLabel": "emr-6.2.0-latest", "jobDriver": { "sparkSubmitJobDriver": { "entryPoint": "entryPoint_location", "entryPointArguments": ["arguments_list"], "sparkSubmitParameters": "--class <main_class> --conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.executor.cores=2 --conf spark.driver.cores=1" } }, "configurationOverrides": { "applicationConfiguration": [ { "classification": "spark-defaults", "properties": { "spark.driver.memory":"2G" } } ], "monitoringConfiguration": { "persistentAppUI": "ENABLED", "cloudWatchMonitoringConfiguration": { "logGroupName": "my_log_group", "logStreamNamePrefix": "log_stream_prefix" }, "s3MonitoringConfiguration": { "logUri": "s3://my_s3_log_location" } } } }
  2. Use the start-job-run command with a path to the start-job-run-request.json file stored locally or in Amazon S3.

    aws emr-containers start-job-run \ --cli-input-json file://./start-job-run-request.json

To start a job run using the start-job-run command

  • Supply all the specified parameters in the StartJobRun command, as the following example demonstrates.

    aws emr-containers start-job-run \ --virtual-cluster-id 123456 \ --name myjob \ --execution-role-arn execution-role-arn \ --release-label emr-6.2.0-latest \ --job-driver '{"sparkSubmitJobDriver": {"entryPoint": "entryPoint_location", "entryPointArguments": [arguments_list], "sparkSubmitParameters": "--class <main_class> --conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.executor.cores=2 --conf spark.driver.cores=1"}}' \ --configuration-overrides '{"applicationConfiguration": [{"classification": "spark-defaults", "properties": {"spark.driver.memory": "2G"}}], "monitoringConfiguration": {"cloudWatchMonitoringConfiguration": {"logGroupName": "log_group_name", "logStreamNamePrefix": "log_stream_prefix"}, "persistentAppUI":"ENABLED", "s3MonitoringConfiguration": {"logUri": "s3://my_s3_log_location" }}}'

Options for configuring a job run

Use the following options to configure job run parameters:

  • --execution-role-arn: You must provide an IAM role that is used for running jobs. For more information, see Using job execution roles with Amazon EMR on EKS.

  • --release-label: You can deploy Amazon EMR on EKS with Amazon EMR versions 5.32.0 and 6.2.0 and later. Amazon EMR on EKS is not supported in previous Amazon EMR release versions. For more information, see Amazon EMR on EKS release versions.

  • --job-driver: Job driver is used to provide input on the main job. This is a union type field where you can only pass one of the values for the job type that you want to run. Supported job types include:

    • Spark submit jobs - Used to run a command through Spark submit. You can use this job type to run Scala, Pyspark, SparkR, SparkSQL and any other supported jobs through Spark Submit. This job type has the following parameters:

      • Entrypoint - This is the HCFS (Hadoop compatible file system) reference to the main jar/py file you want to run.

      • Entrypoint Args - This is the argument you want to pass to your JAR. You should handle reading this parameter using your entrypoint code.

      • SparkSubmitParameters - These are the additional spark parameters you want to send to the job. Use this parameter to override default Spark properties such as driver memory or number of executors like —conf or —class. For additional information, see Launching Applications with spark-submit.

  • --configuration-overrides: You can override the default configurations for applications by supplying a configuration object. You can use a shorthand syntax to provide the configuration or you can reference the configuration object in a JSON file. Configuration objects consist of a classification, properties, and optional nested configurations. Properties consist of the settings you want to override in that file. You can specify multiple classifications for multiple applications in a single JSON object. The configuration classifications that are available vary by Amazon EMR release version. For a list of configuration classifications that are available for each release version of Amazon EMR, see Amazon EMR on EKS release versions.

    If you pass the same configuration in an application override and in Spark submit parameters, the Spark submit parameters take precedence. The complete configuration priority list follows, in order of highest priority to lowest priority.

    • Configuration supplied when creating SparkSession.

    • Configuration supplied as part of sparkSubmitParameters using —conf.

    • Configuration provided as part of application overrides.

    • Optimized configurations chosen by Amazon EMR for the release.

    • Default open source configurations for the application.

    To monitor job runs using Amazon CloudWatch or Amazon S3, you must provide the configuration details for CloudWatch. For more information, see Configure a job run to use S3 logs and Configure a job run to use CloudWatch Logs. If the S3 bucket or CloudWatch log group does not exist, then Amazon EMR creates it before uploading logs to the bucket.

  • For an additional list of Kubernetes configuration options, see Spark Properties on Kubernetes.

    The following Spark configurations are not supported.

    • spark.kubernetes.authenticate.driver.serviceAccountName

    • spark.kubernetes.authenticate.executor.serviceAccountName

    • spark.kubernetes.container.image

    • spark.kubernetes.namespace

    • spark.kubernetes.driver.pod.name

    • spark.kubernetes.container.image.pullPolicy

Configure a job run to use S3 logs

You can configure your jobs to send log information to Amazon S3. This topic helps you get started publishing application logs to Amazon S3 on your jobs that were launched with Amazon EMR on EKS.

S3 logs IAM policy

Before your jobs can send log data to Amazon S3, the following permissions must be included in the permissions policy for the job execution role.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:ListBucket" ], "Resource": "*" } ] }
Note

Amazon EMR on EKS can also create an Amazon S3 bucket. If an Amazon S3 bucket is not available, the IAM policy should include the “s3:CreateBucket” permission.

After you have given your execution role the proper permissions to send logs to Amazon S3, your log data are sent to the following Amazon S3 locations.

  • Controller Logs - /logUri/virtual-cluster-id/jobs/job-id/containers/pod-name/(stderr.gz/stdout.gz)

  • Driver Logs - /logUri/virtual-cluster-id/jobs/job-id/containers/spark-application-id/spark-job-id-driver/(stderr.gz/stdout.gz)

  • Executor Logs - /logUri/virtual-cluster-id/jobs/job-id/containers/spark-application-id/executor-pod-name/(stderr.gz/stdout.gz)

Configure a job run to use CloudWatch Logs

You can configure your jobs to send log information to Amazon CloudWatch Logs. This topic helps you get started using CloudWatch Logs on your jobs that were launched with an Amazon EMR on EKS. For more information about CloudWatch Logs, see Monitoring Log Files in the Amazon CloudWatch User Guide.

CloudWatch Logs IAM policy

Before your jobs can send log data to CloudWatch Logs, the following permissions must be included in the permissions policy for the job execution role. Amazon EMR on EKS creates the log group and log stream if they do not exist as long as the execution role ARN has appropriate permissions.

{"Version": "2012-10-17", "Statement": [ {"Effect": "Allow", "Action": [ "logs:PutLogEvents", "logs:CreateLogStream", "logs:DescribeLogGroups", "logs:DescribeLogStreams" ], "Resource": [ "arn:aws:logs:*:*:*" ] } ] }
Note

Amazon EMR on EKS can also create log stream. If log stream does not exist, the IAM policy should include the"logs:CreateLogGroup" permission.

After you have given your execution role the proper permissions, your application should send its log data to CloudWatch Logs.

In the StartJobRun API, log_group_name is the log group name for CloudWatch, and log_stream_prefix is the log stream name prefix for CloudWatch. You can view and search these logs in the AWS Management Console.

  • Controller logs - logGroup/logStreamPrefix/virtual-cluster-id/jobs/job-id/containers/pod-name/(stderr/stdout)

  • Driver logs - logGroup/logStreamPrefix/virtual-cluster-id/jobs/job-id/containers/spark-application-id/spark-job-id-driver/(stderrstdout)

  • Executor logs - logGroup/logStreamPrefix/virtual-cluster-id/jobs/job-id/containers/spark-application-id/executor-pod-name/(stderr/stdout)

List job runs

You can run list-job-run to show the states of job runs, as the following example demonstrates.

aws emr-containers list-job-runs --virtual-cluster-id <cluster-id>

Describe a job run

You can run describe-job-run to get more details about the job, such as job state, state details, and job name, as the following example demonstrates.

aws emr-containers describe-job-run --virtual-cluster-id cluster-id --id job-run-id

Cancel a job run

You can run cancel-job-run to cancel running jobs, as the following example demonstrates.

aws emr-containers cancel-job-run --virtual-cluster-id cluster-id --id job-run-id