Managing job runs with the AWS CLI
This topic covers how to manage job runs with the AWS Command Line Interface (AWS CLI).
Topics
Submit a job run
To submit a job run with a JSON file with specified parameters
-
Create a
start-job-run-request.json
file and specify the required parameters for your job run, as the following example JSON file demonstrates. For more information about the parameters, see Options for configuring a job run.{ "name": "
myjob
", "virtualClusterId": "123456
", "executionRoleArn": "iam_role_name_for_job_execution
", "releaseLabel": "emr-6.2.0-latest
", "jobDriver": { "sparkSubmitJobDriver": { "entryPoint": "entryPoint_location
", "entryPointArguments": ["argument1
", "argument2
", ...], "sparkSubmitParameters": "--class <main_class> --conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.executor.cores=2 --conf spark.driver.cores=1" } }, "configurationOverrides": { "applicationConfiguration": [ { "classification": "spark-defaults", "properties": { "spark.driver.memory":"2G" } } ], "monitoringConfiguration": { "persistentAppUI": "ENABLED", "cloudWatchMonitoringConfiguration": { "logGroupName": "my_log_group
", "logStreamNamePrefix": "log_stream_prefix
" }, "s3MonitoringConfiguration": { "logUri": "s3://my_s3_log_location
" } } } } -
Use the
start-job-run
command with a path to thestart-job-run-request.json
file stored locally.aws emr-containers start-job-run \ --cli-input-json
file://./start-job-run-request.json
To start a job run using the start-job-run
command
-
Supply all the specified parameters in the
StartJobRun
command, as the following example demonstrates.aws emr-containers start-job-run \ --virtual-cluster-id
123456
\ --namemyjob
\ --execution-role-arnexecution-role-arn
\ --release-labelemr-6.2.0-latest
\ --job-driver '{"sparkSubmitJobDriver": {"entryPoint": "entryPoint_location
", "entryPointArguments": ["argument1
", "argument2
", ...], "sparkSubmitParameters": "--class <main_class> --conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.executor.cores=2 --conf spark.driver.cores=1"}}' \ --configuration-overrides '{"applicationConfiguration": [{"classification": "spark-defaults", "properties": {"spark.driver.memory": "2G"}}], "monitoringConfiguration": {"cloudWatchMonitoringConfiguration": {"logGroupName": "log_group_name
", "logStreamNamePrefix": "log_stream_prefix
"}, "persistentAppUI":"ENABLED", "s3MonitoringConfiguration": {"logUri": "s3://my_s3_log_location
" }}}' -
For Spark SQL, supply all the specified parameters in the
StartJobRun
command, as the following example demonstrates.aws emr-containers start-job-run \ --virtual-cluster-id
123456
\ --namemyjob
\ --execution-role-arnexecution-role-arn
\ --release-labelemr-6.7.0-latest
\ --job-driver '{"sparkSqlJobDriver": {"entryPoint": "entryPoint_location
", "sparkSqlParameters": "--conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.executor.cores=2 --conf spark.driver.cores=1"}}' \ --configuration-overrides '{"applicationConfiguration": [{"classification": "spark-defaults", "properties": {"spark.driver.memory": "2G"}}], "monitoringConfiguration": {"cloudWatchMonitoringConfiguration": {"logGroupName": "log_group_name
", "logStreamNamePrefix": "log_stream_prefix
"}, "persistentAppUI":"ENABLED", "s3MonitoringConfiguration": {"logUri": "s3://my_s3_log_location
" }}}'
Options for configuring a job run
Use the following options to configure job run parameters:
-
--execution-role-arn
: You must provide an IAM role that is used for running jobs. For more information, see Using job execution roles with Amazon EMR on EKS. -
--release-label
: You can deploy Amazon EMR on EKS with Amazon EMR versions 5.32.0 and 6.2.0 and later. Amazon EMR on EKS is not supported in previous Amazon EMR release versions. For more information, see Amazon EMR on EKS release versions. -
--job-driver
: Job driver is used to provide input on the main job. This is a union type field where you can only pass one of the values for the job type that you want to run. Supported job types include:-
Spark submit jobs - Used to run a command through Spark submit. You can use this job type to run Scala, PySpark, SparkR, SparkSQL and any other supported jobs through Spark Submit. This job type has the following parameters:
-
Entrypoint - This is the HCFS (Hadoop compatible file system) reference to the main jar/py file you want to run.
-
EntryPointArguments - This is an array of arguments you want to pass to your main jar/py file. You should handle reading these parameters using your entrypoint code. Each argument in the array should be separated by a comma. EntryPointArguments cannot contain brackets or parentheses, such as (), {}, or [].
-
SparkSubmitParameters - These are the additional spark parameters you want to send to the job. Use this parameter to override default Spark properties such as driver memory or number of executors like —conf or —class. For additional information, see Launching Applications with spark-submit
.
-
-
Spark SQL jobs - Used to run a SQL query file through Spark SQL. You can use this job type to run SparkSQL jobs. This job type has the following parameters:
-
Entrypoint - This is the HCFS (Hadoop compatible file system) reference to the SQL query file you want to run.
For a list of additional Spark parameters you can use for a Spark SQL job, see Running Spark SQL scripts through the StartJobRun API.
-
-
-
--configuration-overrides
: You can override the default configurations for applications by supplying a configuration object. You can use a shorthand syntax to provide the configuration or you can reference the configuration object in a JSON file. Configuration objects consist of a classification, properties, and optional nested configurations. Properties consist of the settings you want to override in that file. You can specify multiple classifications for multiple applications in a single JSON object. The configuration classifications that are available vary by Amazon EMR release version. For a list of configuration classifications that are available for each release version of Amazon EMR, see Amazon EMR on EKS release versions.If you pass the same configuration in an application override and in Spark submit parameters, the Spark submit parameters take precedence. The complete configuration priority list follows, in order of highest priority to lowest priority.
-
Configuration supplied when creating
SparkSession
. -
Configuration supplied as part of
sparkSubmitParameters
using—conf
. -
Configuration provided as part of application overrides.
-
Optimized configurations chosen by Amazon EMR for the release.
-
Default open source configurations for the application.
To monitor job runs using Amazon CloudWatch or Amazon S3, you must provide the configuration details for CloudWatch. For more information, see Configure a job run to use S3 logs and Configure a job run to use Amazon CloudWatch Logs. If the S3 bucket or CloudWatch log group does not exist, then Amazon EMR creates it before uploading logs to the bucket.
-
-
For an additional list of Kubernetes configuration options, see Spark Properties on Kubernetes
. The following Spark configurations are not supported.
-
spark.kubernetes.authenticate.driver.serviceAccountName
-
spark.kubernetes.authenticate.executor.serviceAccountName
-
spark.kubernetes.namespace
-
spark.kubernetes.driver.pod.name
-
spark.kubernetes.container.image.pullPolicy
-
spark.kubernetes.container.image
Note
You can use
spark.kubernetes.container.image
for customized Docker images. For more information, see Customizing Docker images for Amazon EMR on EKS.
-
Configure a job run to use S3 logs
To be able to monitor the job progress and to troubleshoot failures, you must configure your jobs to send log information to Amazon S3, Amazon CloudWatch Logs, or both. This topic helps you get started publishing application logs to Amazon S3 on your jobs that are launched with Amazon EMR on EKS.
S3 logs IAM policy
Before your jobs can send log data to Amazon S3, the following permissions must be included
in the permissions policy for the job execution role. Replace
DOC-EXAMPLE-BUCKET-LOGGING
with the name of your logging
bucket.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::
DOC-EXAMPLE-BUCKET-LOGGING
", "arn:aws:s3:::DOC-EXAMPLE-BUCKET-LOGGING
/*", ] } ] }
Note
Amazon EMR on EKS can also create an Amazon S3 bucket. If an Amazon S3 bucket is not available, include
the “s3:CreateBucket”
permission in the IAM policy.
After you have given your execution role the proper permissions to send logs to Amazon S3,
your log data are sent to the following Amazon S3 locations when
s3MonitoringConfiguration
is passed in the
monitoringConfiguration
section of a start-job-run
request, as
shown in Managing job runs with the AWS CLI.
-
Controller Logs - /
logUri
/virtual-cluster-id
/jobs/job-id
/containers/pod-name
/(stderr.gz/stdout.gz) -
Driver Logs - /
logUri
/virtual-cluster-id
/jobs/job-id
/containers/spark-application-id
/spark-job-id
-driver/(stderr.gz/stdout.gz) -
Executor Logs - /
logUri
/virtual-cluster-id
/jobs/job-id
/containers/spark-application-id
/executor-pod-name
/(stderr.gz/stdout.gz)
Configure a job run to use Amazon CloudWatch Logs
To monitor job progress and to troubleshoot failures, you must configure your jobs to send log information to Amazon S3, Amazon CloudWatch Logs, or both. This topic helps you get started using CloudWatch Logs on your jobs that are launched with Amazon EMR on EKS. For more information about CloudWatch Logs, see Monitoring Log Files in the Amazon CloudWatch User Guide.
CloudWatch Logs IAM policy
For your jobs to send log data to CloudWatch Logs, the following permissions must be included in
the permissions policy for the job execution role. Replace
my_log_group_name
and
my_log_stream_prefix
with names of your CloudWatch log group and log
stream names, respectively. Amazon EMR on EKS creates the log group and log stream if they do not
exist as long as the execution role ARN has appropriate permissions.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogStream", "logs:DescribeLogGroups", "logs:DescribeLogStreams" ], "Resource": [ "arn:aws:logs:*:*:*" ] }, { "Effect": "Allow", "Action": [ "logs:PutLogEvents" ], "Resource": [ "arn:aws:logs:*:*:log-group:
my_log_group_name
:log-stream:my_log_stream_prefix
/*" ] } ] }
Note
Amazon EMR on EKS can also create a log stream. If a log stream does not exist, the IAM policy
should include the"logs:CreateLogGroup"
permission.
After you have given your execution role the proper permissions, your application sends
its log data to CloudWatch Logs when cloudWatchMonitoringConfiguration
is passed in the
monitoringConfiguration
section of a start-job-run
request, as
shown in Managing job runs with the AWS CLI.
In the StartJobRun
API, log_group_name
is the
log group name for CloudWatch, and log_stream_prefix
is the log stream
name prefix for CloudWatch. You can view and search these logs in the AWS Management Console.
-
Controller logs -
logGroup
/logStreamPrefix
/virtual-cluster-id
/jobs/job-id
/containers/pod-name
/(stderr/stdout) -
Driver logs -
logGroup
/logStreamPrefix
/virtual-cluster-id
/jobs/job-id
/containers/spark-application-id
/spark-job-id
-driver/(stderrstdout) -
Executor logs -
logGroup
/logStreamPrefix
/virtual-cluster-id
/jobs/job-id
/containers/spark-application-id
/executor-pod-name
/(stderr/stdout)
List job runs
You can run list-job-run
to show the states of job runs, as the following
example demonstrates.
aws emr-containers list-job-runs --virtual-cluster-id <cluster-id>
Describe a job run
You can run describe-job-run
to get more details about the job, such as job
state, state details, and job name, as the following example demonstrates.
aws emr-containers describe-job-run --virtual-cluster-id
cluster-id
--idjob-run-id
Cancel a job run
You can run cancel-job-run
to cancel running jobs, as the following example
demonstrates.
aws emr-containers cancel-job-run --virtual-cluster-id
cluster-id
--idjob-run-id