使用 StartJobRun 提交作業執行

使用具有指定參數JSON的檔案提交任務執行

1. 建立start-job-run-request.json檔案並指定任務執行所需的參數，如下列範例JSON檔案所示。如需這些參數的詳細資訊，請參閱 用於設定作業執行的選項。

   {
     "name": "myjob", 
     "virtualClusterId": "123456",  
     "executionRoleArn": "iam_role_name_for_job_execution", 
     "releaseLabel": "emr-6.2.0-latest", 
     "jobDriver": {
       "sparkSubmitJobDriver": {
         "entryPoint": "entryPoint_location",
         "entryPointArguments": ["argument1", "argument2", ...],  
          "sparkSubmitParameters": "--class <main_class> --conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.executor.cores=2 --conf spark.driver.cores=1"
       }
     }, 
     "configurationOverrides": {
       "applicationConfiguration": [
         {
           "classification": "spark-defaults", 
           "properties": {
             "spark.driver.memory":"2G"
            }
         }
       ], 
       "monitoringConfiguration": {
         "persistentAppUI": "ENABLED", 
         "cloudWatchMonitoringConfiguration": {
           "logGroupName": "my_log_group", 
           "logStreamNamePrefix": "log_stream_prefix"
         }, 
         "s3MonitoringConfiguration": {
           "logUri": "s3://my_s3_log_location"
         }
       }
     }
   }

2. 搭配使用 start-job-run 命令與儲存在本機的 start-job-run-request.json 檔案路徑。

   aws emr-containers start-job-run \
   --cli-input-json file://./start-job-run-request.json

使用 start-job-run 命令啟動作業執行

1. 請在 StartJobRun 命令中提供所有指定的參數，如下列範例所示。

   aws emr-containers start-job-run \
   --virtual-cluster-id 123456 \
   --name myjob \
   --execution-role-arn execution-role-arn \
   --release-label emr-6.2.0-latest \
   --job-driver '{"sparkSubmitJobDriver": {"entryPoint": "entryPoint_location", "entryPointArguments": ["argument1", "argument2", ...], "sparkSubmitParameters": "--class <main_class> --conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.executor.cores=2 --conf spark.driver.cores=1"}}' \
   --configuration-overrides '{"applicationConfiguration": [{"classification": "spark-defaults", "properties": {"spark.driver.memory": "2G"}}], "monitoringConfiguration": {"cloudWatchMonitoringConfiguration": {"logGroupName": "log_group_name", "logStreamNamePrefix": "log_stream_prefix"}, "persistentAppUI":"ENABLED",  "s3MonitoringConfiguration": {"logUri": "s3://my_s3_log_location" }}}'   
   

2. 對於 Spark SQL，請在StartJobRun命令中提供所有指定的參數，如下列範例所示。

   aws emr-containers start-job-run \
   --virtual-cluster-id 123456 \
   --name myjob \
   --execution-role-arn execution-role-arn \
   --release-label emr-6.7.0-latest \
   --job-driver '{"sparkSqlJobDriver": {"entryPoint": "entryPoint_location", "sparkSqlParameters": "--conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.executor.cores=2 --conf spark.driver.cores=1"}}' \
   --configuration-overrides '{"applicationConfiguration": [{"classification": "spark-defaults", "properties": {"spark.driver.memory": "2G"}}], "monitoringConfiguration": {"cloudWatchMonitoringConfiguration": {"logGroupName": "log_group_name", "logStreamNamePrefix": "log_stream_prefix"}, "persistentAppUI":"ENABLED",  "s3MonitoringConfiguration": {"logUri": "s3://my_s3_log_location" }}}'