Table Of Contents

Feedback

User Guide

First time using the AWS CLI? See the User Guide for help getting started.

Note: You are viewing the documentation for an older major version of the AWS CLI (version 1).

AWS CLI version 2, the latest major version of AWS CLI, is now stable and recommended for general use. To view this page for the AWS CLI version 2, click here. For more information see the AWS CLI version 2 installation instructions and migration guide.

[ aws . emr ]

add-steps

Description

Add a list of steps to a cluster.

See 'aws help' for descriptions of global parameters.

Synopsis

  add-steps
--cluster-id <value>
--steps <value> [<value>...]

Options

--cluster-id (string)

A unique string that identifies a cluster. The create-cluster command returns this identifier. You can use the list-clusters command to get cluster IDs.

--steps (list)

Specifies a list of steps to be executed by the cluster. Steps run only on the master node after applications are installed and are used to submit work to a cluster. A step can be specified using the shorthand syntax, by referencing a JSON file or by specifying an inline JSON structure. Args supplied with steps should be a comma-separated list of values (Args=arg1,arg2,arg3 ) or a bracket-enclosed list of values and key-value pairs (Args=[arg1,arg2=value,arg4 ).

Shorthand Syntax:

Name=string,Args=string,string,Jar=string,ActionOnFailure=string,MainClass=string,Type=string,Properties=string ...

JSON Syntax:

[
  {
    "Name": "string",
    "Args": ["string", ...],
    "Jar": "string",
    "ActionOnFailure": "TERMINATE_CLUSTER"|"CANCEL_AND_WAIT"|"CONTINUE",
    "MainClass": "string",
    "Type": "CUSTOM_JAR"|"STREAMING"|"HIVE"|"PIG"|"IMPALA",
    "Properties": "string"
  }
  ...
]

See 'aws help' for descriptions of global parameters.

Examples

1. To add Custom JAR steps to a cluster

  • Command:

    aws emr add-steps --cluster-id j-XXXXXXXX --steps Type=CUSTOM_JAR,Name=CustomJAR,ActionOnFailure=CONTINUE,Jar=s3://mybucket/mytest.jar,Args=arg1,arg2,arg3 Type=CUSTOM_JAR,Name=CustomJAR,ActionOnFailure=CONTINUE,Jar=s3://mybucket/mytest.jar,MainClass=mymainclass,Args=arg1,arg2,arg3
  • Required parameters:

    Jar
    
  • Optional parameters:

    Type, Name, ActionOnFailure, Args
    
  • Output:

    {
        "StepIds":[
            "s-XXXXXXXX",
            "s-YYYYYYYY"
        ]
    }
    

2. To add Streaming steps to a cluster

  • Command:

    aws emr add-steps --cluster-id j-XXXXXXXX --steps Type=STREAMING,Name='Streaming Program',ActionOnFailure=CONTINUE,Args=[-files,s3://elasticmapreduce/samples/wordcount/wordSplitter.py,-mapper,wordSplitter.py,-reducer,aggregate,-input,s3://elasticmapreduce/samples/wordcount/input,-output,s3://mybucket/wordcount/output]
  • Required parameters:

    Type, Args
    
  • Optional parameters:

    Name, ActionOnFailure
    
  • JSON equivalent (contents of step.json):

     [
      {
        "Name": "JSON Streaming Step",
        "Args": ["-files","s3://elasticmapreduce/samples/wordcount/wordSplitter.py","-mapper","wordSplitter.py","-reducer","aggregate","-input","s3://elasticmapreduce/samples/wordcount/input","-output","s3://mybucket/wordcount/output"],
        "ActionOnFailure": "CONTINUE",
        "Type": "STREAMING"
      }
    ]

NOTE: JSON arguments must include options and values as their own items in the list.

  • Command (using step.json):

    aws emr add-steps --cluster-id j-XXXXXXXX --steps file://./step.json
  • Output:

    {
        "StepIds":[
            "s-XXXXXXXX",
            "s-YYYYYYYY"
        ]
    }
    

3. To add a Streaming step with multiple files to a cluster (JSON only)

  • JSON (multiplefiles.json):

    [
      {
         "Name": "JSON Streaming Step",
         "Type": "STREAMING",
         "ActionOnFailure": "CONTINUE",
         "Args": [
             "-files",
             "s3://mybucket/mapper.py,s3://mybucket/reducer.py",
             "-mapper",
             "mapper.py",
             "-reducer",
             "reducer.py",
             "-input",
             "s3://mybucket/input",
             "-output",
             "s3://mybucket/output"]
      }
    ]
    
  • Command:

    aws emr add-steps --cluster-id j-XXXXXXXX  --steps file://./multiplefiles.json
  • Required parameters:

    Type, Args
    
  • Optional parameters:

    Name, ActionOnFailure
    
  • Output:

    {
        "StepIds":[
            "s-XXXXXXXX",
        ]
    }
    

4. To add Hive steps to a cluster

  • Command:

    aws emr add-steps --cluster-id j-XXXXXXXX --steps Type=HIVE,Name='Hive program',ActionOnFailure=CONTINUE,Args=[-f,s3://mybucket/myhivescript.q,-d,INPUT=s3://mybucket/myhiveinput,-d,OUTPUT=s3://mybucket/myhiveoutput,arg1,arg2] Type=HIVE,Name='Hive steps',ActionOnFailure=TERMINATE_CLUSTER,Args=[-f,s3://elasticmapreduce/samples/hive-ads/libs/model-build.q,-d,INPUT=s3://elasticmapreduce/samples/hive-ads/tables,-d,OUTPUT=s3://mybucket/hive-ads/output/2014-04-18/11-07-32,-d,LIBS=s3://elasticmapreduce/samples/hive-ads/libs]
  • Required parameters:

    Type, Args
    
  • Optional parameters:

    Name, ActionOnFailure
    
  • Output:

    {
        "StepIds":[
            "s-XXXXXXXX",
            "s-YYYYYYYY"
        ]
    }
    

5. To add Pig steps to a cluster

  • Command:

    aws emr add-steps --cluster-id j-XXXXXXXX --steps Type=PIG,Name='Pig program',ActionOnFailure=CONTINUE,Args=[-f,s3://mybucket/mypigscript.pig,-p,INPUT=s3://mybucket/mypiginput,-p,OUTPUT=s3://mybucket/mypigoutput,arg1,arg2] Type=PIG,Name='Pig program',Args=[-f,s3://elasticmapreduce/samples/pig-apache/do-reports2.pig,-p,INPUT=s3://elasticmapreduce/samples/pig-apache/input,-p,OUTPUT=s3://mybucket/pig-apache/output,arg1,arg2]
  • Required parameters:

    Type, Args
    
  • Optional parameters:

    Name, ActionOnFailure
    
  • Output:

    {
        "StepIds":[
            "s-XXXXXXXX",
            "s-YYYYYYYY"
        ]
    }
    

6. To add Impala steps to a cluster

  • Command:

    aws emr add-steps --cluster-id j-XXXXXXXX --steps Type=IMPALA,Name='Impala program',ActionOnFailure=CONTINUE,Args=--impala-script,s3://myimpala/input,--console-output-path,s3://myimpala/output
  • Required parameters:

    Type, Args
    
  • Optional parameters:

    Name, ActionOnFailure
    
  • Output:

    {
        "StepIds":[
            "s-XXXXXXXX",
            "s-YYYYYYYY"
        ]
    }