View a markdown version of this page

Amazon EMR examples using AWS CLI - AWS SDK Code Examples
Actions

There are more AWS SDK examples available in the AWS Doc SDK Examples GitHub repo.

Amazon EMR examples using AWS CLI

The following code examples show you how to perform actions and implement common scenarios by using the AWS Command Line Interface with Amazon EMR.

Actions are code excerpts from larger programs and must be run in context. While actions show you how to call individual service functions, you can see actions in context in their related scenarios.

Each example includes a link to the complete source code, where you can find instructions on how to set up and run the code in context.

Topics

Actions

The following code example shows how to use add-instance-fleet.

AWS CLI

To add a task instance fleet to a cluster

The following add-instance-fleet example adds a new task instance fleet to the cluster specified.

aws emr add-instance-fleet \
    --cluster-id 'j-12ABCDEFGHI34JK' \
    --instance-fleet InstanceFleetType=TASK,TargetSpotCapacity=1,LaunchSpecifications={SpotSpecification='{TimeoutDurationMinutes=20,TimeoutAction=TERMINATE_CLUSTER}'},InstanceTypeConfigs=['{InstanceType=m3.xlarge,BidPrice=0.5}']

Output:

{
    "ClusterId": "j-12ABCDEFGHI34JK",
    "InstanceFleetId": "if-23ABCDEFGHI45JJ"
}

For more information, see Planning and configuring instance fleets for your Amazon EMR cluster in the Amazon EMR Management Guide.

The following code example shows how to use add-steps.

AWS CLI

Example 1: To add Custom JAR steps to a cluster

The following add-steps example adds two Custom JAR steps to the specified cluster.

aws emr add-steps \
    --cluster-id j-XXXXXXXX \
    --steps Type=CUSTOM_JAR,Name=CustomJAR,ActionOnFailure=CONTINUE,Jar=s3://amzn-s3-demo-bucket/mytest.jar,Args=arg1,arg2,arg3 Type=CUSTOM_JAR,Name=CustomJAR,ActionOnFailure=CONTINUE,Jar=s3://amzn-s3-demo-bucket/mytest.jar,MainClass=mymainclass,Args=arg1,arg2,arg3

Required parameters:

Jar

Optional parameters:

Type, Name, ActionOnFailure, Args

Output:

{
    "StepIds":[
        "s-XXXXXXXX",
        "s-YYYYYYYY"
    ]
}

Example 2: To add Streaming steps to a cluster

The following add-steps example adds a Streaming step to the specified cluster.

aws emr add-steps \
    --cluster-id j-XXXXXXXX \
    --steps Type=STREAMING,Name='Streaming Program',ActionOnFailure=CONTINUE,Args=[-files,s3://elasticmapreduce/samples/wordcount/wordSplitter.py,-mapper,wordSplitter.py,-reducer,aggregate,-input,s3://elasticmapreduce/samples/wordcount/input,-output,s3://amzn-s3-demo-bucket/wordcount/output]

Required parameters:

Type, Args

Optional parameters:

Name, ActionOnFailure

The following add-steps example adds the Streaming step that is defined in the step.json file.

aws emr add-steps \
    --cluster-id j-XXXXXXXX \
    --steps file://./step.json

Contents of step.json:

[
    {
        "Name": "JSON Streaming Step",
        "Args": ["-files","s3://elasticmapreduce/samples/wordcount/wordSplitter.py","-mapper","wordSplitter.py","-reducer","aggregate","-input","s3://elasticmapreduce/samples/wordcount/input","-output","s3://amzn-s3-demo-bucket/wordcount/output"],
        "ActionOnFailure": "CONTINUE",
        "Type": "STREAMING"
    }
]

NOTE: JSON arguments must include options and values as their own items in the list.

Output:

{
    "StepIds":[
        "s-XXXXXXXX",
        "s-YYYYYYYY"
    ]
}

Example 3: To add a Streaming step with multiple files to a cluster (JSON only)

The following add-steps example adds a Streaming step with multiple files to the specified cluster. The step is defined in the multiplefiles.json file.

The following command adds the step that is defined in the multiplefiles.json file.

aws emr add-steps \
    --cluster-id j-XXXXXXXX \
    --steps file://./multiplefiles.json

Contents of multiplefiles.json:

[
    {
        "Name": "JSON Streaming Step",
        "Type": "STREAMING",
        "ActionOnFailure": "CONTINUE",
        "Args": [
            "-files",
            "s3://amzn-s3-demo-bucket/mapper.py,s3://amzn-s3-demo-bucket/reducer.py",
            "-mapper",
            "mapper.py",
            "-reducer",
            "reducer.py",
            "-input",
            "s3://amzn-s3-demo-bucket/input",
            "-output",
            "s3://amzn-s3-demo-bucket/output"
        ]
    }
]

Required parameters:

Type, Args

Optional parameters:

Name, ActionOnFailure

Output:

{
    "StepIds":[
        "s-XXXXXXXX"
    ]
}

Example 4: To add Hive steps to a cluster

The following add-steps example adds two Hive steps to the specified cluster.

aws emr add-steps \
    --cluster-id j-XXXXXXXX \
    --steps Type=HIVE,Name='Hive program',ActionOnFailure=CONTINUE,Args=[-f,s3://amzn-s3-demo-bucket/myhivescript.q,-d,INPUT=s3://amzn-s3-demo-bucket/myhiveinput,-d,OUTPUT=s3://amzn-s3-demo-bucket/myhiveoutput,arg1,arg2] Type=HIVE,Name='Hive steps',ActionOnFailure=TERMINATE_CLUSTER,Args=[-f,s3://elasticmapreduce/samples/hive-ads/libs/model-build.q,-d,INPUT=s3://elasticmapreduce/samples/hive-ads/tables,-d,OUTPUT=s3://amzn-s3-demo-bucket/hive-ads/output/2014-04-18/11-07-32,-d,LIBS=s3://elasticmapreduce/samples/hive-ads/libs]

Required parameters:

Type, Args

Optional parameters:

Name, ActionOnFailure

Output:

{
    "StepIds":[
        "s-XXXXXXXX",
        "s-YYYYYYYY"
    ]
}

Example 5: To add Pig steps to a cluster

The following add-steps example adds two Pig steps to the specified cluster.

aws emr add-steps \
    --cluster-id j-XXXXXXXX \
    --steps Type=PIG,Name='Pig program',ActionOnFailure=CONTINUE,Args=[-f,s3://amzn-s3-demo-bucket/mypigscript.pig,-p,INPUT=s3://amzn-s3-demo-bucket/mypiginput,-p,OUTPUT=s3://amzn-s3-demo-bucket/mypigoutput,arg1,arg2] Type=PIG,Name='Pig program',Args=[-f,s3://elasticmapreduce/samples/pig-apache/do-reports2.pig,-p,INPUT=s3://elasticmapreduce/samples/pig-apache/input,-p,OUTPUT=s3://amzn-s3-demo-bucket/pig-apache/output,arg1,arg2]

Required parameters:

Type, Args

Optional parameters:

Name, ActionOnFailure

Output:

{
    "StepIds":[
        "s-XXXXXXXX",
        "s-YYYYYYYY"
    ]
}

Example 6: To add Impala steps to a cluster

The following add-steps example adds an Impala step to the specified cluster.

aws emr add-steps \
    --cluster-id j-XXXXXXXX \
    --steps Type=IMPALA,Name='Impala program',ActionOnFailure=CONTINUE,Args=--impala-script,s3://amzn-s3-demo-bucket/input,--console-output-path,s3://amzn-s3-demo-bucket/output

Required parameters:

Type, Args

Optional parameters:

Name, ActionOnFailure

Output:

{
    "StepIds":[
        "s-XXXXXXXX",
        "s-YYYYYYYY"
    ]
}

For more information, see Submit work to an Amazon EMR cluster in the Amazon EMR Management Guide.

  • For API details, see AddSteps in AWS CLI Command Reference.

The following code example shows how to use add-tags.

AWS CLI

To add tags to a cluster

The following add-tags example adds tags to the specified cluster.

aws emr add-tags \
    --resource-id j-xxxxxxx \
    --tags name="John Doe" age=29 sex=male address="123 East NW Seattle"

This command produces no output.

To list the tags of the cluster, run the following describe-cluster command.

aws emr describe-cluster \
    --cluster-id j-xxxxxxx \
    --query Cluster.Tags

Output:

[
    {
        "Value": "male",
        "Key": "sex"
    },
    {
        "Value": "123 East NW Seattle",
        "Key": "address"
    },
    {
        "Value": "John Doe",
        "Key": "name"
    },
    {
        "Value": "29",
        "Key": "age"
    }
]

For more information, see Tag and categorize Amazon EMR cluster resources in the Amazon EMR Management Guide.

  • For API details, see AddTags in AWS CLI Command Reference.

The following code example shows how to use create-cluster.

AWS CLI

Example 1: To create a cluster

The following create-cluster example creates a simple EMR cluster.

aws emr create-cluster \
    --release-label emr-5.14.0 \
    --instance-type m4.large \
    --instance-count 2

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

This example and most of the following examples assume that you specified your Amazon EMR service role and Amazon EC2 instance profile. If you have not done this, you must specify each required IAM role or use the --use-default-roles parameter when creating your cluster. For more information about specifying IAM roles, see Configure IAM Roles for Amazon EMR Permissions to AWS Services in the Amazon EMR Management Guide.

Example 2: To create an Amazon EMR cluster with default ServiceRole and InstanceProfile roles

The following create-cluster example creates an Amazon EMR cluster that uses the --instance-groups configuration.

aws emr create-cluster \
    --release-label emr-5.14.0 \
    --service-role EMR_DefaultRole \
    --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 3: To create an Amazon EMR cluster that uses an instance fleet

The following create-cluster example creates an Amazon EMR cluster that uses the --instance-fleets configuration, specifying two instance types for each fleet and two EC2 Subnets.

aws emr create-cluster \
    --release-label emr-5.14.0 \
    --service-role EMR_DefaultRole \
    --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole,SubnetIds=['subnet-ab12345c','subnet-de67890f'] \
    --instance-fleets InstanceFleetType=MASTER,TargetOnDemandCapacity=1,InstanceTypeConfigs=['{InstanceType=m4.large}'] InstanceFleetType=CORE,TargetSpotCapacity=11,InstanceTypeConfigs=['{InstanceType=m4.large,BidPrice=0.5,WeightedCapacity=3}','{InstanceType=m4.2xlarge,BidPrice=0.9,WeightedCapacity=5}'],LaunchSpecifications={SpotSpecification='{TimeoutDurationMinutes=120,TimeoutAction=SWITCH_TO_ON_DEMAND}'}

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 4: To create a cluster with default roles

The following create-cluster example uses the --use-default-roles parameter to specify the default service role and instance profile.

aws emr create-cluster \
    --release-label emr-5.9.0 \
    --use-default-roles \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
    --auto-terminate

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 5: To create a cluster and specify the applications to install

The following create-cluster example uses the --applications parameter to specify the applications that Amazon EMR installs. This example installs Hadoop, Hive and Pig.

aws emr create-cluster \
    --applications Name=Hadoop Name=Hive Name=Pig \
    --release-label emr-5.9.0 \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
    --auto-terminate

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 6: To create a cluster that includes Spark

The following create-cluster example installs Spark.

aws emr create-cluster \
    --release-label emr-5.9.0 \
    --applications Name=Spark \
    --ec2-attributes KeyName=myKey \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
    --auto-terminate

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 7: To specify a custom AMI to use for cluster instances

The following create-cluster example creates a cluster instance based on the Amazon Linux AMI with ID ami-a518e6df.

aws emr create-cluster \
    --name "Cluster with My Custom AMI" \
    --custom-ami-id ami-a518e6df \
    --ebs-root-volume-size 20 \
    --release-label emr-5.9.0 \
    --use-default-roles \
    --instance-count 2 \
    --instance-type m4.large

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 8: To customize application configurations

The following create-cluster examples use the --configurations parameter to specify a JSON configuration file that contains application customizations for Hadoop. For more information, see Configuring Applications in the Amazon EMR Release Guide.

Contents of configurations.json:

[
    {
        "Classification": "mapred-site",
        "Properties": {
            "mapred.tasktracker.map.tasks.maximum": 2
        }
    },
    {
        "Classification": "hadoop-env",
        "Properties": {},
        "Configurations": [
            {
                "Classification": "export",
                "Properties": {
                    "HADOOP_DATANODE_HEAPSIZE": 2048,
                    "HADOOP_NAMENODE_OPTS": "-XX:GCTimeRatio=19"
                }
            }
        ]
    }
]

The following example references configurations.json as a local file.

aws emr create-cluster \
    --configurations file://configurations.json \
    --release-label emr-5.9.0 \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
    --auto-terminate

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following example references configurations.json as a file in Amazon S3.

aws emr create-cluster \
    --configurations https://s3.amazonaws.com/amzn-s3-demo-bucket/configurations.json \
    --release-label emr-5.9.0 \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
    --auto-terminate

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 9: To create a cluster with master, core, and task instance groups

The following create-cluster example uses --instance-groups to specify the type and number of EC2 instances to use for master, core, and task instance groups.

aws emr create-cluster \
    --release-label emr-5.9.0 \
    --instance-groups Name=Master,InstanceGroupType=MASTER,InstanceType=m4.large,InstanceCount=1 Name=Core,InstanceGroupType=CORE,InstanceType=m4.large,InstanceCount=2 Name=Task,InstanceGroupType=TASK,InstanceType=m4.large,InstanceCount=2

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 10: To specify that a cluster should terminate after completing all steps

The following create-cluster example uses --auto-terminate to specify that the cluster should shut down automatically after completing all steps.

aws emr create-cluster \
    --release-label emr-5.9.0 \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large  InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
    --auto-terminate

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 11: To specify cluster configuration details such as the Amazon EC2 key pair, network configuration, and security groups

The following create-cluster example creates a cluster with the Amazon EC2 key pair named myKey and a customized instance profile named myProfile. Key pairs are used to authorize SSH connections to cluster nodes, most often the master node. For more information, see Use an Amazon EC2 Key Pair for SSH Credentials in the Amazon EMR Management Guide.

aws emr create-cluster \
    --ec2-attributes KeyName=myKey,InstanceProfile=myProfile \
    --release-label emr-5.9.0 \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
    --auto-terminate

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following example creates a cluster in an Amazon VPC subnet.

aws emr create-cluster \
    --ec2-attributes SubnetId=subnet-xxxxx \
    --release-label emr-5.9.0 \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
    --auto-terminate

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following example creates a cluster in the us-west-2b availability zone.

aws emr create-cluster \
    --ec2-attributes AvailabilityZone=us-west-2b \
    --release-label emr-5.9.0 \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following example creates a cluster and specifies only the Amazon EMR-managed security groups.

aws emr create-cluster \
    --release-label emr-5.9.0 \
    --service-role myServiceRole \
    --ec2-attributes InstanceProfile=myRole,EmrManagedMasterSecurityGroup=sg-master1,EmrManagedSlaveSecurityGroup=sg-slave1 \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following example creates a cluster and specifies only additional Amazon EC2 security groups.

aws emr create-cluster \
    --release-label emr-5.9.0 \
    --service-role myServiceRole \
    --ec2-attributes InstanceProfile=myRole,AdditionalMasterSecurityGroups=[sg-addMaster1,sg-addMaster2,sg-addMaster3,sg-addMaster4],AdditionalSlaveSecurityGroups=[sg-addSlave1,sg-addSlave2,sg-addSlave3,sg-addSlave4] \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following example creates a cluster and specifies the EMR-Managed security groups, as well as additional security groups.

aws emr create-cluster \
    --release-label emr-5.9.0 \
    --service-role myServiceRole \
    --ec2-attributes InstanceProfile=myRole,EmrManagedMasterSecurityGroup=sg-master1,EmrManagedSlaveSecurityGroup=sg-slave1,AdditionalMasterSecurityGroups=[sg-addMaster1,sg-addMaster2,sg-addMaster3,sg-addMaster4],AdditionalSlaveSecurityGroups=[sg-addSlave1,sg-addSlave2,sg-addSlave3,sg-addSlave4] \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following example creates a cluster in a VPC private subnet and use a specific Amazon EC2 security group to enable Amazon EMR service access, which is required for clusters in private subnets.

aws emr create-cluster \
    --release-label emr-5.9.0 \
    --service-role myServiceRole \
    --ec2-attributes InstanceProfile=myRole,ServiceAccessSecurityGroup=sg-service-access,EmrManagedMasterSecurityGroup=sg-master,EmrManagedSlaveSecurityGroup=sg-slave \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following example specifies security group configuration parameters using a JSON file named ec2_attributes.json that is stored locally. NOTE: JSON arguments must include options and values as their own items in the list.

aws emr create-cluster \
    --release-label emr-5.9.0 \
    --service-role myServiceRole \
    --ec2-attributes file://ec2_attributes.json  \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large

Contents of ec2_attributes.json:

[
    {
        "SubnetId": "subnet-xxxxx",
        "KeyName": "myKey",
        "InstanceProfile":"myRole",
        "EmrManagedMasterSecurityGroup": "sg-master1",
        "EmrManagedSlaveSecurityGroup": "sg-slave1",
        "ServiceAccessSecurityGroup": "sg-service-access",
        "AdditionalMasterSecurityGroups": ["sg-addMaster1","sg-addMaster2","sg-addMaster3","sg-addMaster4"],
        "AdditionalSlaveSecurityGroups": ["sg-addSlave1","sg-addSlave2","sg-addSlave3","sg-addSlave4"]
    }
]

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 12: To enable debugging and specify a log URI

The following create-cluster example uses the --enable-debugging parameter, which allows you to view log files more easily using the debugging tool in the Amazon EMR console. The --log-uri parameter is required with --enable-debugging.

aws emr create-cluster \
    --enable-debugging \
    --log-uri s3://amzn-s3-demo-bucket/myLog \
    --release-label emr-5.9.0 \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
    --auto-terminate

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 13: To add tags when creating a cluster

The following create-cluster example uses the --tags parameter to create three tags for a cluster, one with the key name name and the value Shirley Rodriguez, a second with the key name age and the value 29, and a third tag with the key name department and the value Analytics. Tags are key-value pairs that help you identify and manage clusters.

aws emr create-cluster \
    --tags name="Shirley Rodriguez" age=29 department="Analytics" \
    --release-label emr-5.32.0 \
    --instance-type m5.xlarge \
    --instance-count 3 \
    --use-default-roles

Output:

{
    "ClusterId": "j-XXXXXXYY",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-XXXXXXYY"
}

The following example lists the tags applied to a cluster.

aws emr describe-cluster \
    --cluster-id j-XXXXXXYY \
    --query Cluster.Tags

Output:

[
    {
        "Key": "name",
        "Value": "Shirley Rodriguez"
    },
    {
        "Key": "age",
        "Value": "29"
    },
    {
        "Key": "department",
        "Value": "Analytics"
    }
]

Example 14: To use a security configuration that enables encryption and other security features

The following create-cluster example uses the --security-configuration parameter to specify a security configuration for an EMR cluster. You can use security configurations with Amazon EMR version 4.8.0 or later.

aws emr create-cluster \
    --instance-type m4.large \
    --release-label emr-5.9.0 \
    --security-configuration mySecurityConfiguration

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 15: To create a cluster with additional EBS storage volumes configured for the instance groups

The following create-cluster example creates a cluster with multiple EBS volumes attached to EC2 instances in the core instance group. When specifying additional EBS volumes, the following arguments are required: VolumeType, SizeInGB if EbsBlockDeviceConfigs is specified.

aws emr create-cluster \
    --release-label emr-5.9.0  \
    --use-default-roles \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=d2.xlarge 'InstanceGroupType=CORE,InstanceCount=2,InstanceType=d2.xlarge,EbsConfiguration={EbsOptimized=true,EbsBlockDeviceConfigs=[{VolumeSpecification={VolumeType=gp2,SizeInGB=100}},{VolumeSpecification={VolumeType=io1,SizeInGB=100,Iops=100},VolumesPerInstance=4}]}' \
    --auto-terminate

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following example creates a cluster with multiple EBS volumes attached to EC2 instances in the master instance group.

aws emr create-cluster \
    --release-label emr-5.9.0 \
    --use-default-roles \
    --instance-groups 'InstanceGroupType=MASTER, InstanceCount=1, InstanceType=d2.xlarge, EbsConfiguration={EbsOptimized=true, EbsBlockDeviceConfigs=[{VolumeSpecification={VolumeType=io1, SizeInGB=100, Iops=100}},{VolumeSpecification={VolumeType=standard,SizeInGB=50},VolumesPerInstance=3}]}' InstanceGroupType=CORE,InstanceCount=2,InstanceType=d2.xlarge \
    --auto-terminate

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 16: To create a cluster with an automatic scaling policy

The following create-cluster example creates a cluster with an automatic scaling policy. You can attach automatic scaling policies to core and task instance groups using Amazon EMR version 4.0 and later. The automatic scaling policy dynamically adds and removes EC2 instances in response to an Amazon CloudWatch metric. For more information, see Using Automatic Scaling in Amazon EMR <https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-automatic-scaling.html>`_ in the Amazon EMR Management Guide.

When attaching an automatic scaling policy, you must also specify the default role for automatic scaling using --auto-scaling-role EMR_AutoScaling_DefaultRole.

The following create-cluster example specifies the automatic scaling policy for the CORE instance group using the AutoScalingPolicy argument with an embedded JSON structure, which specifies the scaling policy configuration. Instance groups with an embedded JSON structure must have the entire collection of arguments enclosed in single quotes. Using single quotes is optional for instance groups without an embedded JSON structure.

aws emr create-cluster \
    --release-label emr-5.9.0 \
    --use-default-roles \
    --auto-scaling-role EMR_AutoScaling_DefaultRole \
    --instance-groups InstanceGroupType=MASTER,InstanceType=d2.xlarge,InstanceCount=1 'InstanceGroupType=CORE,InstanceType=d2.xlarge,InstanceCount=2,AutoScalingPolicy={Constraints={MinCapacity=1,MaxCapacity=5},Rules=[{Name=TestRule,Description=TestDescription,Action={Market=ON_DEMAND,SimpleScalingPolicyConfiguration={AdjustmentType=EXACT_CAPACITY,ScalingAdjustment=2}},Trigger={CloudWatchAlarmDefinition={ComparisonOperator=GREATER_THAN,EvaluationPeriods=5,MetricName=TestMetric,Namespace=EMR,Period=3,Statistic=MAXIMUM,Threshold=4.5,Unit=NONE,Dimensions=[{Key=TestKey,Value=TestValue}]}}}]}'

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following example uses a JSON file, instancegroupconfig.json, to specify the configuration of all instance groups in a cluster. The JSON file specifies the automatic scaling policy configuration for the core instance group.

aws emr create-cluster \
    --release-label emr-5.9.0 \
    --service-role EMR_DefaultRole \
    --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
    --instance-groups file://myfolder/instancegroupconfig.json \
    --auto-scaling-role EMR_AutoScaling_DefaultRole

Contents of instancegroupconfig.json:

[
    {
        "InstanceCount": 1,
        "Name": "MyMasterIG",
        "InstanceGroupType": "MASTER",
        "InstanceType": "m4.large"
    },
    {
        "InstanceCount": 2,
        "Name": "MyCoreIG",
        "InstanceGroupType": "CORE",
        "InstanceType": "m4.large",
        "AutoScalingPolicy": {
            "Constraints": {
                "MinCapacity": 2,
                "MaxCapacity": 10
            },
            "Rules": [
                {
                    "Name": "Default-scale-out",
                    "Description": "Replicates the default scale-out rule in the console for YARN memory.",
                    "Action": {
                        "SimpleScalingPolicyConfiguration": {
                            "AdjustmentType": "CHANGE_IN_CAPACITY",
                            "ScalingAdjustment": 1,
                            "CoolDown": 300
                        }
                    },
                    "Trigger": {
                        "CloudWatchAlarmDefinition": {
                            "ComparisonOperator": "LESS_THAN",
                            "EvaluationPeriods": 1,
                            "MetricName": "YARNMemoryAvailablePercentage",
                            "Namespace": "AWS/ElasticMapReduce",
                            "Period": 300,
                            "Threshold": 15,
                            "Statistic": "AVERAGE",
                            "Unit": "PERCENT",
                            "Dimensions": [
                                {
                                    "Key": "JobFlowId",
                                    "Value": "${emr.clusterId}"
                                }
                            ]
                        }
                    }
                }
            ]
        }
    }
]

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 17: To add custom JAR steps when creating a cluster

The following create-cluster example adds steps by specifying a JAR file stored in Amazon S3. Steps submit work to a cluster. The main function defined in the JAR file executes after EC2 instances are provisioned, any bootstrap actions have executed, and applications are installed. The steps are specified using Type=CUSTOM_JAR.

Custom JAR steps require the Jar= parameter, which specifies the path and file name of the JAR. Optional parameters are Type, Name, ActionOnFailure, Args, and MainClass. If main class is not specified, the JAR file should specify Main-Class in its manifest file.

aws emr create-cluster \
    --steps Type=CUSTOM_JAR,Name=CustomJAR,ActionOnFailure=CONTINUE,Jar=s3://amzn-s3-demo-bucket/mytest.jar,Args=arg1,arg2,arg3 Type=CUSTOM_JAR,Name=CustomJAR,ActionOnFailure=CONTINUE,Jar=s3://amzn-s3-demo-bucket/mytest.jar,MainClass=mymainclass,Args=arg1,arg2,arg3  \
    --release-label emr-5.3.1 \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
    --auto-terminate

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 18: To add streaming steps when creating a cluster

The following create-cluster examples add a streaming step to a cluster that terminates after all steps run. Streaming steps require parameters Type and Args. Streaming steps optional parameters are Name and ActionOnFailure.

The following example specifies the step inline.

aws emr create-cluster \
    --steps Type=STREAMING,Name='Streaming Program',ActionOnFailure=CONTINUE,Args=[-files,s3://elasticmapreduce/samples/wordcount/wordSplitter.py,-mapper,wordSplitter.py,-reducer,aggregate,-input,s3://elasticmapreduce/samples/wordcount/input,-output,s3://amzn-s3-demo-bucket/wordcount/output] \
    --release-label emr-5.3.1 \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
    --auto-terminate

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following example uses a locally stored JSON configuration file named multiplefiles.json. The JSON configuration specifies multiple files. To specify multiple files within a step, you must use a JSON configuration file to specify the step. JSON arguments must include options and values as their own items in the list.

aws emr create-cluster \
    --steps file://./multiplefiles.json \
    --release-label emr-5.9.0  \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
    --auto-terminate

Contents of multiplefiles.json:

[
    {
        "Name": "JSON Streaming Step",
        "Args": [
            "-files",
            "s3://elasticmapreduce/samples/wordcount/wordSplitter.py",
            "-mapper",
            "wordSplitter.py",
            "-reducer",
            "aggregate",
            "-input",
            "s3://elasticmapreduce/samples/wordcount/input",
            "-output",
            "s3://amzn-s3-demo-bucket/wordcount/output"
        ],
        "ActionOnFailure": "CONTINUE",
        "Type": "STREAMING"
    }
]

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 19: To add Hive steps when creating a cluster

The following create-cluster example adds Hive steps when creating a cluster. Hive steps require parameters Type and Args. Hive steps optional parameters are Name and ActionOnFailure.

aws emr create-cluster \
    --steps Type=HIVE,Name='Hive program',ActionOnFailure=CONTINUE,ActionOnFailure=TERMINATE_CLUSTER,Args=[-f,s3://elasticmapreduce/samples/hive-ads/libs/model-build.q,-d,INPUT=s3://elasticmapreduce/samples/hive-ads/tables,-d,OUTPUT=s3://amzn-s3-demo-bucket/hive-ads/output/2014-04-18/11-07-32,-d,LIBS=s3://elasticmapreduce/samples/hive-ads/libs] \
    --applications Name=Hive \
    --release-label emr-5.3.1 \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 20: To add Pig steps when creating a cluster

The following create-cluster example adds Pig steps when creating a cluster. Pig steps required parameters are Type and Args. Pig steps optional parameters are Name and ActionOnFailure.

aws emr create-cluster \
    --steps Type=PIG,Name='Pig program',ActionOnFailure=CONTINUE,Args=[-f,s3://elasticmapreduce/samples/pig-apache/do-reports2.pig,-p,INPUT=s3://elasticmapreduce/samples/pig-apache/input,-p,OUTPUT=s3://amzn-s3-demo-bucket/pig-apache/output] \
    --applications Name=Pig \
    --release-label emr-5.3.1 \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 21: To add bootstrap actions

The following create-cluster example runs two bootstrap actions defined as scripts that are stored in Amazon S3.

aws emr create-cluster \
    --bootstrap-actions Path=s3://amzn-s3-demo-bucket/myscript1,Name=BootstrapAction1,Args=[arg1,arg2] Path=s3://amzn-s3-demo-bucket/myscript2,Name=BootstrapAction2,Args=[arg1,arg2] \
    --release-label emr-5.3.1 \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
    --auto-terminate

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 22: To enable EMRFS consistent view and customize the RetryCount and RetryPeriod settings

The following create-cluster example specifies the retry count and retry period for EMRFS consistent view. The Consistent=true argument is required.

aws emr create-cluster \
    --instance-type m4.large \
    --release-label emr-5.9.0 \
    --emrfs Consistent=true,RetryCount=6,RetryPeriod=30

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following example specifies the same EMRFS configuration as the previous example, using a locally stored JSON configuration file named emrfsconfig.json.

aws emr create-cluster \
    --instance-type m4.large \
    --release-label emr-5.9.0 \
    --emrfs file://emrfsconfig.json

Contents of emrfsconfig.json:

{
    "Consistent": true,
    "RetryCount": 6,
    "RetryPeriod": 30
}

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 23: To create a cluster with Kerberos configured

The following create-cluster examples create a cluster using a security configuration with Kerberos enabled, and establishes Kerberos parameters for the cluster using --kerberos-attributes.

The following command specifies Kerberos attributes for the cluster inline.

aws emr create-cluster \
    --instance-type m3.xlarge \
    --release-label emr-5.10.0 \
    --service-role EMR_DefaultRole \
    --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
    --security-configuration mySecurityConfiguration \
    --kerberos-attributes Realm=EC2.INTERNAL,KdcAdminPassword=123,CrossRealmTrustPrincipalPassword=123

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following command specifies the same attributes, but references a locally stored JSON file named kerberos_attributes.json. In this example, the file is saved in the same directory where you run the command. You can also reference a configuration file saved in Amazon S3.

aws emr create-cluster \
    --instance-type m3.xlarge \
    --release-label emr-5.10.0 \
    --service-role EMR_DefaultRole \
    --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
    --security-configuration mySecurityConfiguration \
    --kerberos-attributes file://kerberos_attributes.json

Contents of kerberos_attributes.json:

{
    "Realm": "EC2.INTERNAL",
    "KdcAdminPassword": "123",
    "CrossRealmTrustPrincipalPassword": "123",
}

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following create-cluster example creates an Amazon EMR cluster that uses the --instance-groups configuration and has a managed scaling policy.

aws emr create-cluster \
    --release-label emr-5.30.0 \
    --service-role EMR_DefaultRole \
    --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
    --managed-scaling-policy ComputeLimits='{MinimumCapacityUnits=2,MaximumCapacityUnits=4,UnitType=Instances}'

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following create-cluster example creates an Amazon EMR cluster that uses the "--log-encryption-kms-key-id" to define KMS key ID utilized for Log encryption.

aws emr create-cluster \
    --release-label emr-5.30.0 \
    --log-uri s3://amzn-s3-demo-bucket/myLog \
    --log-encryption-kms-key-id arn:aws:kms:us-west-2:123456789012:key/a1b2c3d4-5678-90ab-cdef-EXAMPLE11111 \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following create-cluster example creates an Amazon EMR cluster that uses the "--placement-group-configs" configuration to place master nodes in a high-availability (HA) cluster within an EC2 placement group using SPREAD placement strategy.

aws emr create-cluster \
    --release-label emr-5.30.0 \
    --service-role EMR_DefaultRole \
    --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=3,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=1,InstanceType=m4.large \
    --placement-group-configs InstanceRole=MASTER

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following create-cluster example creates an Amazon EMR cluster that uses the "--auto-termination-policy" configuration to place an automatic idle termination threshold for the cluster.

aws emr create-cluster \
    --release-label emr-5.34.0 \
    --service-role EMR_DefaultRole \
    --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=1,InstanceType=m4.large \
    --auto-termination-policy IdleTimeout=100

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

The following create-cluster example creates an Amazon EMR cluster that uses the "--os-release-label" to define an Amazon Linux release for cluster launch.

aws emr create-cluster \
    --release-label emr-6.6.0 \
    --os-release-label 2.0.20220406.1 \
    --service-role EMR_DefaultRole \
    --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
    --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=1,InstanceType=m4.large

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

Example 24: To specify an EBS root volume attributes: size, iops and throughput for cluster instances created with EMR releases 6.15.0 and later

The following create-cluster example creates an Amazon EMR cluster that uses root volume attributes to configure root volumes specifications for the EC2 instances.

aws emr create-cluster \
    --name "Cluster with My Custom AMI" \
    --custom-ami-id ami-a518e6df \
    --ebs-root-volume-size 20 \
    --ebs-root-volume-iops 3000 \
    --ebs-root-volume-throughput 125 \
    --release-label emr-6.15.0 \
    --use-default-roles \
    --instance-count 2 \
    --instance-type m4.large

Output:

{
    "ClusterId": "j-1K48XXXXXXHCB",
    "ClusterArn": "arn:aws:elasticmapreduce:us-west-2:123456789012:cluster/j-1K48XXXXXXHCB"
}

  • For API details, see CreateCluster in AWS CLI Command Reference.

The following code example shows how to use create-default-roles.

AWS CLI

To create the default IAM role for EC2

The following create-default-roles example creates the default IAM role for EC2. If the role already exists, the command returns nothing.

aws emr create-default-roles

Output:

[
    {
        "RolePolicy": {
            "Version":"2012-10-17",
            "Statement": [
                {
                    "Action": [
                        "cloudwatch:*",
                        "dynamodb:*",
                        "ec2:Describe*",
                        "elasticmapreduce:Describe*",
                        "elasticmapreduce:ListBootstrapActions",
                        "elasticmapreduce:ListClusters",
                        "elasticmapreduce:ListInstanceGroups",
                        "elasticmapreduce:ListInstances",
                        "elasticmapreduce:ListSteps",
                        "kinesis:CreateStream",
                        "kinesis:DeleteStream",
                        "kinesis:DescribeStream",
                        "kinesis:GetRecords",
                        "kinesis:GetShardIterator",
                        "kinesis:MergeShards",
                        "kinesis:PutRecord",
                        "kinesis:SplitShard",
                        "rds:Describe*",
                        "s3:*",
                        "sdb:*",
                        "sns:*",
                        "sqs:*"
                    ],
                    "Resource": "*",
                    "Effect": "Allow"
                }
            ]
        },
        "Role": {
            "AssumeRolePolicyDocument": {
                "Version":"2012-10-17",
                "Statement": [
                    {
                        "Action": "sts:AssumeRole",
                        "Sid": "",
                        "Effect": "Allow",
                        "Principal": {
                            "Service": "ec2.amazonaws.com"
                        }
                    }
                ]
            },
            "RoleId": "AROAIQ5SIQUGL5KMYBJX6",
            "CreateDate": "2015-06-09T17:09:04.602Z",
            "RoleName": "EMR_EC2_DefaultRole",
            "Path": "/",
            "Arn": "arn:aws:iam::123456789012:role/EMR_EC2_DefaultRole"
        }
    },
    {
        "RolePolicy": {
            "Version":"2012-10-17",
            "Statement": [
                {
                    "Action": [
                        "ec2:AuthorizeSecurityGroupIngress",
                        "ec2:CancelSpotInstanceRequests",
                        "ec2:CreateSecurityGroup",
                        "ec2:CreateTags",
                        "ec2:DeleteTags",
                        "ec2:DescribeAvailabilityZones",
                        "ec2:DescribeAccountAttributes",
                        "ec2:DescribeInstances",
                        "ec2:DescribeInstanceStatus",
                        "ec2:DescribeKeyPairs",
                        "ec2:DescribePrefixLists",
                        "ec2:DescribeRouteTables",
                        "ec2:DescribeSecurityGroups",
                        "ec2:DescribeSpotInstanceRequests",
                        "ec2:DescribeSpotPriceHistory",
                        "ec2:DescribeSubnets",
                        "ec2:DescribeVpcAttribute",
                        "ec2:DescribeVpcEndpoints",
                        "ec2:DescribeVpcEndpointServices",
                        "ec2:DescribeVpcs",
                        "ec2:ModifyImageAttribute",
                        "ec2:ModifyInstanceAttribute",
                        "ec2:RequestSpotInstances",
                        "ec2:RunInstances",
                        "ec2:TerminateInstances",
                        "iam:GetRole",
                        "iam:GetRolePolicy",
                        "iam:ListInstanceProfiles",
                        "iam:ListRolePolicies",
                        "iam:PassRole",
                        "s3:CreateBucket",
                        "s3:Get*",
                        "s3:List*",
                        "sdb:BatchPutAttributes",
                        "sdb:Select",
                        "sqs:CreateQueue",
                        "sqs:Delete*",
                        "sqs:GetQueue*",
                        "sqs:ReceiveMessage"
                    ],
                    "Resource": "*",
                    "Effect": "Allow"
                }
            ]
        },
        "Role": {
            "AssumeRolePolicyDocument": {
                "Version":"2012-10-17",
                "Statement": [
                    {
                        "Action": "sts:AssumeRole",
                        "Sid": "",
                        "Effect": "Allow",
                        "Principal": {
                            "Service": "elasticmapreduce.amazonaws.com"
                        }
                    }
                ]
            },
            "RoleId": "AROAI3SRVPPVSRDLARBPY",
            "CreateDate": "2015-06-09T17:09:10.401Z",
            "RoleName": "EMR_DefaultRole",
            "Path": "/",
            "Arn": "arn:aws:iam::123456789012:role/EMR_DefaultRole"
        }
    }
]

For more information, see Configure IAM service roles for Amazon EMR permissions to AWS services and resources in the Amazon EMR Management Guide.

The following code example shows how to use create-security-configuration.

AWS CLI

Example 1: To create a security configuration with in-transit encryption enabled with PEM for certificate provider, and at-rest encryption enabled with SSE-S3 for S3 encryption and AWS-KMS for local disk key provider

The following create-security-configuration example creates a security configuration named MySecurityConfig with in-transit encryption enabled with PEM for certificate provider, and at-rest encryption enabled with SSE-S3 for S3 encryption and AWS-KMS for local disk key provider.

aws emr create-security-configuration \
    --name MySecurityConfig \
    --security-configuration '{
        "EncryptionConfiguration": {
            "EnableInTransitEncryption" : true,
            "EnableAtRestEncryption" : true,
            "InTransitEncryptionConfiguration" : {
                "TLSCertificateConfiguration" : {
                    "CertificateProviderType" : "PEM",
                    "S3Object" : "s3://amzn-s3-demo-bucket/artifacts/MyCerts.zip"
                }
            },
            "AtRestEncryptionConfiguration" : {
                "S3EncryptionConfiguration" : {
                    "EncryptionMode" : "SSE-S3"
                },
                "LocalDiskEncryptionConfiguration" : {
                    "EncryptionKeyProviderType" : "AwsKms",
                    "AwsKmsKey" : "arn:aws:kms:us-west-2:123456789012:key/a1b2c3d4-5678-90ab-cdef-EXAMPLE11111"
                }
            }
        }
    }'

Output:

{
    "CreationDateTime": "2016-09-16T20:08:09.129000-0400",
    "Name": "MySecurityConfig"
}

The following create-security-configuration example creates the same security configuration by using the security_configuration.json file.

aws emr create-security-configuration \
    --name "MySecurityConfig" \
    --security-configuration file://./security_configuration.json

Contents of security_configuration.json:

{
    "EncryptionConfiguration": {
        "EnableInTransitEncryption": true,
        "EnableAtRestEncryption": true,
        "InTransitEncryptionConfiguration": {
            "TLSCertificateConfiguration": {
                "CertificateProviderType": "PEM",
                "S3Object": "s3://amzn-s3-demo-bucket/artifacts/MyCerts.zip"
            }
        },
        "AtRestEncryptionConfiguration": {
            "S3EncryptionConfiguration": {
                "EncryptionMode": "SSE-S3"
            },
            "LocalDiskEncryptionConfiguration": {
                "EncryptionKeyProviderType": "AwsKms",
                "AwsKmsKey": "arn:aws:kms:us-west-2:123456789012:key/a1b2c3d4-5678-90ab-cdef-EXAMPLE11111"
            }
        }
    }
}

Output:

{
    "CreationDateTime": "2016-09-16T20:08:09.129000-0400",
    "Name": "MySecurityConfig"
}

Example 2: To create a security configuration with Kerberos enabled using cluster-dedicated KDC and cross-realm trust

The following create-security-configuration example creates a security configuration named MySecurityConfig with Kerberos enabled using cluster-dedicated KDC and cross-realm trust.

aws emr create-security-configuration \
    --name MySecurityConfig \
    --security-configuration '{
        "AuthenticationConfiguration": {
            "KerberosConfiguration": {
                "Provider": "ClusterDedicatedKdc",
                "ClusterDedicatedKdcConfiguration": {
                    "TicketLifetimeInHours": 24,
                    "CrossRealmTrustConfiguration": {
                        "Realm": "AD.DOMAIN.COM",
                        "Domain": "ad.domain.com",
                        "AdminServer": "ad.domain.com",
                        "KdcServer": "ad.domain.com"
                    }
                }
            }
        }
    }'

Output:

{
    "CreationDateTime": "2017-03-22T19:32:38.982000-0400",
    "Name": "MySecurityConfig"
}

The following create-security-configuration example creates the same security configuration by using the security_configuration.json file.

aws emr create-security-configuration \
    --name "MySecurityConfig" \
    --security-configuration file://./security_configuration.json

Contents of security_configuration.json:

{
    "AuthenticationConfiguration": {
        "KerberosConfiguration": {
            "Provider": "ClusterDedicatedKdc",
            "ClusterDedicatedKdcConfiguration": {
                "TicketLifetimeInHours": 24,
                "CrossRealmTrustConfiguration": {
                    "Realm": "AD.DOMAIN.COM",
                    "Domain": "ad.domain.com",
                    "AdminServer": "ad.domain.com",
                    "KdcServer": "ad.domain.com"
                }
            }
        }
    }
}

Output:

{
    "CreationDateTime": "2017-03-22T19:32:38.982000-0400",
    "Name": "MySecurityConfig"
}

For more information, see Create a security configuration with the Amazon EMR console or with the AWS CLI in the Amazon EMR Management Guide.

The following code example shows how to use delete-security-configuration.

AWS CLI

To delete a security configuration in the current region

The following delete-security-configuration example deletes the security configuration named MySecurityConfig in the current region.

aws emr delete-security-configuration \
    --name MySecurityConfig

This command produces no output.

For more information, see Use security configurations to set up Amazon EMR cluster security in the Amazon EMR Management Guide.

The following code example shows how to use describe-cluster.

AWS CLI

To describe a cluster

The following describe-cluster example describes the specified cluster.

aws emr describe-cluster \
    --cluster-id j-XXXXXXXX

For a release-label based uniform instance groups cluster, the output looks like the following.

Output:

{
    "Cluster": {
        "Status": {
            "Timeline": {
                "ReadyDateTime": "2015-07-09T16:51:15.199000-0400",
                "CreationDateTime": "2015-07-09T16:44:16.563000-0400"
            },
            "State": "WAITING",
            "StateChangeReason": {
                "Message": "Waiting for steps to run"
            }
        },
        "Ec2InstanceAttributes": {
            "ServiceAccessSecurityGroup": "sg-xxxxxxxx",
            "EmrManagedMasterSecurityGroup": "sg-xxxxxxxx",
            "IamInstanceProfile": "EMR_EC2_DefaultRole",
            "Ec2KeyName": "myKey",
            "Ec2AvailabilityZone": "us-west-2c",
            "EmrManagedSlaveSecurityGroup": "sg-yyyyyyyyy"
        },
        "Name": "My Cluster",
        "ServiceRole": "EMR_DefaultRole",
        "Tags": [],
        "TerminationProtected": true,
        "UnhealthyNodeReplacement": true,
        "ReleaseLabel": "emr-4.0.0",
        "NormalizedInstanceHours": 96,
        "InstanceGroups": [
            {
                "RequestedInstanceCount": 2,
                "Status": {
                    "Timeline": {
                        "ReadyDateTime": "2015-07-09T16:51:14.245000-0400",
                        "CreationDateTime": "2015-07-09T16:44:16.564000-0400",
                        "EndDateTime": "2015-07-11T14:09:18.387000-0400"
                    },
                    "State": "RUNNING",
                    "StateChangeReason": {
                        "Message": ""
                    }
                },
                "Name": "CORE",
                "InstanceGroupType": "CORE",
                "Id": "ig-YYYYYYY",
                "Configurations": [],
                "InstanceType": "m3.large",
                "Market": "ON_DEMAND",
                "RunningInstanceCount": 2
            },
            {
                "RequestedInstanceCount": 1,
                "Status": {
                    "Timeline": {
                        "ReadyDateTime": "2015-07-09T16:51:14.245000-0400",
                        "CreationDateTime": "2015-07-09T16:44:16.564000-0400",
                        "EndDateTime": "2015-07-11T14:09:18.387000-0400"
                    },
                    "State": "RUNNING",
                    "StateChangeReason": {
                        "Message": ""
                    }
                },
                "Name": "MASTER",
                "InstanceGroupType": "MASTER",
                "Id": "ig-XXXXXXXXX",
                "Configurations": [],
                "InstanceType": "m3.large",
                "Market": "ON_DEMAND",
                "RunningInstanceCount": 1
            }
        ],
        "Applications": [
            {
                "Name": "Hadoop"
            }
        ],
        "VisibleToAllUsers": true,
        "BootstrapActions": [],
        "MasterPublicDnsName": "ec2-54-147-144-78.compute-1.amazonaws.com",
        "AutoTerminate": false,
        "Id": "j-XXXXXXXX",
        "Configurations": [
            {
                "Properties": {
                    "fs.s3.consistent.retryPeriodSeconds": "20",
                    "fs.s3.enableServerSideEncryption": "true",
                    "fs.s3.consistent": "false",
                    "fs.s3.consistent.retryCount": "2"
                },
                "Classification": "emrfs-site"
            }
        ]
    }
}

For a release-label based instance fleet cluster, the output looks like the following.

Output:

{
    "Cluster": {
        "Status": {
            "Timeline": {
                "ReadyDateTime": "2017-02-23T19:48:09.705000-0500",
                "CreationDateTime": "2017-02-23T19:42:13.942000-0500"
            },
            "State": "WAITING",
            "StateChangeReason": {
                "Message": "Waiting for steps to run"
            }
        },
        "Ec2InstanceAttributes": {
            "EmrManagedMasterSecurityGroup": "sg-xxxxx",
            "RequestedEc2AvailabilityZones": [],
            "RequestedEc2SubnetIds": [],
            "IamInstanceProfile": "EMR_EC2_DefaultRole",
            "Ec2AvailabilityZone": "us-west-2a",
            "EmrManagedSlaveSecurityGroup": "sg-xxxxx"
        },
        "Name": "My Cluster",
        "ServiceRole": "EMR_DefaultRole",
        "Tags": [],
        "TerminationProtected": false,
        "UnhealthyNodeReplacement": false,
        "ReleaseLabel": "emr-5.2.0",
        "NormalizedInstanceHours": 472,
        "InstanceCollectionType": "INSTANCE_FLEET",
        "InstanceFleets": [
            {
                "Status": {
                    "Timeline": {
                        "ReadyDateTime": "2017-02-23T19:46:52.740000-0500",
                        "CreationDateTime": "2017-02-23T19:42:13.948000-0500"
                    },
                    "State": "RUNNING",
                    "StateChangeReason": {
                        "Message": ""
                    }
                },
                "ProvisionedSpotCapacity": 1,
                "Name": "MASTER",
                "InstanceFleetType": "MASTER",
                "LaunchSpecifications": {
                    "SpotSpecification": {
                        "TimeoutDurationMinutes": 60,
                        "TimeoutAction": "TERMINATE_CLUSTER"
                    }
                },
                "TargetSpotCapacity": 1,
                "ProvisionedOnDemandCapacity": 0,
                "InstanceTypeSpecifications": [
                    {
                        "BidPrice": "0.5",
                        "InstanceType": "m3.xlarge",
                        "WeightedCapacity": 1
                    }
                ],
                "Id": "if-xxxxxxx",
                "TargetOnDemandCapacity": 0
            }
        ],
        "Applications": [
            {
                "Version": "2.7.3",
                "Name": "Hadoop"
            }
        ],
        "ScaleDownBehavior": "TERMINATE_AT_INSTANCE_HOUR",
        "VisibleToAllUsers": true,
        "BootstrapActions": [],
        "MasterPublicDnsName": "ec2-xxx-xx-xxx-xx.compute-1.amazonaws.com",
        "AutoTerminate": false,
        "Id": "j-xxxxx",
        "Configurations": []
    }
}

For an AMI based uniform instance group cluster, the output looks like the following.

Output:

{
    "Cluster": {
        "Status": {
            "Timeline": {
                "ReadyDateTime": "2014-05-06T14:22:44.432000-0400",
                "CreationDateTime": "2014-05-06T14:17:48.62000-0400"
            },
            "State": "WAITING",
            "StateChangeReason": {
                "Message": "Waiting for steps to run"
            }
        },
        "Ec2InstanceAttributes": {
            "IamInstanceProfile": "EMR_EC2_DefaultRole",
            "Ec2AvailabilityZone": "us-west-2c"
        },
        "Name": "My Cluster",
        "Tags": [],
        "TerminationProtected": true,
        "UnhealthyNodeReplacement": true,
        "RunningAmiVersion": "2.5.4",
        "InstanceGroups": [
            {
                "RequestedInstanceCount": 1,
                "Status": {
                    "Timeline": {
                        "ReadyDateTime": "2014-05-06T14:22:38.848000-0400",
                        "CreationDateTime": "2014-05-06T14:17:48.621000-0400"
                    },
                    "State": "RUNNING",
                    "StateChangeReason": {
                        "Message": ""
                    }
                },
                "Name": "Master instance group",
                "InstanceGroupType": "MASTER",
                "InstanceType": "m1.small",
                "Id": "ig-ABCD",
                "Market": "ON_DEMAND",
                "RunningInstanceCount": 1
            },
            {
                "RequestedInstanceCount": 2,
                "Status": {
                    "Timeline": {
                        "ReadyDateTime": "2014-05-06T14:22:44.439000-0400",
                        "CreationDateTime": "2014-05-06T14:17:48.621000-0400"
                    },
                    "State": "RUNNING",
                    "StateChangeReason": {
                        "Message": ""
                    }
                },
                "Name": "Core instance group",
                "InstanceGroupType": "CORE",
                "InstanceType": "m1.small",
                "Id": "ig-DEF",
                "Market": "ON_DEMAND",
                "RunningInstanceCount": 2
            }
        ],
        "Applications": [
            {
                "Version": "1.0.3",
                "Name": "hadoop"
            }
        ],
        "BootstrapActions": [],
        "VisibleToAllUsers": false,
        "RequestedAmiVersion": "2.4.2",
        "LogUri": "s3://amzn-s3-demo-bucketLogUri/",
        "AutoTerminate": false,
        "Id": "j-XXXXXXXX"
    }
}

For more information, see View Amazon EMR cluster status and details in the Amazon EMR Management Guide.

The following code example shows how to use describe-step.

AWS CLI

To describe a step

The following describe-step example describes a step with the step ID s-3LZC0QUT43AM in a cluster with the cluster ID j-3SD91U2E1L2QX.

aws emr describe-step \
    --cluster-id j-3SD91U2E1L2QX \
    --step-id s-3LZC0QUT43AM

Output:

{
    "Step": {
        "Status": {
            "Timeline": {
                "EndDateTime": "2015-06-01T19:14:30.481000-0400",
                "CreationDateTime": "2015-06-01T19:05:26.597000-0400",
                "StartDateTime": "2015-06-01T19:13:24.959000-0400"
            },
            "State": "COMPLETED",
            "StateChangeReason": {}
        },
        "Config": {
            "Args": [
                "s3://us-west-2.elasticmapreduce/libs/hive/hive-script",
                "--base-path",
                "s3://us-west-2.elasticmapreduce/libs/hive/",
                "--install-hive",
                "--hive-versions",
                "0.13.1"
            ],
            "Jar": "s3://us-west-2.elasticmapreduce/libs/script-runner/script-runner.jar",
            "Properties": {}
        },
        "Id": "s-3LZC0QUT43AM",
        "ActionOnFailure": "TERMINATE_CLUSTER",
        "Name": "Setup hive"
    }
}

For more information, see Submit work to an Amazon EMR cluster in the Amazon EMR Management Guide.

  • For API details, see DescribeStep in AWS CLI Command Reference.

The following code example shows how to use get.

AWS CLI

To download a file from the primary node of a cluster

The following get example downloads the hadoop-examples.jar archive from the master instance in a cluster with the cluster ID j-3SD91U2E1L2QX. The command displays the scp command that it runs, followed by any output from scp.

aws emr get \
    --cluster-id j-3SD91U2E1L2QX \
    --key-pair-file ~/.ssh/mykey.pem \
    --src /home/hadoop-examples.jar \
    --dest ~

Output:

scp -r -o StrictHostKeyChecking=accept-new -i /home/local/user/.ssh/mykey.pem hadoop@ec2-52-52-41-150.us-west-2.compute.amazonaws.com:/home/hadoop-examples.jar /home/local/user

For more information, see Connect to the Amazon EMR cluster primary node using SSH in the Amazon EMR Management Guide.

  • For API details, see Get in AWS CLI Command Reference.

The following code example shows how to use list-clusters.

AWS CLI

To list the active clusters

The following list-clusters example lists all active EMR clusters in the current region.

aws emr list-clusters \
    --active

Output:

{
    "Clusters": [
        {
            "Status": {
                "Timeline": {
                    "ReadyDateTime": "2025-10-28T11:11:52.228000-04:00",
                    "CreationDateTime": "2025-10-28T11:02:22.179000-04:00"
                },
                "State": "WAITING",
                "StateChangeReason": {
                    "Message": "Waiting after step completed"
                }
            },
            "NormalizedInstanceHours": 6,
            "Id": "j-3SD91U2E1L2QX",
            "Name": "my-cluster"
        }
    ]
}

For more information, see View Amazon EMR cluster status and details in the Amazon EMR Management Guide.

  • For API details, see ListClusters in AWS CLI Command Reference.

The following code example shows how to use list-instance-fleets.

AWS CLI

To get configuration details of instance fleets in a cluster

The following list-instance-fleets example lists the details of instance fleets in the cluster specified.

list-instance-fleets --cluster-id 'j-12ABCDEFGHI34JK'

Output:

{
    "InstanceFleets": [
        {
            "Status": {
                "Timeline": {
                    "ReadyDateTime": "2017-03-05T19:11:34.637000-0500",
                    "CreationDateTime": "2017-03-05T19:05:19.817000-0500"
                },
                "State": "RUNNING",
                "StateChangeReason": {
                    "Message": ""
                }
            },
            "ProvisionedSpotCapacity": 6,
            "Name": "CORE",
            "InstanceFleetType": "CORE",
            "LaunchSpecifications": {
                "SpotSpecification": {
                    "TimeoutDurationMinutes": 60,
                    "TimeoutAction": "TERMINATE_CLUSTER"
                }
            },
            "ProvisionedOnDemandCapacity": 2,
            "InstanceTypeSpecifications": [
                {
                    "BidPrice": "0.5",
                    "InstanceType": "m3.xlarge",
                    "WeightedCapacity": 2
                }
            ],
            "Id": "if-1ABC2DEFGHIJ3"
        },
        {
            "Status": {
                "Timeline": {
                    "ReadyDateTime": "2017-03-05T19:10:58.598000-0500",
                    "CreationDateTime": "2017-03-05T19:05:19.811000-0500"
                },
                "State": "RUNNING",
                "StateChangeReason": {
                    "Message": ""
                }
            },
            "ProvisionedSpotCapacity": 0,
            "Name": "MASTER",
            "InstanceFleetType": "MASTER",
            "ProvisionedOnDemandCapacity": 1,
            "InstanceTypeSpecifications": [
                {
                    "BidPriceAsPercentageOfOnDemandPrice": 100.0,
                    "InstanceType": "m3.xlarge",
                    "WeightedCapacity": 1
                }
            ],
            "Id": "if-2ABC4DEFGHIJ4"
        }
    ]
}

For more information, see Planning and configuring instance fleets for your Amazon EMR cluster in the Amazon EMR Management Guide.

The following code example shows how to use list-instances.

AWS CLI

To list the instances in a cluster

The following list-instances example lists all of the instances in a cluster with the cluster ID j-3C6XNQ39VR9WL.

aws emr list-instances \
    --cluster-id j-3C6XNQ39VR9WL

For a uniform instance group based cluster, the output looks like the following.

Output:

{
    "Instances": [
        {
            "Status": {
                "Timeline": {
                    "ReadyDateTime": "2015-06-01T19:13:20.431000-0400",
                    "CreationDateTime": "2015-06-01T19:05:49.263000-0400"
                },
                "State": "RUNNING",
                "StateChangeReason": {}
            },
            "Ec2InstanceId": "i-f19ecfee",
            "PublicDnsName": "ec2-52-52-41-150.us-west-2.compute.amazonaws.com",
            "PrivateDnsName": "ip-172-21-11-216.us-west-2.compute.internal",
            "PublicIpAddress": "52.52.41.150",
            "Id": "ci-3NNHQUQ2TWB6Y",
            "PrivateIpAddress": "172.21.11.216"
        },
        {
            "Status": {
                "Timeline": {
                    "ReadyDateTime": "2015-06-01T19:13:20.431000-0400",
                    "CreationDateTime": "2015-06-01T19:05:49.263000-0400"
                },
                "State": "RUNNING",
                "StateChangeReason": {}
            },
            "Ec2InstanceId": "i-1feee4c2",
            "PublicDnsName": "ec2-52-63-246-32.us-west-2.compute.amazonaws.com",
            "PrivateDnsName": "ip-172-31-24-130.us-west-2.compute.internal",
            "PublicIpAddress": "52.63.246.32",
            "Id": "ci-GAOCMKNKDCV7",
            "PrivateIpAddress": "172.21.11.215"
        },
        {
            "Status": {
                "Timeline": {
                    "ReadyDateTime": "2015-06-01T19:13:20.431000-0400",
                    "CreationDateTime": "2015-06-01T19:05:49.263000-0400"
                },
                "State": "RUNNING",
                "StateChangeReason": {}
            },
            "Ec2InstanceId": "i-15cfeee3",
            "PublicDnsName": "ec2-52-25-246-63.us-west-2.compute.amazonaws.com",
            "PrivateDnsName": "ip-172-31-24-129.us-west-2.compute.internal",
            "PublicIpAddress": "52.25.246.63",
            "Id": "ci-2W3TDFFB47UAD",
            "PrivateIpAddress": "172.21.11.214"
        }
    ]
}

For an instance fleet based cluster, the output looks like the following.

Output:

{
    "Instances": [
        {
            "Status": {
                "Timeline": {
                    "ReadyDateTime": "2017-02-22T19:46:50.878000-0500",
                    "CreationDateTime": "2017-02-22T19:43:08.878000-0500",
                    "EndDateTime": "2017-02-25T06:43:10.924000-0500"
                },
                "State": "TERMINATED",
                "StateChangeReason": {
                    "Message": "Instance was terminated."
                }
            },
            "Ec2InstanceId": "i-xxxxx",
            "InstanceFleetId": "if-xxxxx",
            "EbsVolumes": [],
            "PublicDnsName": "ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com",
            "InstanceType": "m3.xlarge",
            "PrivateDnsName": "ip-xx-xx-xxx-xx.ec2.internal",
            "Market": "SPOT",
            "PublicIpAddress": "xx.xx.xxx.xxx",
            "Id": "ci-xxxxx",
            "PrivateIpAddress": "10.47.191.80"
        }
    ]
}

For more information, see View Amazon EMR cluster status and details in the Amazon EMR Management Guide.

  • For API details, see ListInstances in AWS CLI Command Reference.

The following code example shows how to use list-security-configurations.

AWS CLI

To list security configurations in the current region

The following list-security-configurations example lists the security configurations in the current region.

aws emr list-security-configurations

Output:

{
    "SecurityConfigurations": [
        {
            "CreationDateTime": "2025-10-28T10:15:36.207000-04:00",
            "Name": "MySecurityConfig-1"
        },
        {
            "CreationDateTime": "2025-10-28T10:15:36.207000-04:00",
            "Name": "MySecurityConfig-2"
        }
    ]
}

For more information, see Use security configurations to set up Amazon EMR cluster security in the Amazon EMR Management Guide.

The following code example shows how to use list-steps.

AWS CLI

To list the steps in a cluster

The following list-steps example lists all of the steps in a cluster with the cluster ID j-3SD91U2E1L2QX.

aws emr list-steps \
    --cluster-id j-3SD91U2E1L2QX

Output:

{
    "Steps": [
        {
            "Id": "s-2BRWVHXA3EXBT",
            "Name": "Custom JAR",
            "Config": {
                "Jar": "s3://amzn-s3-demo-bucket/mytest.jar",
                "Properties": {},
                "MainClass": "mymainclass",
                "Args": [
                    "arg1",
                    "arg2"
                ]
            },
            "ActionOnFailure": "CONTINUE",
            "Status": {
                "State": "COMPLETED",
                "StateChangeReason": {},
                "Timeline": {
                    "CreationDateTime": "2024-05-14T18:07:31.123000+00:00",
                    "StartDateTime": "2024-05-14T18:10:45.456000+00:00",
                    "EndDateTime": "2024-05-14T18:12:20.789000+00:00"
                }
            }
        },
        {
            "Id": "s-1AXF2SNQKPLZR",
            "Name": "Setup Hadoop Debugging",
            "Config": {
                "Jar": "command-runner.jar",
                "Properties": {},
                "Args": [
                    "state-pusher-script"
                ]
            },
            "ActionOnFailure": "TERMINATE_CLUSTER",
            "Status": {
                "State": "COMPLETED",
                "StateChangeReason": {},
                "Timeline": {
                    "CreationDateTime": "2024-05-14T18:01:02.345000+00:00",
                    "StartDateTime": "2024-05-14T18:09:30.678000+00:00",
                    "EndDateTime": "2024-05-14T18:10:40.901000+00:00"
                }
            }
        }
    ]
}

For more information, see Submit work to an Amazon EMR cluster in the Amazon EMR Management Guide.

  • For API details, see ListSteps in AWS CLI Command Reference.

The following code example shows how to use list-studios.

AWS CLI

To list available EMR Studios

The following list-studios example lists the EMR Studios in the AWS account.

aws emr list-studios

Output:

{
    "Studios": [
        {
            "StudioId": "es-XXXXXXX132E0X7R0W7GAS1MVB",
            "Name": "My_EMR_Studio",
            "Url": "https://es-XXXXXXX132E0X7R0W7GAS1MVB.emrstudio-prod.us-west-2.amazonaws.com",
            "AuthMode": "IAM",
            "CreationTime": "2025-10-28T11:09:33.624000-04:00"
        }
    ]
}

For more information, see Monitor, update and delete Amazon EMR Studio resources in the Amazon EMR Management Guide.

  • For API details, see ListStudios in AWS CLI Command Reference.

The following code example shows how to use modify-cluster-attributes.

AWS CLI

To make a cluster visible to all users

The following modify-cluster-attributes example sets the visibility of an EMR cluster with the ID j-301CDNY0J5XM4 to all users.

aws emr modify-cluster-attributes \
    --cluster-id j-301CDNY0J5XM4 \
    --visible-to-all-users

This command produces no output.

For more information, see AWS Identity and Access Management for Amazon EMR in the Amazon EMR Management Guide.

The following code example shows how to use modify-instance-fleet.

AWS CLI

To change the target capacities of an instance fleet

The following modify-instance-fleet example changes the On-Demand and Spot target capacities to 1 for the specified instance fleet.

aws emr modify-instance-fleet \
    --cluster-id 'j-12ABCDEFGHI34JK' \
    --instance-fleet InstanceFleetId='if-2ABC4DEFGHIJ4',TargetOnDemandCapacity=1,TargetSpotCapacity=1

This command produces no output.

For more information, see Planning and configuring instance fleets for your Amazon EMR cluster in the Amazon EMR Management Guide.

The following code example shows how to use put.

AWS CLI

To upload a file to the primary node of a cluster

The following put example uploads a file named healthcheck.sh to the master instance in a cluster with the cluster ID j-3SD91U2E1L2QX. The command displays the scp command that it runs, followed by any output from scp.

aws emr put \
    --cluster-id j-3SD91U2E1L2QX \
    --key-pair-file ~/.ssh/mykey.pem \
    --src ~/scripts/healthcheck.sh \
    --dest /home/hadoop/bin/healthcheck.sh

Output:

scp -r -o StrictHostKeyChecking=accept-new -i /home/local/user/.ssh/mykey.pem /home/local/user/scripts/healthcheck.sh hadoop@ec2-52-52-41-150.us-west-2.compute.amazonaws.com:/home/hadoop/bin/healthcheck.sh

For more information, see Connect to an Amazon EMR cluster in the Amazon EMR Management Guide.

  • For API details, see Put in AWS CLI Command Reference.

The following code example shows how to use remove-tags.

AWS CLI

To remove a tag from a cluster

The following remove-tags example removes a tag with the key prod from a cluster with the cluster ID j-3SD91U2E1L2QX.

aws emr remove-tags \
    --resource-id j-3SD91U2E1L2QX \
    --tag-keys prod

This command produces no output.

For more information, see Tag and categorize Amazon EMR cluster resources in the Amazon EMR Management Guide.

  • For API details, see RemoveTags in AWS CLI Command Reference.

The following code example shows how to use schedule-hbase-backup.

AWS CLI

Example 1: To schedule a full HBase backup

The following schedule-hbase-backup example schedules a full, consistent HBase backup to Amazon S3 every 10 hours. This command can only be used with HBase on AMI version 2.x and 3.x. The command adds an HBase backup step to the cluster and returns the ID of the step.

aws emr schedule-hbase-backup \
    --cluster-id j-XXXXXXYY \
    --type full \
    --dir s3://amzn-s3-demo-bucket/backup \
    --interval 10 \
    --unit hours \
    --start-time 2014-04-21T05:26:10Z \
    --consistent

Output:

{
    "StepIds": [
        "s-1A2B3C4D5E6F7"
    ]
}

Example 2: To schedule an incremental HBase backup

The following schedule-hbase-backup example schedules an incremental, consistent HBase backup to Amazon S3 every 30 minutes. This command can only be used with HBase on AMI version 2.x and 3.x. The command adds an HBase backup step to the cluster and returns the ID of the step.

aws emr schedule-hbase-backup \
    --cluster-id j-XXXXXXYY \
    --type incremental \
    --dir s3://amzn-s3-demo-bucket/backup \
    --interval 30 \
    --unit minutes \
    --start-time 2014-04-21T05:26:10Z \
    --consistent

Output:

{
    "StepIds": [
        "s-2B3C4D5E6F7G8"
    ]
}

For more information, see HBase application specifics for earlier AMI versions of Amazon EMR in the Amazon EMR Release Guide.

The following code example shows how to use socks.

AWS CLI

To open a SOCKS connection to the primary node of a cluster

The following socks example opens a socks connection with the master instance in a cluster with the cluster ID j-3SD91U2E1L2QX.

aws emr socks \
    --cluster-id j-3SD91U2E1L2QX \
    --key-pair-file ~/.ssh/mykey.pem

The key pair file option takes a local path to a private key file. The command displays the ssh command that it runs and keeps the tunnel open on local port 8157 until you press Ctrl+C.

Output:

ssh -o StrictHostKeyChecking=accept-new -o ServerAliveInterval=10 -ND 8157 -i /home/local/user/.ssh/mykey.pem hadoop@ec2-52-52-41-150.us-west-2.compute.amazonaws.com
Disabling Socks Tunnel.

For more information, see Option 2, part 1: Set up an SSH tunnel to the primary node using dynamic port forwarding in the Amazon EMR Management Guide.

  • For API details, see Socks in AWS CLI Command Reference.

The following code example shows how to use ssh.

AWS CLI

To open an SSH connection to the primary node of a cluster

The following ssh example opens an ssh connection with the master instance in a cluster with the cluster ID j-3SD91U2E1L2QX.

aws emr ssh \
    --cluster-id j-3SD91U2E1L2QX \
    --key-pair-file ~/.ssh/mykey.pem

The key pair file option takes a local path to a private key file.

Output:

ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=10 -i /home/local/user/.ssh/mykey.pem hadoop@ec2-52-52-41-150.us-west-2.compute.amazonaws.com
Warning: Permanently added 'ec2-52-52-41-150.us-west-2.compute.amazonaws.com,52.52.41.150' (ECDSA) to the list of known hosts.
Last login: Mon Jun  1 23:15:38 2015

      __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2015.03-release-notes/
26 package(s) needed for security, out of 39 available
Run "sudo yum update" to apply all updates.

--------------------------------------------------------------------------------

Welcome to Amazon Elastic MapReduce running Hadoop and Amazon Linux.

Hadoop is installed in /home/hadoop. Log files are in /mnt/var/log/hadoop. Check
/mnt/var/log/hadoop/steps for diagnosing step failures.

The Hadoop UI can be accessed via the following commands:

  ResourceManager    lynx http://ip-172-21-11-216:9026/
  NameNode           lynx http://ip-172-21-11-216:9101/

--------------------------------------------------------------------------------

[hadoop@ip-172-31-16-216 ~]$

For more information, see Connect to the Amazon EMR cluster primary node using SSH in the Amazon EMR Management Guide.

  • For API details, see Ssh in AWS CLI Command Reference.