Menu
Amazon EMR
Management Guide

Use Amazon EMR–Managed Security Groups

When you launch an Amazon EMR cluster, you must specify one Amazon EMR–managed security group for the master instance, one for Amazon EMR–managed security group for the core/task (slave) instances, and optionally, one for the Amazon EMR resources used to manage clusters in private subnets. The core/task instances share the same security group.

You can use the default Amazon EMR–managed security groups for cluster resources or you can choose your own security groups for the master and core/task instances. You cannot combine your own security groups with the default groups.

When you launch a cluster using the console, if the default security groups do not exist, the Amazon EMR–managed security groups fields are populated with Create ElasticMapReduce-master and Create ElasticMapReduce-slave or Create ElasticMapReduce-Master-Private, Create ElasticMapReduce-Slave-Private, and Create ElasticMapReduce-ServiceAccess. If the default security groups exist, the fields are populated with the default security group IDs; for example, Default: sg-01XXXX6a (ElasticMapReduce-master) and Default: sg-07XXXX6c (ElasticMapReduce-slave). To use your own security groups, choose them from the lists. For more information about the default Amazon EMR–managed security group options, see Default Options for Amazon EMR–Managed Security Groups.

Amazon EMR-specific inbound and outbound access rules are written to and maintained in the Amazon EMR–managed security groups. Whether you choose the default security groups or your own, Amazon EMR modifies the security group rules to ensure proper communication between instances in a cluster. For more information about the rules written to the Amazon EMR–managed security groups, see Rules in Amazon EMR–Managed Security Groups.

If you need to control instance membership in the Amazon EMR–managed security groups, you can create your own security groups for the master and core/task instances. For more information, see Security Groups for Your VPC in the Amazon VPC User Guide. When you create your own security groups, any inbound and outbound rules required for proper intra-cluster communication are written to the groups with some exceptions.

Typically, you would create your own Amazon EMR–managed security groups to isolate clusters so they cannot communicate with each other. This alleviates the need to create separate VPCs or AWS accounts for each of your clusters. For example, to isolate Amazon EMR clusters in a VPC, you create a security group for each cluster's master node, and you create a security group for each cluster's core/task nodes. When you launch a cluster, the rules in your security groups are applied only to the instances in that cluster, effectively preventing instances in separate clusters from communicating with each other.

If you need separate clusters to communicate, and you are using your own security groups, you can create an additional security group containing the rules necessary for the cluster instances to communicate with each other, or you can use the same security groups for both clusters. If you do not specify an additional security group when launching the clusters, and the clusters use different security groups, the clusters cannot communicate with each other unless you modify the rule sets in your master and core/task security groups. For more information about additional security groups, see Configure Additional Security Groups .

To specify Amazon EMR–managed security groups using the console

  1. Open the Amazon EMR console at https://console.aws.amazon.com/elasticmapreduce/.

  2. Choose Create cluster.

  3. In the Security and Access section, in the EC2 Security Groups subsection:

    • For Master, if the default security groups do not exist, the field is populated with Create ElasticMapReduce-master or Create ElasticMapReduce-Master-Private. If the default security groups exist, the field is populated with the default security group ID; for example, Default: sg-01XXXX6a (ElasticMapReduce-master). To specify your own master security group, choose it from the list.

    • For Core & Task, if the default security groups do not exist, the field is populated with Create ElasticMapReduce-slave or Create ElasticMapReduce-Slave-Private. If the default security groups exist, the field is populated with the default security group ID; for example, Default: sg-07XXXX6c (ElasticMapReduce-slave). To specify your own core/task security group, choose it from the list.

    • (Optional) For Service Access to clusters in private subnets, if the default security groups do not exist, the field is populated with Create ElasticMapReduce-ServiceAccess. If the default security groups exist, the field is populated with the default security group ID; for example, Default: sg-4dXXXX34 (ElasticMapReduce-ServiceAccess). To specify your own service access security group, choose it from the list.

  4. Proceed with creating the cluster as described in Use an Amazon EC2 Key Pair for SSH Credentials.

To specify Amazon EMR–managed security groups using the AWS CLI

Use the create-cluster command with the --emr-managed-master-security-group and --emr-managed-slave-security-group parameters.

If you are using the default options for Amazon EMR–managed security groups, no additional parameters are required. Use the create-cluster command as you normally would using the CLI. If the default security groups do not exist, they are created before your cluster is launched. If they do exist, they are automatically assigned.

Note

Amazon EMR–managed security groups are not supported in the Amazon EMR CLI.

  1. To launch a cluster using the default security group options, type the following command, replace myKey with the name of your Amazon EC2 key pair.

    aws emr create-cluster --name "Test cluster" --release-label emr-4.2.0 --applications Name=Hive Name=Pig --use-default-roles --ec2-attributes KeyName=myKey --instance-type m3.xlarge --instance-count 3

    When you specify the instance count without using the --instance-groups parameter, a single master node is launched, and the remaining instances are launched as core nodes. All nodes use the instance type specified in the command.

  2. To launch a cluster using your own security groups, type the following command, replace myKey with the name of your Amazon EC2 key pair, replace masterSecurityGroupId with the ID of the master security group, and replace slaveSecurityGroupId with the ID of the core/task security group.

    aws emr create-cluster --name "Test cluster" --release-label emr-4.2.0 --applications Name=Hive Name=Pig --ec2-attributes KeyName=myKey,EmrManagedMasterSecurityGroup=sg-masterId,EmrManagedSlaveSecurityGroup=sg-slaveId --instance-type m3.xlarge --instance-count 3 --use-default-roles

    Note

    For private subnets, you can additionally specify ServiceAccessSecurityGroup=sg-service-accessId with the --ec2-attributes parameter.

    When you specify the instance count without using the --instance-groups parameter, a single master node is launched, and the remaining instances are launched as core nodes. All nodes use the instance type specified in the command.

  3. To retrieve security group information for your cluster, type the following command and replace j-1K48XXXXXXHCB with your cluster ID:

    aws emr describe-cluster --cluster-id j-1K48XXXXXXHCB

    For more information, see Amazon EMR commands in the AWS CLI.

Default Options for Amazon EMR–Managed Security Groups

If you launch an Amazon EMR cluster using the default security groups, two groups are created for public subnets: ElasticMapReduce-master and ElasticMapReduce-slave. For private subnets, three groups are created:

  • Create ElasticMapReduce-Master-Private

  • Create ElasticMapReduce-Slave-Private

  • Create ElasticMapReduce-ServiceAccess

The inbound and outbound access rules written to these groups ensure that the master and core/task instances in a cluster can communicate properly.

In addition, if you launch other Amazon EMR clusters in the same VPC using the default security groups, the instances in those clusters can communicate with the instances in any other Amazon EMR cluster within that VPC whose instances also belong to the same security groups.

You can launch a cluster with the default security groups using the console, the API, the CLI, or the SDK. If you use the default security groups, there is no need to change your existing code or to add parameters to your CLI commands.

You can launch a cluster with the default security groups using the console. If the default security groups do not exist, they are created before your cluster is launched. If they do exist, they are automatically assigned.

Rules in Amazon EMR–Managed Security Groups

The tables in this section list the inbound and outbound access rules added to the Amazon EMR–managed security groups. IP address ranges and rules are automatically updated for Amazon EMR–managed security groups.

Warning

We do not recommend creating an inbound rule for any protocol or port that allows inbound traffic from all IP addresses (0.0.0.0/0). This opens access to everyone and creates a security vulnerability.

The following rules are added to Amazon EMR–managed master security groups:

Type Protocol Port Range Source Details
Inbound rules
All ICMP ICMP All The default ElasticMapReduce-master security group ID or the master security group ID that you specify; for example, sg-88XXXXed

Allows inbound traffic from any instance in the ElasticMapReduce-master security group. By default, the master nodes in all Amazon EMR clusters in a single VPC can communicate with each other over any TCP, UDP, or ICMP port.

If you choose your own master security group, only master instances in the group can communicate with each other over any TCP, UDP, or ICMP port.

All TCP TCP All
All UDP UDP All
All ICMP ICMP All The default ElasticMapReduce-slave security group ID or the core/task security group ID that you specify; for example, sg-8bXXXXee

Allows inbound traffic from any instance in the ElasticMapReduce-slave security group. By default, the master node accepts inbound communication from any core/task node in any Amazon EMR cluster in a single VPC over any TCP, UDP, or ICMP port.

If you choose your own core/task security group, only core/task instances in this group can communicate with the master node over any TCP, UDP, or ICMP port.

All TCP TCP All
All UDP UDP All
HTTPS TCP 8443 Various Amazon IP address ranges Allows the cluster manager to communicate with the master nodes in each Amazon EMR cluster in a single VPC.
Outbound rules
All traffic All All 0.0.0.0/0 Provides outbound access to the Internet from any instance in the ElasticMapReduce-master security group or the group that you specify.

The following rules are added to Amazon EMR–managed core/task security groups:

Type Protocol Port Range Source Details
Inbound rules
All ICMP ICMP All The default ElasticMapReduce-master security group ID or the master security group ID that you specify; for example, sg-88XXXXed

Allows inbound traffic from any instance in the ElasticMapReduce-master security group or the group that you specify. By default, the core/task nodes in all Amazon EMR clusters in a single VPC accept inbound communication from master nodes over any TCP, UDP, or ICMP port.

If you choose your own master security group, only master instances in this group can communicate with the core/task nodes over any TCP, UDP, or ICMP port.

All TCP TCP All
All UDP UDP All
All ICMP ICMP All The default ElasticMapReduce-slave security group ID or the core/task security group ID that you specify; for example, sg-8bXXXXee

Allows inbound traffic from any instance in the ElasticMapReduce-slave security group. By default, the core/task nodes accept inbound communication from any other core/task node in any Amazon EMR cluster in a single VPC over any TCP, UDP, or ICMP port.

If you choose your own core/task security group, only core/task instances in this group can communicate with each other over any TCP, UDP, or ICMP port.

All TCP TCP All
All UDP UDP All
Custom TCP TCP 8443
Outbound rules
All traffic All All 0.0.0.0/0 Provides outbound access to the Internet from all instances in the ElasticMapReduce-slave security group or the group that you specify.