Working with Amazon EMR-managed security groups - Amazon EMR
ElasticMapReduce-primaryElasticMapReduce-coreElasticMapReduce-primary-privateElasticMapReduce-core-privateElasticMapReduce-ServiceAccess

Working with Amazon EMR-managed security groups

Note

Amazon EMR aims to use inclusive alternatives for potentially offensive or non-inclusive industry terms such as "master" and "slave". We've transitioned to new terminology to foster a more inclusive experience and to facilitate your understanding of the service components.

We now describe "nodes" as instances, and we describe Amazon EMR instance types as primary, core, and task instances. During the transition, you might still find legacy references to the outdated terms, such as those that pertain to security groups for Amazon EMR.

Different managed security groups are associated with the primary instance and with the core and task instances in a cluster. An additional managed security group for service access is required when you create a cluster in a private subnet. For more information about the role of managed security groups with respect to your network configuration, see Amazon VPC options when you launch a cluster.

When you specify managed security groups for a cluster, you must use the same type of security group, default or custom, for all managed security groups. For example, you can't specify a custom security group for the primary instance, and then not specify a custom security group for core and task instances.

If you use default managed security groups, you don't need to specify them when you create a cluster. Amazon EMR automatically uses the defaults. Moreover, if the defaults don't exist in the cluster's VPC yet, Amazon EMR creates them. Amazon EMR also creates them if you explicitly specify them and they don't exist yet.

You can edit rules in managed security groups after clusters are created. When you create a new cluster, Amazon EMR checks the rules in the managed security groups that you specify, and then creates any missing inbound rules that the new cluster needs in addition to rules that may have been added earlier. Unless specifically stated otherwise, each rule for default Amazon EMR-managed security groups is also added to custom Amazon EMR-managed security groups that you specify.

The default managed security groups are as follows:

Amazon EMR-managed security group for the primary instance (public subnets)

The default managed security group for the primary instance in public subnets has the Group Name of ElasticMapReduce-primary. It has the following rules. If you specify a custom managed security group, Amazon EMR adds all the same rules to your custom security group.

Type Protocol Port range Source Details
Inbound rules
All ICMP-IPv4 All N/A The Group ID of the managed security group for the primary instance. In other words, the same security group in which the rule appears.

These reflexive rules allow inbound traffic from any instance associated with the specified security group. Using the default ElasticMapReduce-primary for multiple clusters allows the core and task nodes of those clusters to communicate with each other over ICMP or any TCP or UDP port. Specify custom managed security groups to restrict cross-cluster access.
All TCP TCP All
All UDP UDP All
All ICMP-IPV4 All N/A The Group ID of the managed security group specified for core and task nodes.

These rules allow all inbound ICMP traffic and traffic over any TCP or UDP port from any core and task instances that are associated with the specified security group, even if the instances are in different clusters.
All TCP TCP All
All UDP UDP All
Custom TCP 8443 Various Amazon IP address ranges These rules allow the cluster manager to communicate with the primary node.
To grant trusted sources SSH access to the primary security group with the console

To edit your security groups, you must have permission to manage security groups for the VPC that the cluster is in. For more information, see Changing Permissions for a user and the Example Policy that allows managing EC2 security groups in the IAM User Guide.

  1. Sign in to the AWS Management Console, and open the Amazon EMR console at https://console.aws.amazon.com/emr.

  2. Choose Clusters. Choose the ID of the cluster you want to modify.

  3. In the Network and security pane, expand the EC2 security groups (firewall) dropdown.

  4. Under Primary node, choose your security group.

  5. Choose Edit inbound rules.

  6. Check for an inbound rule that allows public access with the following settings. If it exists, choose Delete to remove it.

    • Type

      SSH

    • Port

      22

    • Source

      Custom 0.0.0.0/0

    Warning

    Before December 2020, there was a pre-configured rule to allow inbound traffic on Port 22 from all sources. This rule was created to simplify initial SSH connections to the primary node. We strongly recommend that you remove this inbound rule and restrict traffic to trusted sources.

  7. Scroll to the bottom of the list of rules and choose Add Rule.

  8. For Type, select SSH.

    Selecting SSH automatically enters TCP for Protocol and 22 for Port Range.

  9. For source, select My IP to automatically add your IP address as the source address. You can also add a range of Custom trusted client IP addresses, or create additional rules for other clients. Many network environments dynamically allocate IP addresses, so you might need to update your IP addresses for trusted clients in the future.

  10. Choose Save.

  11. Optionally, choose the other security group under Core and task nodes in the Network and security pane and repeat the steps above to allow SSH client access to core and task nodes.

Amazon EMR-managed security group for core and task instances (public subnets)

The default managed security group for core and task instances in public subnets has the Group Name of ElasticMapReduce-core. The default managed security group has the following rules, and Amazon EMR adds the same rules if you specify a custom managed security group.

Type Protocol Port range Source Details
Inbound rules
All ICMP-IPV4 All N/A The Group ID of the managed security group for core and task instances. In other words, the same security group in which the rule appears.

These reflexive rules allow inbound traffic from any instance associated with the specified security group. Using the default ElasticMapReduce-core for multiple clusters allows the core and task instances of those clusters to communicate with each other over ICMP or any TCP or UDP port. Specify custom managed security groups to restrict cross-cluster access.
All TCP TCP All
All UDP UDP All
All ICMP-IPV4 All N/A The Group ID of the managed security group for the primary instance.

These rules allow all inbound ICMP traffic and traffic over any TCP or UDP port from any primary instances that are associated with the specified security group, even if the instances are in different clusters.

All TCP TCP All
All UDP UDP All

Amazon EMR-managed security group for the primary instance (private subnets)

The default managed security group for the primary instance in private subnets has the Group Name of ElasticMapReduce-Primary-Private. The default managed security group has the following rules, and Amazon EMR adds the same rules if you specify a custom managed security group.

Type Protocol Port range Source Details
Inbound rules
All ICMP-IPv4 All N/A The Group ID of the managed security group for the primary instance. In other words, the same security group in which the rule appears.

These reflexive rules allow inbound traffic from any instance associated with the specified security group and reachable from within the private subnet. Using the default ElasticMapReduce-Primary-Private for multiple clusters allows the core and task nodes of those clusters to communicate with each other over ICMP or any TCP or UDP port. Specify custom managed security groups to restrict cross-cluster access.
All TCP TCP All
All UDP UDP All
All ICMP-IPV4 All N/A The Group ID of the managed security group for core and task nodes.

These rules allow all inbound ICMP traffic and traffic over any TCP or UDP port from any core and task instances that are associated with the specified security group and reachable from within the private subnet, even if the instances are in different clusters.
All TCP TCP All
All UDP UDP All
HTTPS (8443) TCP 8443 The Group ID of the managed security group for service access in a private subnet. This rule allows the cluster manager to communicate with the primary node.
Outbound rules
All traffic All All 0.0.0.0/0 Provides outbound access to the internet.
Custom TCP TCP 9443 The Group ID of the managed security group for service access in a private subnet.

If the above "All traffic" default outbound rule is removed, this rule is a minimum requirement for Amazon EMR 5.30.0 and later.

Note

Amazon EMR does not add this rule when you use a custom managed security group.
Custom TCP TCP 80 (http) or 443 (https) The Group ID of the managed security group for service access in a private subnet.

If the above "All traffic" default outbound rule is removed, this rule is a minimum requirement for Amazon EMR 5.30.0 and later to connect to Amazon S3 over https.

Note

Amazon EMR does not add this rule when you use a custom managed security group.

Amazon EMR-managed security group for core and task instances (private subnets)

The default managed security group for core and task instances in private subnets has the Group Name of ElasticMapReduce-Core-Private. The default managed security group has the following rules, and Amazon EMR adds the same rules if you specify a custom managed security group.

Type Protocol Port range Source Details
Inbound rules
All ICMP-IPV4 All N/A The Group ID of the managed security group for core and task instances. In other words, the same security group in which the rule appears.

These reflexive rules allow inbound traffic from any instance associated with the specified security group. Using the default ElasticMapReduce-core for multiple clusters allows the core and task instances of those clusters to communicate with each other over ICMP or any TCP or UDP port. Specify custom managed security groups to restrict cross-cluster access.
All TCP TCP All
All UDP UDP All
All ICMP-IPV4 All N/A The Group ID of the managed security group for the primary instance.

These rules allow all inbound ICMP traffic and traffic over any TCP or UDP port from any primary instances that are associated with the specified security group, even if the instances are in different clusters.

All TCP TCP All
All UDP UDP All
HTTPS (8443) TCP 8443 The Group ID of the managed security group for service access in a private subnet. This rule allows the cluster manager to communicate with core and task nodes.
Outbound rules
All traffic All All 0.0.0.0/0 See Editing outbound rules below.
Custom TCP TCP 80 (http) or 443 (https) The Group ID of the managed security group for service access in a private subnet.

If the above "All traffic" default outbound rule is removed, this rule is a minimum requirement for Amazon EMR 5.30.0 and later to connect to Amazon S3 over https.

Note

Amazon EMR does not add this rule when you use a custom managed security group.

Editing outbound rules

By default, Amazon EMR creates this security group with outbound rules that allow all outbound traffic on all protocols and ports. Allowing all outbound traffic is selected because various Amazon EMR and customer applications that can run on Amazon EMR clusters may require different egress rules. Amazon EMR is not able to anticipate these specific settings when creating default security groups. You can scope down egress in your security groups to include only those rules that suit your use cases and security policies. At minimum, this security group requires the following outbound rules, but some applications might need additional egress.

Type Protocol Port range Destination Details
All TCP TCP All pl-xxxxxxxx Managed Amazon S3 prefix list com.amazonaws.MyRegion.s3.
All Traffic All All sg-xxxxxxxxxxxxxxxxx The ID of the ElasticMapReduce-Core-Private security group.
All Traffic All All sg-xxxxxxxxxxxxxxxxx The ID of the ElasticMapReduce-Primary-Private security group.
Custom TCP TCP 9443 sg-xxxxxxxxxxxxxxxxx The ID of the ElasticMapReduce-ServiceAccess security group.

Amazon EMR-managed security group for service access (private subnets)

The default managed security group for service access in private subnets has the Group Name of ElasticMapReduce-ServiceAccess. It has inbound rules, and outbound rules that allow traffic over HTTPS (port 8443, port 9443) to the other managed security groups in private subnets. These rules allow the cluster manager to communicate with the primary node and with core and task nodes. The same rules are needed if you are using custom security groups.

Type Protocol Port range Source Details
Inbound rules Required for Amazon EMR clusters with Amazon EMR release 5.30.0 and later.
Custom TCP TCP 9443 The Group ID of the managed security group for primary instance.

This rule allows the communication between primary instance's security group to the service access security group.
Outbound rules Required for all Amazon EMR clusters
Custom TCP TCP 8443 The Group ID of the managed security group for primary instance.

These rules allow the cluster manager to communicate with the primary node and with core and task nodes.
Custom TCP TCP 8443 The Group ID of the managed security group for core and task instances.

These rules allow the cluster manager to communicate with the primary node and with core and task nodes.