Working with Amazon EMR-managed security groups - Amazon EMR
ElasticMapReduce-masterElasticMapReduce-slaveElasticMapReduce-master-privateElasticMapReduce-slave-privateElasticMapReduce-ServiceAccess

Working with Amazon EMR-managed security groups

Different managed security groups are associated with the master instance and with the core and task instances in a cluster. An additional managed security group for service access is required when you create a cluster in a private subnet. For more information about the role of managed security groups with respect to your network configuration, see Amazon VPC options.

When you specify managed security groups for a cluster, you must use the same type of security group, default or custom, for all managed security groups. For example, you can't specify a custom security group for the master instance, and then not specify a custom security group for core and task instances.

If you use default managed security groups, you don't need to specify them when you create a cluster. Amazon EMR automatically uses the defaults. Moreover, if the defaults don't exist in the cluster's VPC yet, Amazon EMR creates them. Amazon EMR also creates them if you explicitly specify them and they don't exist yet.

You can edit rules in managed security groups after clusters are created. When you create a new cluster, Amazon EMR checks the rules in the managed security groups that you specify, and then creates any missing inbound rules that the new cluster needs in addition to rules that may have been added earlier. Unless specifically stated otherwise, each rule for default EMR-managed security groups is also added to custom EMR-managed security groups that you specify.

The default managed security groups are as follows:

Amazon EMR-managed security group for the master instance (public subnets)

The default managed security group for the master instance in public subnets has the Group Name of ElasticMapReduce-master. It has the following rules. If you specify a custom managed security group, Amazon EMR will add all the same rules to your custom security group.

Type Protocol Port range Source Details
Inbound rules
All ICMP-IPv4 All N/A The Group ID of the managed security group for the master instance. In other words, the same security group in which the rule appears.

These reflexive rules allow inbound traffic from any instance associated with the specified security group. Using the default ElasticMapReduce-master for multiple clusters allows the core and task nodes of those clusters to communicate with each other over ICMP or any TCP or UDP port. Specify custom managed security groups to restrict cross-cluster access.

All TCP TCP All
All UDP UDP All
All ICMP-IPV4 All N/A The Group ID of the managed security group specified for core and task nodes.

These rules allow all inbound ICMP traffic and traffic over any TCP or UDP port from any core and task instances that are associated with the specified security group, even if the instances are in different clusters.

All TCP TCP All
All UDP UDP All
Custom TCP 8443 Various Amazon IP address ranges These rules allow the cluster manager to communicate with the master node.

To allow SSH access for trusted sources for the ElasticMapReduce-master security group

You must first be logged in to AWS as a root user or as an IAM principal that is allowed to manage security groups for the VPC that the cluster is in. For more information, see Changing Permissions for an IAM User and the Example Policy that allows managing EC2 security groups in the IAM User Guide.

  1. Open the Amazon EMR console at https://console.aws.amazon.com/elasticmapreduce/.

  2. Choose Clusters.

  3. Choose the Name of the cluster.

  4. Under Security and access choose the Security groups for Master link.

  5. Choose ElasticMapReduce-master from the list.

  6. Choose Inbound, Edit.

  7. Check for an inbound rule that allows public access with the following settings. If it exists, choose Delete to remove it.

    • Type

      SSH

    • Port

      22

    • Source

      Custom 0.0.0.0/0

    Warning

    Before December 2020, the default EMR-managed security group for the master instance in public subnets was created with a pre-configured rule to allow inbound traffic on Port 22 from all sources. This rule was created to simplify initial SSH connections to the master node. We strongly recommend that you remove this inbound rule and restrict traffic only from trusted sources.

  8. Scroll to the bottom of the list of rules and choose Add Rule.

  9. For Type, select SSH.

    This automatically enters TCP for Protocol and 22 for Port Range.

  10. For source, select My IP.

    This automatically adds the IP address of your client computer as the source address. Alternatively, you can add a range of Custom trusted client IP addresses and choose Add rule to create additional rules for other clients. Many network environments dynamically allocate IP addresses, so you might need to periodically edit security group rules to update IP addresses for trusted clients.

  11. Choose Save.

  12. Optionally, choose ElasticMapReduce-slave from the list and repeat the steps above to allow SSH client access to core and task nodes from trusted clients.

Amazon EMR-managed security group for core and task instances (public subnets)

The default managed security group for core and task instances in public subnets has the Group Name of ElasticMapReduce-slave. The default managed security group has the following rules, and Amazon EMR adds the same rules if you specify a custom managed security group.

Type Protocol Port range Source Details
Inbound rules
All ICMP-IPV4 All N/A The Group ID of the managed security group for core and task instances. In other words, the same security group in which the rule appears.

These reflexive rules allow inbound traffic from any instance associated with the specified security group. Using the default ElasticMapReduce-slave for multiple clusters allows the core and task instances of those clusters to communicate with each other over ICMP or any TCP or UDP port. Specify custom managed security groups to restrict cross-cluster access.

All TCP TCP All
All UDP UDP All
All ICMP-IPV4 All N/A The Group ID of the managed security group for the master instance.

These rules allow all inbound ICMP traffic and traffic over any TCP or UDP port from any master instances that are associated with the specified security group, even if the instances are in different clusters.

All TCP TCP All
All UDP UDP All

Amazon EMR-managed security group for the master instance (private subnets)

The default managed security group for the master instance in private subnets has the Group Name of ElasticMapReduce-Master-Private. The default managed security group has the following rules, and Amazon EMR adds the same rules if you specify a custom managed security group.

Type Protocol Port range Source Details
Inbound rules
All ICMP-IPv4 All N/A The Group ID of the managed security group for the master instance. In other words, the same security group in which the rule appears.

These reflexive rules allow inbound traffic from any instance associated with the specified security group and reachable from within the private subnet. Using the default ElasticMapReduce-Master-Private for multiple clusters allows the core and task nodes of those clusters to communicate with each other over ICMP or any TCP or UDP port. Specify custom managed security groups to restrict cross-cluster access.

All TCP TCP All
All UDP UDP All
All ICMP-IPV4 All N/A The Group ID of the managed security group for core and task nodes.

These rules allow all inbound ICMP traffic and traffic over any TCP or UDP port from any core and task instances that are associated with the specified security group and reachable from within the private subnet, even if the instances are in different clusters.

All TCP TCP All
All UDP UDP All
HTTPS (8443) TCP 8443 The Group ID of the managed security group for service access in a private subnet. This rule allows the cluster manager to communicate with the master node.
Outbound rules
All traffic All All 0.0.0.0/0 Provides outbound access to the internet.
Custom TCP TCP 9443 The Group ID of the managed security group for service access in a private subnet. If the above "All traffic" default outbound rule is removed, this rule is a minimum requirement for EMR 5.30.0 and later.

Amazon EMR-managed security group for core and task instances (private subnets)

The default managed security group for core and task instances in private subnets has the Group Name of ElasticMapReduce-Slave-Private. The default managed security group has the following rules, and Amazon EMR adds the same rules if you specify a custom managed security group.

Type Protocol Port range Source Details
Inbound rules
All ICMP-IPV4 All N/A The Group ID of the managed security group for core and task instances. In other words, the same security group in which the rule appears.

These reflexive rules allow inbound traffic from any instance associated with the specified security group. Using the default ElasticMapReduce-slave for multiple clusters allows the core and task instances of those clusters to communicate with each other over ICMP or any TCP or UDP port. Specify custom managed security groups to restrict cross-cluster access.

All TCP TCP All
All UDP UDP All
All ICMP-IPV4 All N/A The Group ID of the managed security group for the master instance.

These rules allow all inbound ICMP traffic and traffic over any TCP or UDP port from any master instances that are associated with the specified security group, even if the instances are in different clusters.

All TCP TCP All
All UDP UDP All
HTTPS (8443) TCP 8443 The Group ID of the managed security group for service access in a private subnet. This rule allows the cluster manager to communicate with core and task nodes.
Outbound rules
All traffic All All 0.0.0.0/0 See Editing outbound rules below.

Editing outbound rules

By default, Amazon EMR creates this security group with outbound rules that allow all outbound traffic on all protocols and ports. Allowing all outbound traffic is selected because various Amazon EMR and customer applications that can run on Amazon EMR clusters may require different egress rules. EMR is not able to anticipate these specific settings when creating default security groups. You can scope down egress in your security groups to include only those rules that suit your use cases and security policies. At minimum, this security group requires the following outbound rules, but some applications might need additional egress.

Type Protocol Port range Destination Details
All TCP TCP All pl-xxxxxxxx Managed Amazon S3 prefix list com.amazonaws.MyRegion.s3.
All Traffic All All sg-xxxxxxxxxxxxxxxxx The ID of the ElasticMapReduce-Slave-Private security group.
All Traffic All All sg-xxxxxxxxxxxxxxxxx The ID of the ElasticMapReduce-Master-Private security group.
Custom TCP TCP 9443 sg-xxxxxxxxxxxxxxxxx The ID of the ElasticMapReduce-ServiceAccess security group.

Amazon EMR-managed security group for service access (private subnets)

The default managed security group for service access in private subnets has the Group Name of ElasticMapReduce-ServiceAccess. It has inbound rules, and outbound rules that allow traffic over HTTPS (port 8443, port 9443) to the other managed security groups in private subnets. These rules allow the cluster manager to communicate with the master node and with core and task nodes. The same rules are needed if you are using custom security groups.

Type Protocol Port range Source Details
Inbound rules Required for EMR clusters with Amazon EMR release 5.30.0 and later.
HTTPS (9443) TCP 9443 The Group ID of the managed security group for master instance.

This rule allows the communication between master instance's security group to the service access security group.

Outbound rules Required for all EMR clusters
HTTPS (8443) TCP 8443 The Group ID of the managed security group for master instance.

These rules allow the cluster manager to communicate with the master node and with core and task nodes.

HTTPS (8443) TCP 8443 The Group ID of the managed security group for core and task instances.

These rules allow the cluster manager to communicate with the master node and with core and task nodes.