Configure cluster logging and debugging - Amazon EMR
Default log filesArchive log files to Amazon S3Log locationsEnable the debugging toolDebugging option information

Configure cluster logging and debugging

One of the things to decide as you plan your cluster is how much debugging support you want to make available. When you are first developing your data processing application, we recommend testing the application on a cluster processing a small, but representative, subset of your data. When you do this, you will likely want to take advantage of all the debugging tools that Amazon EMR offers, such as archiving log files to Amazon S3.

When you've finished development and put your data processing application into full production, you may choose to scale back debugging. Doing so can save you the cost of storing log file archives in Amazon S3 and reduce processing load on the cluster as it no longer needs to write state to Amazon S3. The trade off, of course, is that if something goes wrong, you'll have fewer tools available to investigate the issue.

Default log files

By default, each cluster writes log files on the primary node. These are written to the /mnt/var/log/ directory. You can access them by using SSH to connect to the primary node as described in Connect to the primary node using SSH.

Note

If you use Amazon EMR release 6.8.0 or earlier, log files are saved to Amazon S3 during cluster termination, so you can't access the log files once the primary node terminates. Amazon EMR releases 6.9.0 and later archive logs to Amazon S3 during cluster scale-down, so log files generated on the cluster persist even after the node is terminated.

You do not need to enable anything to have log files written on the primary node. This is the default behavior of Amazon EMR and Hadoop.

A cluster generates several types of log files, including:

  • Step logs — These logs are generated by the Amazon EMR service and contain information about the cluster and the results of each step. The log files are stored in /mnt/var/log/hadoop/steps/ directory on the primary node. Each step logs its results in a separate numbered subdirectory: /mnt/var/log/hadoop/steps/s-stepId1/ for the first step, /mnt/var/log/hadoop/steps/s-stepId2/, for the second step, and so on. The 13-character step identifiers (e.g. stepId1, stepId2) are unique to a cluster.

  • Hadoop and YARN component logs — The logs for components associated with both Apache YARN and MapReduce, for example, are contained in separate folders in /mnt/var/log. The log file locations for the Hadoop components under /mnt/var/log are as follows: hadoop-hdfs, hadoop-mapreduce, hadoop-httpfs, and hadoop-yarn. The hadoop-state-pusher directory is for the output of the Hadoop state pusher process.

  • Bootstrap action logs — If your job uses bootstrap actions, the results of those actions are logged. The log files are stored in /mnt/var/log/bootstrap-actions/ on the primary node. Each bootstrap action logs its results in a separate numbered subdirectory: /mnt/var/log/bootstrap-actions/1/ for the first bootstrap action, /mnt/var/log/bootstrap-actions/2/, for the second bootstrap action, and so on.

  • Instance state logs — These logs provide information about the CPU, memory state, and garbage collector threads of the node. The log files are stored in /mnt/var/log/instance-state/ on the primary node.

Archive log files to Amazon S3

Note

You cannot currently use log aggregation to Amazon S3 with the yarn logs utility.

Amazon EMR releases 6.9.0 and later archive logs to Amazon S3 during cluster scale-down, so log files generated on the cluster persist even after the node is terminated. This behavior is enabled automatically, so you don't need to do anything to turn it on. For Amazon EMR releases 6.8.0 and earlier, you can configure a cluster to periodically archive the log files stored on the primary node to Amazon S3. This ensures that the log files are available after the cluster terminates, whether this is through normal shut down or due to an error. Amazon EMR archives the log files to Amazon S3 at 5 minute intervals.

To have the log files archived to Amazon S3 for Amazon EMR releases 6.8.0 and earlier, you must enable this feature when you launch the cluster. You can do this using the console, the CLI, or the API. By default, clusters launched using the console have log archiving enabled. For clusters launched using the CLI or API, logging to Amazon S3 must be manually enabled.

Note

We’ve redesigned the Amazon EMR console to make it easier to use. See Amazon EMR console to learn about the differences between the old and new console experiences.

New console
To archive log files to Amazon S3 with the new console

  1. Sign in to the AWS Management Console, and open the Amazon EMR console at https://console.aws.amazon.com/emr.

  2. Under EMR on EC2 in the left navigation pane, choose Clusters, and then choose Create cluster.

  3. Under Cluster logs, select the Publish cluster-specific logs to Amazon S3 check box.

  4. In the Amazon S3 location field, type (or browse to) an Amazon S3 path to store your logs. If you type the name of a folder that doesn't exist in the bucket, Amazon S3 creates it.

    When you set this value, Amazon EMR copies the log files from the EC2 instances in the cluster to Amazon S3. This prevents the log files from being lost when the cluster ends and the EC2 terminates the instances hosting the cluster. These logs are useful for troubleshooting purposes. For more information, see View log files.

  5. Optionally, select the Encrypt cluster-specific logs check box. Then, select an AWS KMS key from the list, enter a key ARN, or create a new key. This option is only available with Amazon EMR version 5.30.0 and later, excluding version 6.0.0. To use this option, add permission to AWS KMS for your EC2 instance profile and Amazon EMR role. For more information, see To encrypt log files stored in Amazon S3 with an AWS KMS customer managed key.

  6. Choose any other options that apply to your cluster.

  7. To launch your cluster, choose Create cluster.

Old console
To archive log files to Amazon S3 with the old console

  1. Navigate to the new Amazon EMR console and select Switch to the old console from the side navigation. For more information on what to expect when you switch to the old console, see Using the old console.

  2. Choose Create cluster.

  3. Choose Go to advanced options.

  4. In the General options section, in the Logging field, accept the default option: Enabled.

    This determines whether Amazon EMR captures detailed log data to Amazon S3. You can only set this when the cluster is created. For more information, see View log files.

  5. In the S3 folder field, type (or browse to) an Amazon S3 path to store your logs. You may also allow the console to generate an Amazon S3 path for you. If you type the name of a folder that does not exist in the bucket, it is created.

    When this value is set, Amazon EMR copies the log files from the EC2 instances in the cluster to Amazon S3. This prevents the log files from being lost when the cluster ends and the EC2 instances hosting the cluster are terminated. These logs are useful for troubleshooting purposes.

    For more information, see View log files.

  6. In the Log encryption field, select Encrypt logs stored in S3 with an AWS KMS customer managed key. Then select an AWS KMS key from the list or enter a key ARN. You may also create a new AWS KMS key.

    This option is only available with Amazon EMR version 5.30.0 and later, excluding version 6.0.0. To use this option, add permission to AWS KMS for your EC2 instance profile and Amazon EMR role. For more information, see To encrypt log files stored in Amazon S3 with an AWS KMS customer managed key.

  7. Proceed with creating the cluster as described in Plan and configure clusters.

CLI
To archive log files to Amazon S3 with the AWS CLI

To archive log files to Amazon S3 using the AWS CLI, type the create-cluster command and specify the Amazon S3 log path using the --log-uri parameter.

  1. To log files to Amazon S3 type the following command and replace myKey with the name of your EC2 key pair.

    aws emr create-cluster --name "Test cluster" --release-label emr-7.0.0 --log-uri s3://DOC-EXAMPLE-BUCKET/logs --applications Name=Hadoop Name=Hive Name=Pig --use-default-roles --ec2-attributes KeyName=myKey --instance-type m5.xlarge --instance-count 3

  2. When you specify the instance count without using the --instance-groups parameter, a single primary node is launched, and the remaining instances are launched as core nodes. All nodes will use the instance type specified in the command.

    Note

    If you have not previously created the default Amazon EMR service role and EC2 instance profile, enter aws emr create-default-roles to create them before typing the create-cluster subcommand.

To encrypt log files stored in Amazon S3 with an AWS KMS customer managed key

With Amazon EMR version 5.30.0 and later (except Amazon EMR 6.0.0), you can encrypt log files stored in Amazon S3 with an AWS KMS customer managed key. To enable this option in the console, follow the steps in Archive log files to Amazon S3. Your Amazon EC2 instance profile and your Amazon EMR role must meet the following prerequisites:

  • The Amazon EC2 instance profile used for your cluster must have permission to use kms:GenerateDataKey.

  • The Amazon EMR role used for your cluster must have permission to use kms:DescribeKey.

  • The Amazon EC2 instance profile and Amazon EMR role must be added to the list of key users for the specified AWS KMS customer managed key, as the following steps demonstrate:

    1. Open the AWS Key Management Service (AWS KMS) console at https://console.aws.amazon.com/kms.

    2. To change the AWS Region, use the Region selector in the upper-right corner of the page.

    3. Select the alias of the KMS key to modify.

    4. On the key details page under Key Users, choose Add.

    5. In the Add key users dialog box, select your Amazon EC2 instance profile and Amazon EMR role.

    6. Choose Add.

For more information, see IAM service roles used by Amazon EMR, and Using key policies in the AWS Key Management Service developer guide.

To aggregate logs in Amazon S3 using the AWS CLI

Note

You cannot currently use log aggregation with the yarn logs utility. You can only use aggregation supported by this procedure.

Log aggregation (Hadoop 2.x) compiles logs from all containers for an individual application into a single file. To enable log aggregation to Amazon S3 using the AWS CLI, you use a bootstrap action at cluster launch to enable log aggregation and to specify the bucket to store the logs.

  • To enable log aggregation create the following configuration file called myConfig.json that contains the following:

    [
  {
    "Classification": "yarn-site",
    "Properties": {
      "yarn.log-aggregation-enable": "true",
      "yarn.log-aggregation.retain-seconds": "-1",
      "yarn.nodemanager.remote-app-log-dir": "s3:\/\/DOC-EXAMPLE-BUCKET\/logs"
    }
  }
]

    Type the following command and replace myKey with the name of your EC2 key pair. You can additionally replace any of the red text with your own configurations.

    aws emr create-cluster --name "Test cluster" \
--release-label emr-7.0.0 \
--applications Name=Hadoop \
--use-default-roles \
--ec2-attributes KeyName=myKey \
--instance-type m5.xlarge \
--instance-count 3 \
--configurations file://./myConfig.json

    When you specify the instance count without using the --instance-groups parameter, a single primary node is launched, and the remaining instances are launched as core nodes. All nodes will use the instance type specified in the command.

    Note

    If you have not previously created the default EMR service role and EC2 instance profile, run aws emr create-default-roles to create them before running the create-cluster subcommand.

For more information on using Amazon EMR commands in the AWS CLI, see AWS CLI Command Reference.

Log locations

The following list includes all log types and their locations in Amazon S3. You can use these for troubleshooting Amazon EMR issues.

Step logs

s3://DOC-EXAMPLE-LOG-BUCKET/<cluster-id>/steps/<step-id>/

Application logs

s3://DOC-EXAMPLE-LOG-BUCKET/<cluster-id>/containers/

This location includes container stderr and stdout, directory.info, prelaunch.out, and launch_container.sh logs.

Resource manager logs

s3://DOC-EXAMPLE-LOG-BUCKET/<cluster-id>/node/<leader-instance-id>/applications/hadoop-yarn/

Hadoop HDFS

s3://DOC-EXAMPLE-LOG-BUCKET/<cluster-id>/node/<all-instance-id>/applications/hadoop-hdfs/

This location includes NameNode, DataNode, and YARN TimelineServer logs.

Node manager logs

s3://DOC-EXAMPLE-LOG-BUCKET/<cluster-id>/node/<all-instance-id>/applications/hadoop-yarn/

Instance-state logs

s3://DOC-EXAMPLE-LOG-BUCKET/<cluster-id>/node/<all-instance-id>/daemons/instance-state/

Amazon EMR provisioning logs

s3://DOC-EXAMPLE-LOG-BUCKET/<cluster-id>/node/<leader-instance-id>/provision-node/*

Hive logs

s3://DOC-EXAMPLE-LOG-BUCKET/<cluster-id>/node/<leader-instance-id>/applications/hive/*

  • To find Hive logs on your cluster, remove the asterisk (*) and append /var/log/hive/ to the above link.

  • To find HiveServer2 logs, remove the asterisk (*) and append var/log/hive/hiveserver2.log to the above link.

  • To find HiveCLI logs, remove the asterisk (*) and append /var/log/hive/user/hadoop/hive.log to the above link.

  • To find Hive Metastore Server logs, remove the asterisk (*) and append /var/log/hive/user/hive/hive.log to the above link.

If your failure is in the primary or task node of your Tez application, provide logs of the appropriate Hadoop container.

Enable the debugging tool

The debugging tool allows you to more easily browse log files from the Amazon EMR console. For more information, see View log files in the debugging tool. When you enable debugging on a cluster, Amazon EMR archives the log files to Amazon S3 and then indexes those files. You can then use the console to browse the step, job, task, and task-attempt logs for the cluster in an intuitive way.

To use the debugging tool in the Amazon EMR console, you must enable debugging when you launch the cluster using the console, the CLI, or the API. Note that the new Amazon EMR console doesn't offer the debugging tool.

Old console
To turn on the debugging tool with the old console

  1. Navigate to the new Amazon EMR console and select Switch to the old console from the side navigation. For more information on what to expect when you switch to the old console, see Using the old console.

  2. Choose Create cluster.

  3. Choose Go to advanced options.

  4. In the Cluster Configuration section, in the Logging field, choose Enabled. You cannot enable debugging without enabling logging.

  5. In the Log folder S3 location field, type an Amazon S3 path to store your logs.

  6. In the Debugging field, choose Enabled. The debugging option creates an Amazon SQS exchange to publish debugging messages to the Amazon EMR service backend. Charges for publishing messages to the exchange may apply. For more information, see the Amazon SQS product page.

  7. Proceed with creating the cluster as described in Plan and configure clusters.

AWS CLI
To turn on the debugging tool with the AWS CLI

To enable debugging using the AWS CLI, type the create-cluster subcommand with the --enable-debugging parameter. You must also specify the --log-uri parameter when enabling debugging.

  • To enable debugging using the AWS CLI, type the following command and replace myKey with the name of your EC2 key pair.

    aws emr create-cluster --name "Test cluster" \
--release-label emr-7.0.0 \
--log-uri s3://DOC-EXAMPLE-BUCKET/logs \
--enable-debugging \
--applications Name=Hadoop Name=Hive Name=Pig \
--use-default-roles \
--ec2-attributes KeyName=myKey \
--instance-type m5.xlarge \
--instance-count 3

    When you specify the instance count without using the --instance-groups parameter, a single primary node is launched, and the remaining instances are launched as core nodes. All nodes will use the instance type specified in the command.

    Note

    If you have not previously created the default EMR service role and EC2 instance profile, type aws emr create-default-roles to create them before typing the create-cluster subcommand.

API
To turn on the debugging tool with the Amazon EMR API

  • Enable debugging using the following Java SDK configuration.

        StepFactory stepFactory = new StepFactory(); 
	StepConfig enabledebugging = new StepConfig()
   		.withName("Enable debugging")
   		.withActionOnFailure("TERMINATE_JOB_FLOW")
   		.withHadoopJarStep(stepFactory.newEnableDebuggingStep());

    In this example, new StepFactory() uses us-east-1 as the default region. If your cluster is launched in a different region, you need to specify the region by using new StepFactory("region.elasticmapreduce"), such as new StepFactory("ap-northeast-2.elasticmapreduce").

Debugging option information

Amazon EMR releases 4.1.0 through 5.27.0 support debugging in all Regions. Other Amazon EMR versions do not support the debugging option. Effective January 23, 2023, Amazon EMR will discontinue the debugging tool for all versions.

Amazon EMR creates an Amazon SQS queue to process debugging data. Message charges may apply. However, Amazon SQS does have Free Tier of up to 1,000,000 requests available. For more information, see https://aws.amazon.com/sqs.

Debugging requires the use of roles; your service role and instance profile must allow you to use all Amazon SQS API operations. If your roles are attached to Amazon EMR managed policies, you do not need to do anything to modify your roles. If you have custom roles, you need to add sqs:* permissions. For more information, see Configure IAM service roles for Amazon EMR permissions to AWS services and resources.