Menu
AWS Elastic Beanstalk
Developer Guide

Using Elastic Beanstalk with Amazon CloudWatch Logs

With CloudWatch Logs, you can monitor and archive your Elastic Beanstalk application, system, and custom log files from Amazon EC2 instances of your environments. You can also configure alarms that make it easier for you to react to specific log stream events that your metric filters extract. The CloudWatch Logs agent installed on each Amazon EC2 instance in your environment publishes metric data points to the CloudWatch service for each log group you configure. Each log group applies its own filter patterns to determine what log stream events to send to CloudWatch as data points. Log streams that belong to the same log group share the same retention, monitoring, and access control settings. You can configure Elastic Beanstalk to automatically stream logs to the CloudWatch service, as described in Streaming Instance Logs to CloudWatch Logs. For more information about CloudWatch Logs, including terminology and concepts, see the Amazon CloudWatch Logs User Guide.

In addition to instance logs, if you enable enhanced health for your environment, you can configure the environment to stream health information to CloudWatch Logs. See Streaming Elastic Beanstalk Environment Health Information to Amazon CloudWatch Logs.

The following figure shows the Monitoring page and graphs for an environment that is configured with CloudWatch Logs integration. The example metrics in this environment are named CWLHttp4xx and CWLHttp5xx. One of the graphs shows that the CWLHttp4xx metric has triggered an alarm based on conditions specified in the configuration files.

The following figure shows the Alarms page and graphs for the example alarms named AWSEBCWLHttp4xxPercentAlarm and AWSEBCWLHttp5xxCountAlarm that correspond to the CWLHttp4xx and CWLHttp5xx metrics, respectively.

Prerequisites to Instance Log Streaming to CloudWatch Logs

To enable streaming of logs from your environment's Amazon EC2 instances to CloudWatch Logs, you must meet the following conditions.

  • Platform – Because this feature is only available in platform configurations released on or after this release, if you are using an earlier platform configuration, update your environment to a current one.

  • If you don't have the AWSElasticBeanstalkWebTier or AWSElasticBeanstalkWorkerTier Elastic Beanstalk managed policy in your Elastic Beanstalk instance profile, you must add the following to your profile to enable this feature.

    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:PutLogEvents", "logs:CreateLogStream" ], "Resource": [ "*" ] } ] }

How Elastic Beanstalk Sets Up CloudWatch Logs

Elastic Beanstalk installs a CloudWatch log agent with the default configuration settings on each instance it creates. Learn more in the CloudWatch Logs Agent Reference.

When you enable instance log streaming to CloudWatch Logs, Elastic Beanstalk sends log files from your environment's instances to CloudWatch Logs. Different platforms stream different logs. The following table lists the logs, by platform.

Platform

Logs

Docker

  • /var/log/eb-activity.log

  • /var/log/nginx/error.log

  • /var/log/docker-events.log

  • /var/log/docker

  • /var/log/nginx/access.log

  • /var/log/eb-docker/containers/eb-current-app/stdouterr.log

Multi-Docker(generic)

  • /var/log/eb-activity.log

  • /var/log/ecs/ecs-init.log

  • /var/log/eb-ecs-mgr.log

  • /var/log/ecs/ecs-agent.log

  • /var/log/docker-events.log

Glass fish (Preconfigured Docker)

  • /var/log/eb-activity.log

  • /var/log/nginx/error.log

  • /var/log/docker-events.log

  • /var/log/docker

  • /var/log/nginx/access.log

Go (Preconfigured Docker)

  • /var/log/eb-activity.log

  • /var/log/nginx/error.log

  • /var/log/docker-events.log

  • /var/log/docker

  • /var/log/nginx/access.log

Python (Preconfigured Docker)

  • /var/log/eb-activity.log

  • /var/log/nginx/error.log

  • /var/log/docker-events.log

  • /var/log/docker

  • /var/log/nginx/access.log

Go

  • /var/log/eb-activity.log

  • /var/log/nginx/error.log

  • /var/log/nginx/access.log

Java

  • /var/log/eb-activity.log

  • /var/log/nginx/access.log

  • /var/log/nginx/error.log

  • /var/log/web-1.error.log

  • /var/log/web-1.log

Tomcat

  • /var/log/eb-activity.log

  • /var/log/httpd/error_log

  • /var/log/httpd/access_log

  • /var/log/nginx/error_log

  • /var/log/nginx/access_log

.NET on Windows Server

  • C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log

  • C:\Program Files\Amazon\ElasticBeanstalk\logs\AWSDeployment.log

  • C:\Program Files\Amazon\ElasticBeanstalk\logs\Hooks.log

Node.js

  • /var/log/eb-activity.log

  • /var/log/nodejs/nodejs.log

  • /var/log/nginx/error.log

  • /var/log/nginx/access.log

  • /var/log/httpd/error.log

  • /var/log/httpd/access.log

PHP

  • /var/log/eb-activity.log

  • /var/log/httpd/error_log

  • /var/log/httpd/access_log

Python

  • /var/log/eb-activity.log

  • /var/log/httpd/error_log

  • /var/log/httpd/access_log

  • /opt/python/log/supervisord.log

Ruby (Puma)

  • /var/log/eb-activity.log

  • /var/log/nginx/error.log

  • /var/log/puma/puma.log

  • /var/log/nginx/access.log

Ruby (Passenger)

  • /var/log/eb-activity.log

  • /var/app/support/logs/passenger.log

  • /var/app/support/logs/access.log

  • /var/app/support/logs/error.log

Elastic Beanstalk configures log groups in CloudWatch Logs for the various log files that it streams. To retrieve specific log files from CloudWatch Logs, you have to know the name of the corresponding log group. The log group naming scheme depends on the platform's operating system.

For Linux platforms, prefix the on-instance log file location with /aws/elasticbeanstalk/environment_name to get the log group name. For example, to retrieve the file /var/log/nginx/error.log, specify the log group /aws/elasticbeanstalk/environment_name/var/log/nginx/error.log.

For Windows platforms, see the following table for the log group corresponding to each log file.

On-instance log file

Log group

C:\Program Files\Amazon\ElasticBeanstalk\logs\AWSDeployment.log

/aws/elasticbeanstalk/<environment-name>/EBDeploy-Log

C:\Program Files\Amazon\ElasticBeanstalk\logs\Hooks.log

/aws/elasticbeanstalk/<environment-name>/EBHooks-Log

C:\inetpub\logs\LogFiles (the entire directory)

/aws/elasticbeanstalk/<environment-name>/IIS-Log

Streaming Instance Logs to CloudWatch Logs

You can enable instance log streaming to CloudWatch Logs using the Elastic Beanstalk console, the EB CLI, or configuration options.

Instance Log Streaming Using the Elastic Beanstalk Console

To stream instance logs to CloudWatch Logs

  1. Open the Elastic Beanstalk console.

  2. Navigate to the management page for your environment.

  3. Choose Configuration.

  4. On the Software configuration card, choose Modify.

  5. Under Instance log streaming to CloudWatch Logs:

    • Enable Log streaming.

    • Set Retention to the number of days to save the logs.

    • Select the Lifecycle setting that determines whether the logs are saved after the environment is terminated.

  6. Choose Save, and then choose Apply.

The example in the following figure saves the logs for seven days and keeps the logs after terminating the environment.

After you enable log streaming, you can return to the Software configuration card or page and find the Log Groups link. Click this link to see your logs in the CloudWatch console.

Instance Log Streaming Using the EB CLI

To enable instance log streaming to CloudWatch Logs using the EB CLI, use the eb logs command.

$ eb logs --cloudwatch-logs enable

You can also use eb logs to retrieve logs from CloudWatch Logs. You can retrieve all the environment's instance logs, or use the command's many options to specify subsets of logs to retrieve. For example, the following command retrieves the complete set of instance logs for your environment, and saves them to a directory under .elasticbeanstalk/logs.

$ eb logs --all

In particular, the --log-group option enables you to retrieve instance logs of a specific log group, corresponding to a specific on-instance log file. To do that, you need to know the name of the log group that corresponds to the log file you want to retrieve. You can find this information in How Elastic Beanstalk Sets Up CloudWatch Logs.

Instance Log Streaming Using Configuration Files

When you create or update an environment, you can use a configuration file to set up and configure instance log streaming to CloudWatch Logs. You can also configure the streaming of custom log files. To use each of the examples below, copy the text into a file with the .config extension in the .ebextensions directory at the top level of your application source bundle. You can use each example separately or combine them into a single configuration file.

The following example configuration file enables default instance log streaming. Elastic Beanstalk streams the default set of log files for your environment's platform.

option_settings: - namespace: aws:elasticbeanstalk:cloudwatch:logs option_name: StreamLogs value: true

The following example configuration file configures the streaming of custom log files that your application generates. The example configures the CloudWatch Logs agent on your environment's instances to stream each set of log files into a CloudWatch Logs log group named after the log files, for easy detection and retrieval.

Note

Elastic Beanstalk doesn't support custom log file streaming on the Windows platform. This example only works on Linux environments.

files: "/etc/awslogs/config/customlogs.conf": mode: "000600" owner: root group: root content: | [stdouterr.log] log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/eb-docker/containers/eb-current-app/stdouterr.log"]]}` log_stream_name = {instance_id} file = /var/log/eb-docker/containers/eb-current-app/*stdouterr.log* [sample-app.log] log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/eb-docker/containers/eb-current-app/sampleapp.log"]]}` log_stream_name = {instance_id} file = /var/log/eb-docker/containers/eb-current-app/sample-app.log* commands: "01": command: chkconfig awslogs on "02": command: service awslogs restart

For more information about configuring CloudWatch Logs, see the CloudWatch Logs Agent Reference in the Amazon CloudWatch Logs User Guide.

Before you can configure integration with CloudWatch Logs using configuration files, you must set up IAM permissions to use with the CloudWatch Logs agent. You can attach the following custom policy to the instance profile that you assign to your environment.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:GetLogEvents", "logs:PutLogEvents", "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:PutRetentionPolicy" ], "Resource": [ "*" ] } ] }

Troubleshooting CloudWatch Logs Integration

If you can't find some of the environment's instance logs you expect in CloudWatch Logs, you can investigate the following common issues:

  • Your IAM role lacks the required IAM permissions.

  • You launched your environment in an AWS Region that doesn't support CloudWatch Logs.

  • One of your custom log files doesn't exist in the path you specified.