Using Elastic Beanstalk with Amazon CloudWatch Logs - AWS Elastic Beanstalk

Using Elastic Beanstalk with Amazon CloudWatch Logs

With CloudWatch Logs, you can monitor and archive your Elastic Beanstalk application, system, and custom log files from Amazon EC2 instances of your environments. You can also configure alarms that make it easier for you to react to specific log stream events that your metric filters extract. The CloudWatch Logs agent installed on each Amazon EC2 instance in your environment publishes metric data points to the CloudWatch service for each log group you configure. Each log group applies its own filter patterns to determine what log stream events to send to CloudWatch as data points. Log streams that belong to the same log group share the same retention, monitoring, and access control settings. You can configure Elastic Beanstalk to automatically stream logs to the CloudWatch service, as described in Streaming instance logs to CloudWatch Logs. For more information about CloudWatch Logs, including terminology and concepts, see the Amazon CloudWatch Logs User Guide.

In addition to instance logs, if you enable enhanced health for your environment, you can configure the environment to stream health information to CloudWatch Logs. See Streaming Elastic Beanstalk environment health information to Amazon CloudWatch Logs.

The following figure shows the Monitoring page and graphs for an environment that is configured with CloudWatch Logs integration. The example metrics in this environment are named CWLHttp4xx and CWLHttp5xx. One of the graphs shows that the CWLHttp4xx metric has triggered an alarm based on conditions specified in the configuration files.

The following figure shows the Alarms page and graphs for the example alarms named AWSEBCWLHttp4xxPercentAlarm and AWSEBCWLHttp5xxCountAlarm that correspond to the CWLHttp4xx and CWLHttp5xx metrics, respectively.

Prerequisites to instance log streaming to CloudWatch Logs

To enable streaming of logs from your environment's Amazon EC2 instances to CloudWatch Logs, you must meet the following conditions.

  • Platform – Because this feature is only available in platform versions released on or after this release, if you are using an earlier platform version, update your environment to a current one.

  • If you don't have the AWSElasticBeanstalkWebTier or AWSElasticBeanstalkWorkerTier Elastic Beanstalk managed policy in your Elastic Beanstalk instance profile, you must add the following to your profile to enable this feature.

    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:PutLogEvents", "logs:CreateLogStream" ], "Resource": [ "*" ] } ] }

How Elastic Beanstalk sets up CloudWatch Logs

Elastic Beanstalk installs a CloudWatch log agent with the default configuration settings on each instance it creates. Learn more in the CloudWatch Logs Agent Reference.

When you enable instance log streaming to CloudWatch Logs, Elastic Beanstalk sends log files from your environment's instances to CloudWatch Logs. Different platforms stream different logs. The following table lists the logs, by platform.

Platform / Platform Branch

Logs

Docker /

Platform Branch: Docker Running on 64bit Amazon Linux 2

  • /var/log/eb-engine.log

  • /var/log/eb-hooks.log

  • /var/log/docker

  • /var/log/docker-events.log

  • /var/log/eb-docker/containers/eb-current-app/stdouterr.log

  • /var/log/nginx/access.log

  • /var/log/nginx/error.log

Docker /

Platform Branch: ECS Running on 64bit Amazon Linux 2

  • /var/log/docker-events.log

  • /var/log/eb-ecs-mgr.log

  • /var/log/eb-engine.log

  • /var/log/eb-hooks.log

  • /var/log/ecs/ecs-agent.log

  • /var/log/ecs/ecs-init.log

Go

.NET Core on Linux

Java / Platform Branch: Corretto running on 64bit Amazon Linux 2

  • /var/log/eb-engine.log

  • /var/log/eb-hooks.log

  • /var/log/web.stdout.log

  • /var/log/nginx/access.log

  • /var/log/nginx/error.log

Node.js

Python

  • /var/log/eb-engine.log

  • /var/log/eb-hooks.log

  • /var/log/web.stdout.log

  • /var/log/httpd/access_log

  • /var/log/httpd/error_log

  • /var/log/nginx/access.log

  • /var/log/nginx/error.log

Tomcat

PHP

  • /var/log/eb-engine.log

  • /var/log/eb-hooks.log

  • /var/log/httpd/access_log

  • /var/log/httpd/error_log

  • /var/log/nginx/access.log

  • /var/log/nginx/error.log

.NET on Windows Server

  • C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log

  • C:\Program Files\Amazon\ElasticBeanstalk\logs\AWSDeployment.log

  • C:\Program Files\Amazon\ElasticBeanstalk\logs\Hooks.log

Ruby

  • /var/log/eb-engine.log

  • /var/log/eb-hooks.log

  • /var/log/puma/puma.log

  • /var/log/web.stdout.log

  • /var/log/nginx/access.log

  • /var/log/nginx/error.log

Note

On July 18,2022, Elastic Beanstalk set the status of all platform branches based on Amazon Linux AMI (AL1) to retired. For more information about migrating to a current and fully supported Amazon Linux 2023 platform branch, see Migrating your Elastic Beanstalk Linux application to Amazon Linux 2023 or Amazon Linux 2.

The following table lists the log files streamed from instances on platform branches based on Amazon Linux AMI (preceding Amazon Linux 2), by platform.

Platform / Platform Branch

Logs

Docker /

Platform Branch: Docker Running on 64bit Amazon Linux

  • /var/log/eb-activity.log

  • /var/log/nginx/error.log

  • /var/log/docker-events.log

  • /var/log/docker

  • /var/log/nginx/access.log

  • /var/log/eb-docker/containers/eb-current-app/stdouterr.log

Docker /

Platform Branch: Multicontainer Docker Running on 64bit Amazon Linux

  • /var/log/eb-activity.log

  • /var/log/ecs/ecs-init.log

  • /var/log/eb-ecs-mgr.log

  • /var/log/ecs/ecs-agent.log

  • /var/log/docker-events.log

Glassfish (Preconfigured Docker)

  • /var/log/eb-activity.log

  • /var/log/nginx/error.log

  • /var/log/docker-events.log

  • /var/log/docker

  • /var/log/nginx/access.log

Go

  • /var/log/eb-activity.log

  • /var/log/nginx/error.log

  • /var/log/nginx/access.log

Java /

Platform Branch: Java 8 running on 64bit Amazon Linux

Platform Branch: Java 7 running on 64bit Amazon Linux

  • /var/log/eb-activity.log

  • /var/log/nginx/access.log

  • /var/log/nginx/error.log

  • /var/log/web-1.error.log

  • /var/log/web-1.log

Tomcat

  • /var/log/eb-activity.log

  • /var/log/httpd/error_log

  • /var/log/httpd/access_log

  • /var/log/nginx/error_log

  • /var/log/nginx/access_log

Node.js

  • /var/log/eb-activity.log

  • /var/log/nodejs/nodejs.log

  • /var/log/nginx/error.log

  • /var/log/nginx/access.log

  • /var/log/httpd/error.log

  • /var/log/httpd/access.log

PHP

  • /var/log/eb-activity.log

  • /var/log/httpd/error_log

  • /var/log/httpd/access_log

Python

  • /var/log/eb-activity.log

  • /var/log/httpd/error_log

  • /var/log/httpd/access_log

  • /opt/python/log/supervisord.log

Ruby /

Platform Branch: Puma with Ruby running on 64bit Amazon Linux

  • /var/log/eb-activity.log

  • /var/log/nginx/error.log

  • /var/log/puma/puma.log

  • /var/log/nginx/access.log

Ruby /

Platform Branch: Passenger with Ruby running on 64bit Amazon Linux

  • /var/log/eb-activity.log

  • /var/app/support/logs/passenger.log

  • /var/app/support/logs/access.log

  • /var/app/support/logs/error.log

Elastic Beanstalk configures log groups in CloudWatch Logs for the various log files that it streams. To retrieve specific log files from CloudWatch Logs, you have to know the name of the corresponding log group. The log group naming scheme depends on the platform's operating system.

For Linux platforms, prefix the on-instance log file location with /aws/elasticbeanstalk/environment_name to get the log group name. For example, to retrieve the file /var/log/nginx/error.log, specify the log group /aws/elasticbeanstalk/environment_name/var/log/nginx/error.log.

For Windows platforms, see the following table for the log group corresponding to each log file.

On-instance log file

Log group

C:\Program Files\Amazon\ElasticBeanstalk\logs\AWSDeployment.log

/aws/elasticbeanstalk/<environment-name>/EBDeploy-Log

C:\Program Files\Amazon\ElasticBeanstalk\logs\Hooks.log

/aws/elasticbeanstalk/<environment-name>/EBHooks-Log

C:\inetpub\logs\LogFiles (the entire directory)

/aws/elasticbeanstalk/<environment-name>/IIS-Log

Streaming instance logs to CloudWatch Logs

You can enable instance log streaming to CloudWatch Logs using the Elastic Beanstalk console, the EB CLI, or configuration options.

Before you enable it, set up IAM permissions to use with the CloudWatch Logs agent. You can attach the following custom policy to the instance profile that you assign to your environment.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogStream", "logs:PutLogEvents", "logs:DescribeLogGroups", "logs:DescribeLogStreams" ], "Resource": [ "*" ] } ] }

Instance log streaming using the Elastic Beanstalk console

To stream instance logs to CloudWatch Logs
  1. Open the Elastic Beanstalk console, and in the Regions list, select your AWS Region.

  2. In the navigation pane, choose Environments, and then choose the name of your environment from the list.

    Note

    If you have many environments, use the search bar to filter the environment list.

  3. In the navigation pane, choose Configuration.

  4. In the Updates, monitoring, and logging configuration category, choose Edit.

  5. Under Instance log streaming to CloudWatch Logs:

    • Enable Log streaming.

    • Set Retention to the number of days to save the logs.

    • Select the Lifecycle setting that determines whether the logs are saved after the environment is terminated.

  6. To save the changes choose Apply at the bottom of the page.

After you enable log streaming, you can return to the Software configuration category or page and find the Log Groups link. Click this link to see your logs in the CloudWatch console.

Instance log streaming using the EB CLI

To enable instance log streaming to CloudWatch Logs using the EB CLI, use the eb logs command.

$ eb logs --cloudwatch-logs enable

You can also use eb logs to retrieve logs from CloudWatch Logs. You can retrieve all the environment's instance logs, or use the command's many options to specify subsets of logs to retrieve. For example, the following command retrieves the complete set of instance logs for your environment, and saves them to a directory under .elasticbeanstalk/logs.

$ eb logs --all

In particular, the --log-group option enables you to retrieve instance logs of a specific log group, corresponding to a specific on-instance log file. To do that, you need to know the name of the log group that corresponds to the log file you want to retrieve. You can find this information in How Elastic Beanstalk sets up CloudWatch Logs.

Instance log streaming using configuration files

When you create or update an environment, you can use a configuration file to set up and configure instance log streaming to CloudWatch Logs. The following example configuration file enables default instance log streaming. Elastic Beanstalk streams the default set of log files for your environment's platform. To use the example, copy the text into a file with the .config extension in the .ebextensions directory at the top level of your application source bundle.

option_settings: - namespace: aws:elasticbeanstalk:cloudwatch:logs option_name: StreamLogs value: true

Custom log file streaming

The Elastic Beanstalk integration with CloudWatch Logs doesn't directly support the streaming of custom log files that your application generates. To stream custom logs, use a configuration file to directly install the CloudWatch Logs agent and to configure the files to be pushed. For an example configuration file, see logs-streamtocloudwatch-linux.config.

Note

The example doesn't work on the Windows platform.

For more information about configuring CloudWatch Logs, see the CloudWatch Logs Agent Reference in the Amazon CloudWatch Logs User Guide.

Troubleshooting CloudWatch Logs integration

If you can't find some of the environment's instance logs you expect in CloudWatch Logs, you can investigate the following common issues:

  • Your IAM role lacks the required IAM permissions.

  • You launched your environment in an AWS Region that doesn't support CloudWatch Logs.

  • One of your custom log files doesn't exist in the path you specified.