Monitor application activity by using CloudWatch Logs Insights - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsBest practicesEpicsTroubleshootingRelated resources

Monitor application activity by using CloudWatch Logs Insights

Ram Kandaswamy, Amazon Web Services

Summary

This pattern provides a solution for automatically detecting and alerting on application exceptions by using Amazon CloudWatch Logs Insights. By implementing automated log analysis and alerting, you can quickly identify and respond to application issues in your production environment.

Logs play a crucial role in monitoring system behavior, identifying issues, and ensuring optimal performance. During a migration process, log files are invaluable for validating the system's functioning in the new environment, detecting compatibility problems, and identifying any unexpected behaviors. Issues could be related to operations or security. For security-related issues, enabling the detection of unauthorized access attempts or suspicious activities early is essential for maintaining security and regulatory compliance. This capability is especially important when dealing with sensitive data or critical systems.

This pattern is particularly valuable for teams that need to maintain high application availability and quickly respond to production issues. It’s relevant across various industries and use cases. For example, in telecommunications, it enables quick identification of network misconfigurations or outages and detection of suboptimal routing paths to pinpoint potential congestion. In the Internet of Things (IoT) domain, Greengrass components can publish logs to CloudWatch, allowing this technique to extract relevant log details and present them in a comprehensive dashboard.

Prerequisites and limitations

Prerequisites

  • A production application deployed in active AWS account

  • Basic understanding of the production application's logging format and exception patterns

  • Application logs configured to stream to Amazon CloudWatch Logs

Limitations

Architecture

The following diagram shows how CloudWatch Logs Insights evaluates resource logs and sends a relevant data visualization to a CloudWatch dashboard.

CloudWatch Logs Insights evaluates resource logs and sends data visualization to dashboard.

The diagram shows the following workflow:

  1. The resources publish logs to CloudWatch Logs. Resources can include AWS resources such as Amazon Elastic Compute Cloud (Amazon EC2) instances or Amazon Simple Storage Service (Amazon S3) buckets. Another example includes on-premises systems with CloudWatch Agent installed that can publish logs to CloudWatch.

  2. CloudWatch Logs Insights filters for the relevant pattern string. Examples of search pattern strings include "error", "exception", or a specific regular expression.

  3. Typically, the production support team or developers add the pattern visualization to the CloudWatch dashboard.

Automation and scale

Developers can automate this pattern’s solution by using the AWS Cloud Development Kit (AWS CDK), AWS CloudFormation, or AWS SDKs to handle multiple string patterns. Teams can incorporate this automation into their continuous integration and deployment (CI/CD) DevOps processes.

Tools

AWS services

Best practices

Epics

TaskDescriptionSkills required

Configure IAM permissions.

To configure IAM permissions, do the following:

  1. Identify the IAM roles used by your application for writing logs and by users or services that will create dashboards, queries, and alarms.

  2. For the application role, add the following API actions and resource to a policy:

    {
         "Version": "2012-10-17",
         "Statement": [
             {
                 "Effect": "Allow",
                 "Action": [
                     "logs:CreateLogGroup",
                     "logs:CreateLogStream",
                     "logs:PutLogEvents"
                 ],
                 "Resource": [
                     "arn:aws:logs:*:*:*"
                 ]
             }
         ]
     }

  3. For managing CloudWatch resources, add the following to a policy:

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutDashboard",
                "cloudwatch:GetDashboard",
                "logs:StartQuery",
                "logs:StopQuery",
                "logs:GetQueryResults",
                "cloudwatch:PutMetricAlarm"
            ],
            "Resource": "*"
        }
    ]
}

For information about how to create IAM policies or to add permissions to existing policies, see Define custom IAM permissions with customer managed policies and Edit IAM policies in the IAM User Guide. For more information, see Identity and access management for Amazon CloudWatch Logs and CloudWatch Logs permissions reference in the Amazon CloudWatch Logs User Guide.

AWS administrator, AWS DevOps, AWS systems administrator, Cloud administrator, Cloud architect, DevOps engineer

Create a log group.

To create a log group, use any of the following options:

  • Create or open a CloudFormation template file (YAML or JSON format) with a .yaml or .json extension. (The following code uses YAML format.) Add the following resource definition to the Resources section of your template:

    MyLogGroup:
  Type: AWS::Logs::LogGroup
  Properties:
    LogGroupName: my-log-group

    For more information, see Quick Start: Use AWS CloudFormation to get started with CloudWatch Logs in the Amazon CloudWatch Logs User Guide.

AWS administrator, AWS DevOps, AWS systems administrator, Cloud administrator, Cloud architect, DevOps engineer

Generate a CloudWatch Logs Insights query.

To create and save a CloudWatch Logs Insights query:

  1. Open the CloudWatch console.

  2. Navigate to Logs Insights.

  3. Create your query by using one of the following methods:

    • Start with a sample query from the Queries section.

    • Write a custom query.

    • Modify the following example query:

      fields @timestamp, @message | filter @message like /(?i)exception/|stats count() by bin(30s)

      This query checks the log file, extracts the timestamp and message content, filters for the word "exception" (case-insensitive), and counts occurrences in 30-second intervals.

AWS administrator, AWS DevOps, AWS systems administrator, Cloud administrator, Cloud architect, DevOps engineer

Create visualization in a CloudWatch dashboard.

To use a CloudWatch dashboard to create a visualization, do the following:

  1. Open the CloudWatch console, and create a CloudWatch dashboard (if you don’t have one already) or open an existing dashboard.

  2. Add your exception monitoring visualization. Bar, line, and pie charts are available, depending on your business need. For more information, see Using widgets on CloudWatch dashboards in the Amazon CloudWatch Logs User Guide. For real-time data visualizations, you can customize a widget to meet your requirements.

For more information about dashboard options and capabilities, see Using Amazon CloudWatch dashboards and Creating flexible CloudWatch dashboards with dashboard variables in the Amazon CloudWatch Logs User Guide.

AWS administrator, AWS DevOps, AWS systems administrator, Cloud administrator, Cloud architect, DevOps engineer

Troubleshooting

IssueSolution

Unable to see query results or query seems broken

Start with a working query that was modified from a sample query. Perform small incremental changes to parts of the query (such as a filter or field), and take advantage of the CloudWatch Logs query generator feature.

Log groups not creating log streams

In the IAM policy, make sure that the resource for the CreateLogStream and the CreateLogGroup operations is provided with a wildcard character (*) value. The create operation will not succeed without this wildcard permission.

Related resources