Analyzing log data with CloudWatch Logs Insights - Amazon CloudWatch Logs

Analyzing log data with CloudWatch Logs Insights

With CloudWatch Logs Insights, you can interactively search and analyze your log data in Amazon CloudWatch Logs. You can perform queries to help you more efficiently and effectively respond to operational issues. If an issue occurs, you can use CloudWatch Logs Insights to identify potential causes and validate deployed fixes.

CloudWatch Logs Insights supports three query languages that you can use for your queries:

  • A purpose-built Logs Insights query language (Logs Insights QL) with a few simple but powerful commands.

  • (New) OpenSearch Service Piped Processing Language (PPL). OpenSearch PPL enables you to analyze your logs using a set of commands delimited by pipes (|).

    With OpenSearch PPL you can retrieve, query, and analyze data by using commands that are piped together, making it easier to understand and compose complex queries. The syntax enables the chaining of commands to transform and process data. With PPL, you can filter and aggregate data, and use a rich set of math, string, date, conditional and other functions for analysis.

  • (New) OpenSearch Service Structured Query Language (SQL). With OpenSearch SQL queries, you can analyze your logs in a declarative manner. You can use commands such as SELECT, FROM, WHERE, GROUP BY, HAVING, and various other commands and functions available in SQL. You can execute JOINs across log groups, correlate data across logs using sub-queries, and use the rich set of JSON, Mathematical, String, Conditional and other SQL functions to perform powerful analysis on logs.

CloudWatch Logs Insights offers the following features that are available for use with any of the query languages.

  • Automatic discovery of log fields in logs from AWS services such as Amazon Route 53, AWS Lambda, AWS CloudTrail, and Amazon VPC, and any application or custom log that emits log events as JSON.

  • Creating field indexes to reduce costs and speed results, especially for queries of large number of log groups or log events. After creating field indexes of fields that are common in your log events, you can use them in in a query. The query skips processing log events that are known to not include the indexed field, and processes less data.

    Note

    The filterIndex command is available only in Logs Insights QL.

  • Detection and analysis of patterns in your log events. A pattern is a shared text structure that recurs among your log fields. When you view the results of a query, you can choose the Patterns tab to see the patterns that CloudWatch Logs found based on a sample of your results.

  • Saving queries, seeing your query history, and re-running saved queries. This can help you run complex queries when you need, without having to re-create them each time that you want to run them.

  • Adding queries to dashboards.

  • Encrypting query results with AWS Key Management Service.

The following CloudWatch Logs Insights features are supported only when you use Logs Insights QL.

Important

CloudWatch Logs Insights can't access log events with timestamps that pre-date the creation time of the log group.

If you are signed in to an account set up as a monitoring account in CloudWatch cross-account observability, you can run CloudWatch Logs Insights queries on log groups in source accounts linked to this monitoring account. You can run a query that queries multiple log groups located in different accounts. For more information, see CloudWatch cross-account observability.

When you create queries using Logs Insights QL, you can also use natural language to create CloudWatch Logs Insights queries. To do so, ask questions about or describe the data you're looking for. This AI-assisted capability generates a query based on your prompt and provides a line-by-line explanation of how the query works. For more information, see Use natural language to generate and update CloudWatch Logs Insights queries.

Queries using any of the supported query languages time out after 60 minutes, if they have not completed. Query results are available for seven days.

CloudWatch Logs Insights queries incur charges based on the amount of data that is queried, regardless of query language. For more information, see Amazon CloudWatch Pricing.

You can use CloudWatch Logs Insights to search log data that was sent to CloudWatch Logs on November 5, 2018 or later.

Important

If your network security team doesn't allow the use of web sockets, you can't currently access the CloudWatch Logs Insights portion of the CloudWatch console. You can use the CloudWatch Logs Insights query capabilities using APIs. For more information, see StartQuery in the Amazon CloudWatch Logs API Reference.

Contents