What is Amazon CloudWatch Application Insights? - Amazon CloudWatch

What is Amazon CloudWatch Application Insights?

CloudWatch Application Insights helps you monitor your applications that use Amazon EC2 instances along with other application resources. It identifies and sets up key metrics, logs, and alarms across your application resources and technology stack (for example, your Microsoft SQL Server database, web (IIS) and application servers, OS, load balancers, and queues). It continuously monitors metrics and logs to detect and correlate anomalies and errors. When errors and anomalies are detected, Application Insights generates CloudWatch Events that you can use to set up notifications or take actions. To assist with troubleshooting, it creates automated dashboards for detected problems, which include correlated metric anomalies and log errors, along with additional insights to point you to a potential root cause. The automated dashboards help you to take remedial actions to keep your applications healthy and to prevent impact to the end-users of your application. It also creates OpsItems so that you can resolve problems using AWS SSM OpsCenter.

You can configure important counters, such as Mirrored Write Transaction/sec, Recovery Queue Length, and Transaction Delay, as well as Windows Event Logs on CloudWatch. When a failover event or problem occurs with your SQL HA workload, such as a restricted access to query a target database, CloudWatch Application Insights provides automated insights .

CloudWatch Application Insights integrates with AWS Launch Wizard to provide a one-click monitoring setup experience for deploying SQL Server HA workloads on AWS. When you select the option to set up monitoring and insights with Application Insights on the Launch Wizard console, CloudWatch Application Insights automatically sets up relevant metrics, logs, and alarms on CloudWatch, and starts monitoring newly deployed workloads. You can view automated insights and detected problems, along with the health of your SQL Server HA workloads, on the CloudWatch console.

Features

Application Insights provides the following features.

Automatic set up of monitors for application resources

CloudWatch Application Insights reduces the time it takes to set up monitoring for your applications. It does this by scanning your application resources, providing a customizable list of recommended metrics and logs, and setting them up on CloudWatch to provide necessary visibility into your application resources, such as Amazon EC2 and Elastic Load Balancers (ELB). It also sets up dynamic alarms on monitored metrics. The alarms are automatically updated based on anomalies detected in the previous two weeks.

Problem detection and notification

CloudWatch Application Insights detects signs of potential problems with your application, such as metric anomalies and log errors. It correlates these observations to surface potential problems with your application. It then generates CloudWatch Events, which can be configured to receive notifications or take actions. This eliminates the need for you to create individual alarms on metrics or log errors.

Troubleshooting

CloudWatch Application Insights creates CloudWatch automatic dashboards for problems that are detected. The dashboards show details about the problem, including the associated metric anomalies and log errors to help you with troubleshooting. They also provide additional insights that point to potential root causes of the anomalies and errors.

Concepts

The following concepts are important for understanding how Application Insights monitors your application.

Component

An auto-grouped, standalone, or custom grouping of similar resources that make up an application. We recommend grouping similar resources into custom components for better monitoring.

Observation

An individual event (metric anomaly, log error, or exception) that is detected with an application or application resource.

Problem

Problems are detected by correlating, classifying, and grouping related observations.

For definitions of other key concepts for CloudWatch Application Insights, see Amazon CloudWatch Concepts.

Pricing

CloudWatch Application Insights sets up recommended metrics and logs for selected application resources using CloudWatch metrics, Logs, and Events for notifications on detected problems. These features are charged to your AWS account according to CloudWatch pricing. For detected problems, SSM OpsItems are also created by Application Insights to notify you about problems. Additionally, Application Insights creates SSM Parameter Store parameters to configure the CloudWatch agents on your instances. The Amazon EC2 Systems Manager features are charged according to SSM pricing. You are not charged for setup assistance, monitoring, data analysis, or problem detection.

Costs for CloudWatch Application Insights

Costs for Amazon EC2 include usage of the following features:

  • CloudWatch Agent

    • CloudWatch Agent log groups

    • CloudWatch Agent metrics

    • Prometheus log groups (for JMX workloads)

Costs for all resources include usage of the following features:

  • CloudWatch alarms (majority of cost)

  • SSM OpsItems (minimal cost)

Example cost calculation

The costs in this example are considered according to the following scenario.

You created a resource group that includes the following:

  • An Amazon EC2 instance with SQL Server installed.

  • An attached Amazon EBS volume.

When you onboard this resource group with CloudWatch Application Insights, the SQL Server workload installed on the Amazon EC2 instance is detected. CloudWatch Application Insights starts monitoring the following metrics.

The following metrics are monitored for the SQL Server instance:

  • CPUUtilization

  • StatusCheckFailed

  • Memory % Committed Bytes in Use

  • Memory Available Mbytes

  • Network Interface Bytes Total/sec

  • Paging File % Usage

  • Physical Disk % Disk Time

  • Processor % Processor Time

  • SQLServer:Buffer Manager cache hit ratio

  • SQLServer:Buffer Manager life expectancy

  • SQLServer:General Statistics Processes blocked

  • SQLServer:General Statistics User Connections

  • SQLServer:Locks Number of Deadlocks/sec

  • SQLServer:SQL Statistics Batch Requests/sec

  • System Processor Queue Length

The following metrics are monitored for the volumes attached to the SQL Server instance:

  • VolumeReadBytes

  • VolumeWriteBytes

  • VolumeReadOps

  • VolumeWriteOps

  • VolumeTotalReadTime

  • VolumeTotalWriteTime

  • VolumeIdleTime

  • VolumeQueueLength

  • VolumeThroughputPercentage

  • VolumeConsumedReadWriteOps

  • BurstBalance

For this scenario, the costs are calculated according to the CloudWatch pricing page and the SSM pricing page:

  • Custom metrics

    For this scenario, 13 of the above metrics are emitted to CloudWatch using the CloudWatch agent. These metrics are treated as custom metrics. The cost for each custom metric is $.3/month. The total cost for these custom metrics is 13 * $.3 = $3.90/month.

  • Alarms

    For this scenario, CloudWatch Application Insights monitors 26 metrics in total, which creates 26 alarms. The cost for each alarm is $.1/month. The total cost for alarms is 26 * $.1 = $2.60/month.

  • Data ingestion and error logs

    The cost of data ingestion is $.05/GB and storage for the SQL Server error log is $.03/GB. The total cost for data ingestion and the error log is $.05/GB + $.03/GB= $.08/GB.

  • Amazon EC2 Systems Manager OpsItems

    An SSM OpsItem is created for each problem detected by CloudWatch Application Insights. For n number of problems in your application, the total cost is $.00267 * n/month.

The following services are used along with CloudWatch Application Insights:

Related AWS services
  • Amazon CloudWatch provides system‐wide visibility into resource utilization, application performance, and operational health. It collects and tracks metrics, sends alarm notifications, automatically updates resources that you are monitoring based on the rules that you define, and allows you to monitor your own custom metrics. CloudWatch Application Insights is initiated through CloudWatch—specifically, within the CloudWatch default operational dashboards. For more information, see the Amazon CloudWatch User Guide.

  • CloudWatch Container Insights collects, aggregates, and summarizes metrics and logs from your containerized applications and microservices. You can use Container Insights to monitor Amazon ECS, Amazon Elastic Kubernetes Service, and Kubernetes platforms on Amazon EC2. When Application Insights is enabled on the Container Insights or Application Insights consoles, Application Insights displays detected problems on your Container Insights dashboard. For more information, see Using Container Insights .

  • Amazon DynamoDB is a fully managed NoSQL database service that lets you offload the administrative burdens of operating and scaling a distributed database so that you don't have to worry about hardware provisioning, setup and configuration, replication, software patching, or cluster scaling. DynamoDB also offers encryption at rest, which eliminates the operational burden and complexity involved in protecting sensitive data.

  • Amazon EC2 provides scalable computing capacity in the AWS Cloud. You can use Amazon EC2 to launch as many or as few virtual servers as you need, to configure security and networking, and to manage storage. You can scale up or down to handle changes in requirements or spikes in popularity, which reduces your need to forecast traffic. For more information, see the Amazon EC2 User Guide for Linux Instances or Amazon EC2 Guide for Windows Instances.

  • Amazon Elastic Block Store (Amazon EBS) provides block-level storage volumes for use with Amazon EC2 instances. Amazon EBS volumes behave like raw, unformatted block devices. You can mount these volumes as devices on your instances. Amazon EBS volumes that are attached to an instance are exposed as storage volumes that persist independently from the life of the instance. You can create a file system on top of these volumes, or use them in any way you would use a block device (such as a hard drive). You can dynamically change the configuration of a volume attached to an instance. For more information, see the Amazon EBS User Guide.

  • Amazon EC2 Auto Scaling helps ensure that you have the correct number of EC2 instances available to handle the load for your application. For more information, see the Amazon EC2 Auto Scaling User Guide.

  • Elastic Load Balancing distributes incoming applications or network traffic across multiple targets, such as EC2 instances, containers, and IP addresses, in multiple Availability Zones. For more information, see the Elastic Load Balancing User Guide.

  • IAM is a web service that helps you to securely control access to AWS resources for your users. Use IAM to control who can use your AWS resources (authentication), and to control the resources they can use and how they can use them (authorization). For more information, see Authentication and Access Control for Amazon CloudWatch.

  • AWS Lambda lets you build serverless applications composed of functions that are triggered by events and automatically deploy them using CodePipeline and AWS CodeBuild. For more information, see AWS Lambda Applications.

  • AWS Launch Wizard for SQL Server reduces the time it takes to deploy SQL Server high availability solution to the cloud. You input your application requirements, including performance, number of nodes, and connectivity on the service console, and AWS Launch Wizard identifies the right AWS resources to deploy and run your SQL Server Always On application.

  • AWS Resource Groups help you to organize the resources that make up your application. With Resource Groups, you can manage and automate tasks on a large number of resources at one time. Only one Resource Group can be registered for a single application. For more information, see the AWS Resource Groups User Guide.

  • Amazon SQS offers a secure, durable, and available hosted queue that allows you to integrate and decouple distributed software systems and components. For more information, see the Amazon SQS User Guide.

  • AWS Step Functions is a serverless function composer that allows you to sequence a variety of AWS services and resources, including AWS Lambda functions, into structured, visual workflows. For more information, see the AWS Step Functions User Guide.

  • AWS SSM OpsCenter aggregates and standardizes OpsItems across services while providing contextual investigation data about each OpsItem, related OpsItems, and related resources. OpsCenter also provides Systems Manager Automation documents (runbooks) that you can use to quickly resolve issues. You can specify searchable, custom data for each OpsItem. You can also view automatically-generated summary reports about OpsItems by status and source. For more information, see the AWS Systems Manager User Guide.

  • Amazon API Gateway is an AWS service for creating, publishing, maintaining, monitoring, and securing REST, HTTP, and WebSocket APIs at any scale. API developers can create APIs that access AWS or other web services, as well as data stored in the AWS Cloud. For more information, see the Amazon API Gateway User Guide.

    Note

    Application Insights supports only REST API protocols (v1 of the API Gateway service).

  • Amazon Elastic Container Service (Amazon ECS) is a fully managed container orchestration service. You can use Amazon ECS to run your most sensitive and mission-critical applications. For more information, see the Amazon Elastic Container Service Developer Guide.

  • Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service that you can use to run Kubernetes on AWS without having to install, operate, and maintain your own Kubernetes control plane or nodes. Kubernetes is an open-source system for automating the deployment, scaling, and management of containerized applications. For more information, see the Amazon EKS User Guide.

  • Kubernetes on Amazon EC2. Kubernetes is open-source software that helps you deploy and manage containerized applications at scale. Kubernetes manages clusters of Amazon EC2 compute instances and runs containers on those instances with processes for deployment, maintenance, and scaling. With Kubernetes, you can run any type of containerized application with the same toolset on-premises and in the cloud. For more information, see Kubernetes Documentation: Getting started.

  • Amazon FSx helps you to launch and run popular file systems that are fully managed by AWS. With Amazon FSx, you can leverage the feature sets and performance of common open source and commercially-licensed file systems to avoid time-consuming administrative tasks. For more information, see the Amazon FSx Documentation.

  • Amazon Simple Notification Service (SNS) is a fully-managed messaging service for both application-to-application and application-to-person communication. You can configure Amazon SNS for monitoring by Application Insights. When Amazon SNS is configured as a resource for monitoring, Application Insights tracks SNS metrics to help determine why SNS messages may encounter issues or fail.

  • Amazon Elastic File System (Amazon EFS) is a fully-managed elastic NFS file system for use with AWS Cloud services and on-premises resources. It is built to scale to petabytes on demand without disrupting applications. It grows and shrinks automatically as you add and remove files, which eliminates the need to provision and manage capacity to accommodate growth. For more information, see the Amazon Elastic File System documentation.

Related third-party services
  • For some workloads and applications monitored in Application Insights, Prometheus JMX exporter is installed using AWS Systems Manager Distributor so that CloudWatch Application Insights can retrieve Java-specific metrics. When you choose to monitor a Java application, Application Insights automatically installs the Prometheus JMX exporter for you.

Supported application components

CloudWatch Application Insights scans your resource group to identify application components. Components can be standalone, auto-grouped (such as instances in an Auto Scaling group or behind a load balancer), or custom (by grouping together individual Amazon EC2 instances).

The following components are supported by CloudWatch Application Insights:

AWS components
  • Amazon EC2

  • Amazon EBS

  • Amazon RDS

  • Elastic Load Balancing: Application Load Balancer and Classic Load Balancer (all target instances of these load balancers are identified and configured).

  • Amazon EC2 Auto Scaling groups: AWS Auto Scaling (Auto Scaling groups are dynamically configured for all target instances; if your application scales up, CloudWatch Application Insights automatically configures the new instances). Auto Scaling groups are not supported for CloudFormation stack-based resource groups.

  • AWS Lambda

  • Amazon Simple Queue Service (Amazon SQS)

  • Amazon DynamoDB table

  • Amazon S3 bucket metrics

  • AWS Step Functions

  • Amazon API Gateway REST API stages

  • Amazon Elastic Container Service (Amazon ECS): cluster, service, and task

  • Amazon Elastic Kubernetes Service (Amazon EKS): cluster

  • Kubernetes on Amazon EC2: Kubernetes cluster running on EC2

  • Amazon SNS topic

Any other component type resources are not currently tracked by CloudWatch Application Insights. If a component type that is supported does not appear in your Application Insights application, the component may already be registered and managed by another application you own that is monitored by Application Insights.

Supported technology stacks

You can use CloudWatch Application Insights to monitor your applications running on Windows Server and Linux operating systems by selecting the application tier dropdown menu option for one of the following technologies:

  • Front‐end: Microsoft Internet Information Services (IIS) Web Server

  • Worker‐tier:

    • .NET Framework

    • .NET Core

  • Applications:

    • Java

    • SAP NetWeaver standard, distributed, and high availability deployments

  • Active Directory

  • SharePoint

  • Databases:

    • Microsoft SQL Server running on Amazon RDS or Amazon EC2 (including SQL Server High Availability configurations. See, Component configuration examples).

    • MySQL running on Amazon RDS, Amazon Aurora, or Amazon EC2

    • PostgreSQL running on Amazon RDS or Amazon EC2

    • Amazon DynamoDB table

    • Oracle running on Amazon RDS or Amazon EC2

    • SAP HANA database on a single Amazon EC2 instance and multiple EC2 instances

    • Cross-AZ SAP HANA database high availability setup

    • SAP Sybase ASE database on a single Amazon EC2 instance

    • Cross-AZ SAP Sybase ASE database high availability setup

If none of the technology stacks listed above apply to your application resources, you can monitor your application stack by choosing Custom from the application tier dropdown menu on the Manage monitoring page.