Monitoring performance and operations with Amazon CloudWatch - Scalable Analytics Using Apache Druid on AWS
DashboardAlarmsLogsMetrics

Monitoring performance and operations with Amazon CloudWatch

The solution captures all the Druid data logs in Amazon CloudWatch for monitoring purposes. This includes alarms, logs and a dashboard for reporting purposes.

Amazon CloudWatch gives you an application-level view into this solution and its resources so that you can:

  • Monitor alarms, logs for your deployed clusters, and metrics associated with this solution from a central location.

  • View operations data for the solution’s AWS resources (such as deployment status, Amazon CloudWatch alarms, resource configurations, and operational issues) in the context of an application.

Sign in to the AWS Management Console, and navigate to CloudWatch. Use the left hand navigation to view this data for your Druid deployment.

Note

You must activate the CloudWatch Application Insights before you can use CloudWatch to monitor any alarms, logs, or dashboards for the solution. For more information, refer to the Activate CloudWatch Application Insights section.

Dashboard

  1. From the left, select CloudWatch > Dashboards.

  2. On the Custom Dashboards tab, click to select the dashboard you want to view. For example, druid-ec2-custom-name-ops-dashboard.

CloudWatch dashboard for Scalable Analytics using Apache Druid on AWS.

CloudWatch dashboard for Scalable Analytics using Apache Druid on AWS

The dashboard provides the following information for your Druid deployment.

Item Description
Canary status Displays the availability and latency of your web services and allows you to discover and troubleshoot any issues.
Application Load Balancer (ALB) – Key Performance Indicators

Provides the following data in relation to your ALB application:

  • Request Count

  • Target Response Time

  • HTTP Connection Count

  • Response Code Count

Druid – Key Performance Indicators

Provides the following data in relation to Druid core parameters:

  • Deep Storage

  • Ingestion Count

  • Query Count

  • Query Time

Druid ZooKeeper – Key Performance Indicators

Displays the following data in relation to the Zookeeper cluster state management:

  • CPU Utilization (%)

  • Network In/Out (bytes)

  • Memory Utilization (%)

  • Disk Utilization (%)

Druid data

Displays the following data in relation to the Druid ingestion jobs and queryable data:

  • CPU Utilization (%)

  • Network In/Out (bytes)

  • Memory Utilization (%)

  • Disk Utilization (%)

Druid query

Displays the following data in relation to the Druid queries:

  • CPU Utilization (%)

  • Network In/Out (bytes)

  • Memory Utilization (%)

  • Disk Utilization (%)

Druid master

Displays the following data in relation to the Druid data ingestion and availability:

  • CPU Utilization (%)

  • Network In/Out (bytes)

  • Memory Utilization (%)

  • Disk Utilization (%)

Aurora Cluster

Provides the following data in relation to the Druid databases:

  • CPU Utilization (%)

  • Database connections

  • Throughput

  • Free Memory

Alarms

Amazon CloudWatch provides detailed alarms for your Druid deployment, and displays the different states, conditions, and relevant actions associated with these alarms. To view the Alarms page, from the left, select CloudWatch > Alarms.

For more information about alarms, alarm states, actions, and configuring alarms, refer to the Amazon CloudWatch alarms page.

Logs

You can use Amazon CloudWatch Logs to monitor, store, and access your log files from Amazon Elastic Compute Cloud (Amazon EC2) instances, and other sources. CloudWatch Logs allow you to centralize the logs from all of your Druid solution components, applications, and AWS services, in a single, highly scalable service.

To view logs for the Druid deployment, from the left, select CloudWatch > logs > Log groups.

For more information about working with log groups and log streams, refer to the AWS CloudWatch logs page.

Metrics

In addition to logs and alarms, CloudWatch allows to view, and analyze data about the performance of your Druid deployment via custom namespaces.

To view metrics for the Druid deployment:

  1. From the left, select CloudWatch > Metrics > All metrics.

  2. On the Metrics page, under Custom namespaces, select AWSSolutions/Druid. This will display all metrics for the Druid deployment.

    Screenshot of metrics for Scalable Analytics using Apache Druid on AWS in CloudWatch.

    Metrics for Scalable Analytics using Apache Druid on AWS

  3. Choose to select a relevant dimension to view additional information. The dimensions page provides a breakdown of individual Druid services, source, query, and other metrics.