Amazon Managed Streaming for Apache Kafka
Developer Guide

The AWS Documentation website is getting a new look!
Try it now and let us know what you think. Switch to the new look >>

You can return to the original look by selecting English in the language selector above.

Monitoring Amazon MSK with Amazon CloudWatch

Amazon MSK gathers Apache Kafka metrics and sends them to Amazon CloudWatch where you can view them. For more information about Kafka metrics, including the ones that Amazon MSK surfaces, see Monitoring in the Apache Kafka documentation.

Amazon MSK Monitoring Levels

When creating an Amazon MSK cluster, you can set the enhancedMonitoring property to one of three monitoring levels: DEFAULT, PER_BROKER, or PER_TOPIC_PER_BROKER. The tables in the following section show all the metrics that Amazon MSK makes available starting at each monitoring level.

Amazon MSK Metrics

Amazon MSK integrates with Amazon CloudWatch metrics so that you can collect, view, and analyze CloudWatch metrics for your Amazon MSK cluster. The metrics that you configure for your MSK cluster are automatically collected and pushed to CloudWatch. The following three tables show the metrics that become available at each of the three monitoring levels.

DEFAULT Level Monitoring

The metrics described in the following table are available at the DEFAULT monitoring level. They are free.

Metrics available at the DEFAULT monitoring level

Name When Visible Dimensions Description

ZooKeeperRequestLatencyMsMean

After the cluster gets to the ACTIVE state. Cluster Name, Broker ID Mean latency in milliseconds for ZooKeeper requests from broker.
ZooKeeperSessionState After the cluster gets to the ACTIVE state. Cluster Name, Broker ID Connection status of broker's ZooKeeper session which may be one of the following: NOT_CONNECTED: '0.0', ASSOCIATING: '0.1', CONNECTING: '0.5', CONNECTEDREADONLY: '0.8', CONNECTED: '1.0', CLOSED: '5.0', AUTH_FAILED: '10.0'.
ActiveControllerCount After the cluster gets to the ACTIVE state. Cluster Name Only one controller per cluster should be active at any given time.
GlobalPartitionCount After the cluster gets to the ACTIVE state. Cluster Name Total number of partitions across all brokers in the cluster.
GlobalTopicCount After the cluster gets to the ACTIVE state. Cluster Name Total number of topics across all brokers in the cluster.
OfflinePartitionsCount After the cluster gets to the ACTIVE state. Cluster Name Total number of partitions that are offline in the cluster.

SwapUsed

After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The size in bytes of swap memory that is in use for the broker.
SwapFree After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The size in bytes of swap memory that is available for the broker.
MemoryUsed After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The size in bytes of memory that is in use for the broker.
MemoryBuffered After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The size in bytes of buffered memory for the broker.
MemoryFree After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The size in bytes of memory that is free and available for the broker.
MemoryCached After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The size in bytes of cached memory for the broker.
CpuUser After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The percentage of CPU in user space.
CpuSystem After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The percentage of CPU in kernel space.
CpuIdle After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The percentage of CPU idle time.
RootDiskUsed After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The percentage of the root disk used by the broker.
KafkaAppLogsDiskUsed After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The percentage of disk space used for application logs.
KafkaDataLogsDiskUsed After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The percentage of disk space used for data logs.
NetworkRxErrors After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The number of network receive errors for the broker.
NetworkTxErrors After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The number of network transmit errors for the broker.
NetworkRxDropped After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The number of dropped receive packages.
NetworkTxDropped After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The number of dropped transmit packages.
NetworkRxPackets After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The number of packets received by the broker.
NetworkTxPackets After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The number of packets transmitted by the broker.

PER_BROKER Level Monitoring

When you set the monitoring level to PER_BROKER, you get the metrics described in the following table in addition to all the DEFAULT level metrics. You pay for the metrics in the following table, whereas the DEFAULT level metrics continue to be free.

Additional metrics that are available starting at the PER_BROKER monitoring level

Name When Visible Dimensions Description
MessagesInPerSec After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The number of incoming messages per second for the broker.
NetworkProcessorAvgIdlePercent After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The average percentage of the time the network processors are idle.
RequestHandlerAvgIdlePercent After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The average percentage of the time the request handler threads are idle.
LeaderCount After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The number of leader replicas.
PartitionCount After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The number of partitions for the broker.
ProduceLocalTimeMsMean After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The mean time in milliseconds for the follower to send a response.
ProduceMessageConversionsTimeMsMean After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The mean time in milliseconds spent on message format conversions.
ProduceRequestQueueTimeMsMean After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The mean time in milliseconds that request messages spend in the queue.
ProduceResponseQueueTimeMsMean After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The mean time in milliseconds that response messages spend in the queue.
ProduceResponseSendTimeMsMean After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The mean time in milliseconds spent on sending response messages.
ProduceTotalTimeMsMean After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The mean produce time in milliseconds.
RequestBytesMean After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The mean number of request bytes for the broker.
UnderMinIsrPartitionCount After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The number of under minIsr partitions for the broker.
UnderReplicatedPartitions After the cluster gets to the ACTIVE state. Cluster Name, Broker ID The number of under-replicated partitions for the broker.
BytesInPerSec After you create a topic. Cluster Name, Broker ID The number of bytes per second received from clients.
BytesOutPerSec After you create a topic. Cluster Name, Broker ID The number of bytes per second sent to clients.
MessagesInPerSec After you create a topic. Cluster Name, Broker ID The number of messages received from clients per second.
FetchMessageConversionsPerSec After you create a topic. Cluster Name, Broker ID The number of fetch message conversions per second for the broker.
ProduceMessageConversionsPerSec After you create a topic. Cluster Name, Broker ID The number of produce message conversions per second for the broker.
FetchConsumerTotalTimeMsMean After there's a producer/consumer. Cluster Name, Broker ID The mean total time in milliseconds that consumers spend on fetching data from the broker.
FetchFollowerTotalTimeMsMean After there's a producer/consumer. Cluster Name, Broker ID The mean total time in milliseconds that followers spend on fetching data from the broker.
FetchConsumerRequestQueueTimeMsMean After there's a producer/consumer. Cluster Name, Broker ID The mean time in milliseconds that the consumer request waits in the request queue.
FetchFollowerRequestQueueTimeMsMean After there's a producer/consumer. Cluster Name, Broker ID The mean time in milliseconds that the follower request waits in the request queue.
FetchConsumerLocalTimeMsMean After there's a producer/consumer. Cluster Name, Broker ID The mean time in milliseconds that the consumer request is processed at the leader.
FetchFollowerLocalTimeMsMean After there's a producer/consumer. Cluster Name, Broker ID The mean time in milliseconds that the follower request is processed at the leader.
FetchConsumerResponseQueueTimeMsMean After there's a producer/consumer. Cluster Name, Broker ID The mean time in milliseconds that the consumer request waits in the response queue.
FetchFollowerResponseQueueTimeMsMean After there's a producer/consumer. Cluster Name, Broker ID The mean time in milliseconds that the follower request waits in the response queue.
FetchConsumerResponseSendTimeMsMean After there's a producer/consumer. Cluster Name, Broker ID The mean time in milliseconds for the consumer to send a response.
FetchFollowerResponseSendTimeMsMean After there's a producer/consumer. Cluster Name, Broker ID The mean time in milliseconds for the follower to send a response.
ProduceThrottleTime After bandwidth throttling is applied. Cluster Name, Broker ID The average produce throttle time in milliseconds.
ProduceThrottleByteRate After bandwidth throttling is applied. Cluster Name, Broker ID The number of throttled bytes per second.
ProduceThrottleQueueSize After bandwidth throttling is applied. Cluster Name, Broker ID The number of messages in the throttle queue.
FetchThrottleTime After bandwidth throttling is applied. Cluster Name, Broker ID The average fetch throttle time in milliseconds.
FetchThrottleByteRate After bandwidth throttling is applied. Cluster Name, Broker ID The number of throttled bytes per second.
FetchThrottleQueueSize After bandwidth throttling is applied. Cluster Name, Broker ID The number of messages in the throttle queue.
RequestThrottleTime After request throttling is applied. Cluster Name, Broker ID The average request throttle time in milliseconds.
RequestTime After request throttling is applied. Cluster Name, Broker ID The average time spent in broker network and I/O threads to process requests.
RequestThrottleQueueSize After request throttling is applied. Cluster Name, Broker ID The number of messages in the throttle queue.
RequestExemptFromThrottleTime After request throttling is applied. Cluster Name, Broker ID The average time spent in broker network and I/O threads to process requests that are exempt from throttling.

PER_TOPIC_PER_BROKER Level Monitoring

When you set the monitoring level to PER_TOPIC_PER_BROKER, you get the metrics described in the following table, in addition to all the metrics from the PER_BROKER and DEFAULT levels. Only the DEFAULT level metrics are free.

Additional metrics that are available starting at the PER_TOPIC_PER_BROKER monitoring level

Name When Visible Dimensions Description
BytesInPerSec After you create a topic.

Cluster Name, Broker ID, Topic

The number of bytes received per second.
BytesOutPerSec After you create a topic. Cluster Name, Broker ID, Topic The number of bytes sent per second.
MessagesInPerSec After you create a topic. Cluster Name, Broker ID, Topic The number of messages received per second.
FetchMessageConversionsPerSec After you create a topic. Cluster Name, Broker ID, Topic The number of fetched messages converted per second.
ProduceMessageConversionsPerSec After you create a topic. Cluster Name, Broker ID, Topic The number of conversions per second for produced messages.

Viewing Amazon MSK Metrics Using Amazon CloudWatch

You can monitor metrics for Amazon MSK using the CloudWatch console, the command line, or the CloudWatch API. The following procedures show you how to access metrics using these different methods.

To access metrics using the CloudWatch console

Sign in to the AWS Management Console and open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

  1. In the navigation pane, choose Metrics.

  2. Choose the All metrics tab, and then choose AWS/Kafka.

  3. To view topic-level metrics, choose Topic, Broker ID, Cluster Name; for broker-level metrics, choose Broker ID, Cluster Name; and for cluster-level metrics, choose Cluster Name.

  4. (Optional) In the graph pane, select a statistic and a time period, and then create a CloudWatch alarm using these settings.

To access metrics using the AWS CLI

Use the list-metrics and get-metric-statistics commands.

To access metrics using the CloudWatch CLI

Use the mon-list-metrics and mon-get-stats commands.

To access metrics using the CloudWatch API

Use the ListMetrics and GetMetricStatistics operations.

Consumer Lag Checking with Burrow

Burrow is a monitoring companion for Apache Kafka that provides consumer lag checking. Burrow has a modular design that includes the following subsystems:

  • Clusters run an Apache Kafka client that periodically updates topic lists and the current HEAD offset (the most recent offset) for every partition.

  • Consumers fetch information about consumer groups from a repository. This repository can be an Apache Kafka cluster (consuming the __consumer_offsets topic), ZooKeeper, or some other repository.

  • The storage subsystem stores all of this information in Burrow.

  • The evaluator subsystem retrieves information from the storage subsystem for a specific consumer group and calculates the status of that group. This follows the consumer lag evaluation rules.

  • The notifier subsystem requests status on consumer groups according to a configured interval and sends out notifications (Email, HTTP, or some other method) for groups that meet the configured criteria.

  • The HTTP Server subsystem provides an API interface to Burrow for fetching information about clusters and consumers.

For more information about Burrow, see Burrow - Kafka Consumer Lag Checking.

To set up and use Burrow with Amazon MSK

  1. Create an MSK cluster and launch a client machine in the same VPC as the cluster. For example, you can follow the instructions at Getting Started Using Amazon MSK.

  2. Run the following command on the EC2 instance that serves as your client machine.

    sudo yum install go
  3. Run the following command on the client machine to get the Burrow project t.

    go get github.com/linkedin/Burrow
  4. Run the following command to install dep. It installs it in the /home/ec2-user/go/bin/dep folder.

    curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
  5. Go to the /home/ec2-user/go/src/github.com/linkedin/Burrow folder and run the following command.

    /home/ec2-user/go/bin/dep ensure
  6. Run the following command in the same folder.

    go install
  7. Open the /home/ec2-user/go/src/github.com/linkedin/Burrow/config/burrow.toml configuration file for editing. In the following sections of the configuration file, replace the placeholders with the name of your MSK cluster, the host:port pairs for your ZooKeeper servers, and your bootstrap brokers.

    To get your ZooKeeper host:port pairs, describe your MSK cluster and look for the value of ZookeeperConnectString. See Get the ZooKeeper Connection String for an Amazon MSK Cluster.

    To get your bootstrap brokers, see Get the Bootstrap Brokers for an Amazon MSK Cluster.

    Follow the formatting shown below when you edit the configuration file.

    [zookeeper] servers=[ "ZooKeeper-host-port-pair-1", "ZooKeeper-host-port-pair-2", "ZooKeeper-host-port-pair-3" ] timeout=6 root-path="/burrow" [client-profile.test] client-id="burrow-test" kafka-version="0.10.0" [cluster.MSK-cluster-name] class-name="kafka" servers=[ "bootstrap-broker-host-port-pair-1", "bootstrap-broker-host-port-pair-2", "bootstrap-broker-host-port-pair-3" ] client-profile="test" topic-refresh=120 offset-refresh=30 [consumer.MSK-cluster-name] class-name="kafka" cluster="MSK-cluster-name" servers=[ "bootstrap-broker-host-port-pair-1", "bootstrap-broker-host-port-pair-2", "bootstrap-broker-host-port-pair-3" ] client-profile="test" group-blacklist="^(console-consumer-|python-kafka-consumer-|quick-).*$" group-whitelist="" [consumer.MSK-cluster-name_zk] class-name="kafka_zk" cluster="MSK-cluster-name" servers=[ "ZooKeeper-host-port-pair-1", "ZooKeeper-host-port-pair-2", "ZooKeeper-host-port-pair-3" ] zookeeper-path="/kafka-cluster" zookeeper-timeout=30 group-blacklist="^(console-consumer-|python-kafka-consumer-|quick-).*$" group-whitelist=""
  8. In the go/bin folder run the following command.

    ./Burrow --config-dir /home/ec2-user/go/src/github.com/linkedin/Burrow/config
  9. Check for errors in the bin/log/burrow.log file.

  10. You can use the following command to test your setup.

    curl -XGET 'HTTP://your-localhost-ip:8000/v3/kafka'
  11. For all of the supported HTTP requests and links, see Burrow HTTP Endpoint.