Collect Amazon MSK consumer-group metrics in CloudWatch

With an Amazon CloudWatch managed Prometheus collector and a managed Kafka exporter, you can collect consumer-group metrics from your Amazon MSK Provisioned clusters, such as consumer lag, committed offsets, and the number of members in each consumer group. The collector runs the exporter for you and delivers the metrics directly to CloudWatch, so you do not deploy or operate an exporter yourself. The collector connects to your cluster with AWS Identity and Access Management (IAM) authentication, and you grant it access with an Amazon MSK cluster policy.

Consumer-group metrics do not require Open Monitoring. If you also collect JMX or Node Exporter metrics, keep the jobs from Integrate Amazon MSK.

Prerequisites

Before you create a scraper, make sure that you have the following:

• An active Amazon MSK Provisioned cluster with Standard brokers and Apache Kafka version 3.6.0 or later. The cluster must have IAM authentication turned on and be in the same account and AWS Region as the scraper.

• At least two subnets in different Availability Zones in the cluster's VPC, with DNS enabled.

• An inbound rule on the cluster's security group that allows TCP traffic on port 9098 from the security group that you provide for the scraper.

• Permission to create a scraper and to read and change the cluster policy (kafka:GetClusterPolicy and kafka:PutClusterPolicy).

For additional caller-permission and AWS CLI requirements, see Prerequisites in the Amazon Managed Service for Prometheus User Guide.

This exporter does not support Amazon MSK Serverless. Use the metrics that Amazon MSK sends to CloudWatch for Serverless.

Step 1: Create a scraper with a CloudWatch destination

Save the following YAML scrape configuration as msk-consumer.yaml. Replace the cluster ARN. For an existing scraper, add this job to its full configuration and keep any existing jobs. Then update the scraper as described in Update or remove the exporter.

global:
  scrape_interval: 30s
scrape_configs:
  - job_name: kafka-exporter
    relabel_configs:
      - target_label: clusterArn
        replacement: 'arn:aws:kafka:us-west-2:111122223333:cluster/my-cluster/cluster-id'
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: kafka_consumergroup_(current_offset|members|lag)
        action: keep

Keep the job name kafka-exporter. The collector configures its scrape target automatically. The clusterArn label keeps metrics from different clusters apart when you query them. Its value must match the cluster ARN in your request. The metric_relabel_configs rule keeps three consumer-group metrics: lag, committed offsets, and members. The exporter can send more metrics, and Amazon MSK already provides some of that information. For details, see Create a scrape configuration.

Run the following AWS CLI command to create the scraper. Replace the Region, account ID, cluster ARN, subnets, security group, and both filters. The example selects topics that start with orders and consumer groups that start with payments. To select all topics or all consumer groups, omit the filter. Consumer-group metrics can also include other topics that a selected consumer group consumes. To control which metrics reach CloudWatch, use metric_relabel_configs in your scrape configuration.

aws amp create-scraper \
  --region us-west-2 \
  --alias msk-consumer-metrics \
  --source '{"vpcConfiguration":{"subnetIds":["subnet-subnet-id-1","subnet-subnet-id-2"],"securityGroupIds":["sg-security-group-id"]}}' \
  --exporters '[{"kafkaConfiguration":{"clusterArn":"arn:aws:kafka:us-west-2:111122223333:cluster/my-cluster/cluster-id","topicFilter":"^orders.*$","consumerGroupFilter":"^payments.*$"}}]' \
  --cli-binary-format base64 \
  --scrape-configuration "configurationBlob=$(base64 < msk-consumer.yaml | tr -d '\n')" \
  --destination '{"cloudWatchConfiguration":{"datasetArn":"arn:aws:cloudwatch:us-west-2:111122223333:dataset/default"}}'

Record the scraper ID. Check its status with the aws amp describe-scraper AWS CLI command, and continue when scraper.status.statusCode is ACTIVE. The scraper collects no Kafka metrics until you grant it access to the cluster in the next step.

Step 2: Grant the scraper access to the cluster

Add an Amazon MSK cluster policy that allows the scraper's service-linked role to connect to the cluster and to describe its topics and consumer groups. After you add the policy, collection starts on its own, and the first metrics can take a few minutes to appear. You do not need to update the scraper.

Grant the scraper's service-linked role. Do not use an account root principal or a role ARN with a wildcard. Amazon MSK accepts a root principal, but it grants the scraper nothing, and it rejects a wildcard role ARN.

To add the cluster policy

1. Get the scraper's role ARN:

   aws amp describe-scraper \
     --region us-west-2 \
     --scraper-id scraper-id \
     --query scraper.roleArn --output text

2. Save the following policy as policy.json. Replace the role ARN, Region, account ID, cluster name, and cluster ID.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": { "AWS": "scraper-role-arn" },
         "Action": "kafka-cluster:Connect",
         "Resource": "arn:aws:kafka:us-west-2:111122223333:cluster/my-cluster/cluster-id"
       },
       {
         "Effect": "Allow",
         "Principal": { "AWS": "scraper-role-arn" },
         "Action": "kafka-cluster:DescribeTopic",
         "Resource": "arn:aws:kafka:us-west-2:111122223333:topic/my-cluster/cluster-id/*"
       },
       {
         "Effect": "Allow",
         "Principal": { "AWS": "scraper-role-arn" },
         "Action": "kafka-cluster:DescribeGroup",
         "Resource": "arn:aws:kafka:us-west-2:111122223333:group/my-cluster/cluster-id/*"
       }
     ]
   }

3. Check whether the cluster already has a policy:

   aws kafka get-cluster-policy \
     --region us-west-2 \
     --cluster-arn arn:aws:kafka:us-west-2:111122223333:cluster/my-cluster/cluster-id

   A NotFoundException error means that the cluster has no policy.

4. If the cluster has no policy, add yours:

   aws kafka put-cluster-policy \
     --region us-west-2 \
     --cluster-arn arn:aws:kafka:us-west-2:111122223333:cluster/my-cluster/cluster-id \
     --policy file://policy.json

   If the cluster already has a policy, put-cluster-policy replaces it. The Policy value from get-cluster-policy is the existing policy as a JSON string. Add its statements to the Statement list in policy.json. Then run put-cluster-policy with --current-version and the CurrentVersion value from get-cluster-policy.

To grant every scraper in the account, to limit the topics and groups, or to see the full merge command, see Grant the scraper access to the cluster in the Amazon Managed Service for Prometheus User Guide.

Step 3: Validate metrics collection

To confirm that the collector is delivering consumer-group metrics, choose the scraper's Region and run each of the following queries separately in CloudWatch with Query Studio. Replace the cluster ARN.

kafka_consumergroup_members{"@instrumentation.cloudwatch.managed-collector.scraper-type"="msk", clusterArn="arn:aws:kafka:us-west-2:111122223333:cluster/my-cluster/cluster-id"}

kafka_consumergroup_current_offset{"@instrumentation.cloudwatch.managed-collector.scraper-type"="msk", clusterArn="arn:aws:kafka:us-west-2:111122223333:cluster/my-cluster/cluster-id"}

kafka_consumergroup_lag{"@instrumentation.cloudwatch.managed-collector.scraper-type"="msk", clusterArn="arn:aws:kafka:us-west-2:111122223333:cluster/my-cluster/cluster-id"}

A matching group must exist for its metrics to appear. The first metrics can take a few minutes to appear after you add the cluster policy.

If the scraper is ACTIVE but no kafka_consumergroup_* metrics appear under the service name kafka-exporter, check the cluster policy. Confirm that it grants the three actions to the scraper's current service-linked role ARN. If you re-created the scraper, its role ARN changed. If you use the account-wide policy, confirm that every statement keeps both StringEquals on aws:PrincipalAccount and ArnLike on aws:PrincipalArn. You do not need to update the scraper after you fix the policy.

For other checks, see Validate metrics collection. Those checks cover the cluster policy, caller permissions, IAM broker connectivity, collection filters, and metric labels. The up query on that page applies only to an Amazon Managed Service for Prometheus workspace destination. If the cluster policy is correct and metrics still do not appear on a cluster with many topics and consumer groups, try narrowing topicFilter and consumerGroupFilter.

For information about these metrics, including the -1 value when a group has no committed offset for a topic and partition, see Consumer-group metrics.

Update or remove the exporter

Omitting exporters in an update-scraper request keeps the current exporter. An empty list removes it. Removing the cluster policy does not stop collection right away. First remove the exporter or delete the scraper, and confirm with aws amp describe-scraper that the scraper status is ACTIVE and the exporter is gone, or that the scraper no longer exists. Then remove the policy statements. For the commands, see Update or remove the exporter.