Open monitoring with Prometheus
You can monitor your MSK cluster with Prometheus, an open-source monitoring
system for time-series metric data. You can publish this data to Amazon Managed Service
for Prometheus using Prometheus's remote write feature. You can also use tools that are
compatible with Prometheus-formatted metrics or tools that integrate with Amazon MSK
Open Monitoring, like Datadog
Creating an Amazon MSK cluster with open monitoring enabled
Using the AWS Management Console
Sign in to the AWS Management Console, and open the Amazon MSK console at https://console.aws.amazon.com/msk/home?region=us-east-1#/home/
. -
In the Monitoring section, select the check box next to Enable open monitoring with Prometheus.
-
Provide the required information in all the sections of the page, and review all the available options.
-
Choose Create cluster.
Using the AWS CLI
-
Invoke the create-cluster command and specify its
open-monitoring
option. Enable theJmxExporter
, theNodeExporter
, or both. If you specifyopen-monitoring
, the two exporters can't be disabled at the same time.
Using the API
-
Invoke the CreateCluster operation and specify
OpenMonitoring
. Enable thejmxExporter
, thenodeExporter
, or both. If you specifyOpenMonitoring
, the two exporters can't be disabled at the same time.
Enabling open monitoring for an existing Amazon MSK cluster
To enable open monitoring, make sure that the cluster is in the
ACTIVE
state.
Using the AWS Management Console
Sign in to the AWS Management Console, and open the Amazon MSK console at https://console.aws.amazon.com/msk/home?region=us-east-1#/home/
. -
Choose the name of the cluster that you want to update. This takes you to a page the contains details for the cluster.
-
On the Properties tab, scroll down to find the Monitoring section.
-
Choose Edit.
-
Select the check box next to Enable open monitoring with Prometheus.
-
Choose Save changes.
Using the AWS CLI
-
Invoke the update-monitoring command and specify its
open-monitoring
option. Enable theJmxExporter
, theNodeExporter
, or both. If you specifyopen-monitoring
, the two exporters can't be disabled at the same time.
Using the API
-
Invoke the UpdateMonitoring operation and specify
OpenMonitoring
. Enable thejmxExporter
, thenodeExporter
, or both. If you specifyOpenMonitoring
, the two exporters can't be disabled at the same time.
Setting up a Prometheus host on an Amazon EC2 instance
-
Download the Prometheus server from https://prometheus.io/download/#prometheus
to your Amazon EC2 instance. -
Extract the downloaded file to a directory and go to that directory.
-
Create a file with the following contents and name it
prometheus.yml
.# file: prometheus.yml # my global config global: scrape_interval: 60s # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. - job_name: 'prometheus' static_configs: # 9090 is the prometheus server port - targets: ['localhost:9090'] - job_name: 'broker' file_sd_configs: - files: - 'targets.json'
-
Use the ListNodes operation to get a list of your cluster's brokers.
-
Create a file named
targets.json
with the following JSON. Replacebroker_dns_1
,broker_dns_2
, and the rest of the broker DNS names with the DNS names you obtained for your brokers in the previous step. Include all of the brokers you obtained in the previous step. Amazon MSK uses port 11001 for the JMX Exporter and port 11002 for the Node Exporter.Note
To scrape JMX metrics from KRaft controllers, add controller DNS names as targets in the JSON file. For example:
controller_dns_1:11001
, replacingcontroller_dns_1
with the actual controller DNS name. -
To start the Prometheus server on your Amazon EC2 instance, run the following command in the directory where you extracted the Prometheus files and saved
prometheus.yml
andtargets.json
../prometheus
-
Find the IPv4 public IP address of the Amazon EC2 instance where you ran Prometheus in the previous step. You need this public IP address in the following step.
-
To access the Prometheus web UI, open a browser that can access your Amazon EC2 instance, and go to
, wherePrometheus-Instance-Public-IP
:9090Prometheus-Instance-Public-IP
is the public IP address you got in the previous step.
Prometheus metrics
All metrics emitted by Apache Kafka to JMX are accessible using open monitoring
with Prometheus. For information about Apache Kafka metrics, see Monitoringkafka.consumer.group:type=ConsumerLagMetrics
. You can also use the
Prometheus Node Exporter to get CPU and disk metrics for your brokers at port
11002.
Storing Prometheus metrics in Amazon Managed Service for Prometheus
Amazon Managed Service for Prometheus is a Prometheus-compatible monitoring and alerting service that you can use to monitor Amazon MSK clusters. It is a fully-managed service that automatically scales the ingestion, storage, querying, and alerting of your metrics. It also integrates with AWS security services to give you fast and secure access to your data. You can use the open-source PromQL query language to query your metrics and alert on them.
For more information, see Getting started with Amazon Managed Service for Prometheus.