Open Monitoring with Prometheus
You can monitor your MSK cluster with Prometheus, an open-source monitoring
system for time-series metric data. You can also use tools that are compatible with
Prometheus-formatted metrics or tools that integrate with Amazon MSK Open Monitoring,
like Datadog
Creating an Amazon MSK Cluster with Open Monitoring Enabled
Using the AWS Management Console
-
Sign in to the AWS Management Console, and open the Amazon MSK console at https://console.aws.amazon.com/msk/home?region=us-east-1#/home/
. -
In the Monitoring section, select the check box next to Enable open monitoring with Prometheus.
-
Provide the required information in all the sections of the page, and review all the available options.
-
Choose Create cluster.
Using the AWS CLI
-
Invoke the create-cluster command and specify its
open-monitoring
option. Enable theJmxExporter
, theNodeExporter
, or both. If you specifyopen-monitoring
, the two exporters can't be disabled at the same time.
Using the API
-
Invoke the CreateCluster operation and specify
OpenMonitoring
. Enable thejmxExporter
, thenodeExporter
, or both. If you specifyOpenMonitoring
, the two exporters can't be disabled at the same time.
Enabling Open Monitoring for an Existing Amazon MSK Cluster
To enable open monitoring, make sure that the cluster is in the ACTIVE
state.
Using the AWS Management Console
-
Sign in to the AWS Management Console, and open the Amazon MSK console at https://console.aws.amazon.com/msk/home?region=us-east-1#/home/
. -
Choose the name of the cluster that you want to update. This takes you to the Details page for the cluster.
-
On the Details tab, scroll down to find the Monitoring section.
-
Choose Edit.
-
Select the check box next to Enable open monitoring with Prometheus.
-
Choose Save changes.
Using the AWS CLI
-
Invoke the update-monitoring command and specify its
open-monitoring
option. Enable theJmxExporter
, theNodeExporter
, or both. If you specifyopen-monitoring
, the two exporters can't be disabled at the same time.
Using the API
-
Invoke the UpdateMonitoring operation and specify
OpenMonitoring
. Enable thejmxExporter
, thenodeExporter
, or both. If you specifyOpenMonitoring
, the two exporters can't be disabled at the same time.
Setting Up a Prometheus Host on an Amazon EC2 Instance
-
Download the Prometheus server from https://prometheus.io/download/#prometheus
to your Amazon EC2 instance. -
Extract the downloaded file to a directory and go to that directory.
-
Create a file with the following contents and name it
prometheus.yml
.# file: prometheus.yml # my global config global: scrape_interval: 10s # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. - job_name: 'prometheus' static_configs: # 9090 is the prometheus server port - targets: ['localhost:9090'] - job_name: 'broker' file_sd_configs: - files: - 'targets.json'
-
Use the ListNodes operation to get a list of your cluster's brokers.
-
Create a file named
targets.json
with the following JSON. Replacebroker_dns_1
,broker_dns_2
, and the rest of the broker DNS names with the DNS names you obtained for your brokers in the previous step. Include all of the brokers you obtained in the previous step. Amazon MSK uses port 11001 for the JMX Exporter and port 11002 for the Node Exporter.[ { "labels": { "job": "jmx" }, "targets": [ "
broker_dns_1
:11001", "broker_dns_2
:11001", . . . "broker_dns_N
:11001" ] }, { "labels": { "job": "node" }, "targets": [ "broker_dns_1
:11002", "broker_dns_2
:11002", . . . "broker_dns_N
:11002" ] } ] -
To start the Prometheus server on your Amazon EC2 instance, run the following command in the directory where you extracted the Prometheus files and saved
prometheus.yml
andtargets.json
../prometheus
-
Find the IPv4 public IP address of the Amazon EC2 instance where you ran Prometheus in the previous step. You need this public IP address in the following step.
-
To access the Prometheus web UI, open a browser that can access your Amazon EC2 instance, and go to
, wherePrometheus-Instance-Public-IP
:9090Prometheus-Instance-Public-IP
is the public IP address you got in the previous step.
Prometheus Metrics
All metrics emitted by Apache Kafka to JMX are accessible using open monitoring
with Prometheus. For information about Apache Kafka metrics, see Monitoringkafka.consumer.group:type=ConsumerLagMetrics
. You can also use the
Prometheus Node Exporter to get CPU and disk metrics for your brokers at port
11002.