Send high-availability data from your Kubernetes cluster (Deduplication) - Amazon Managed Service for Prometheus

Send high-availability data from your Kubernetes cluster (Deduplication)

Amazon Managed Service for Prometheus supports the use of multiple Prometheus instances that monitor the same set of metrics and send them to a single AMP workspace, for high-availability purposes. AMP deduplicates the metrics. Each Prometheus instance can be either a Prometheus server or an AWS Open Distro for Telemetry agent.

AMP sets one instance as the leader replica and ingests from only that replica. If AMP stops receiving samples from the leader for a 30-second period, it automatically switches to another leader and begins ingesting metrics from the new leader.

To set up a high-availability configuration, you need to specify external labels on all of the instances of a high-availability group to identify them to AMP. Use the cluster label to identify a Prometheus instance agent as part of a high-availability group, and use the __replica__ label to separately identify each replica in the group. For example, on one replica of a pair called prom-team1, you would include the following external labels.

cluster: prom-team1 __replica__: replica1

On the other, you would use the following.

cluster: prom-team1 __replica__: replica2
Note

In the __replica__ external labels, there are two underscore characters before the word replica and two underscore characters after it.

When AMP stores samples from the high-availability replicas with these labels, it strips the replica label when the sample is accepted. This means that you’ll only have a 1:1 series mapping for your current series instead of a series per replica. The cluster label is kept.