FAQ: High availability configuration - Amazon Managed Service for Prometheus

FAQ: High availability configuration

Should I include the value __replica__ into another label to track the sample points?

In a high availability setting, Amazon Managed Service for Prometheus ensures data samples are not duplicated by electing a leader in the cluster of Prometheus instances. If the leader replica stops sending data samples for 30 seconds, Amazon Managed Service for Prometheus automatically makes another Prometheus instance a leader replica and ingests data from the new leader, including any missed data. Therefore, the answer is no, it is not recommended.  Doing so may cause issues like:

  • Querying a count in PromQL may return higher than expected value during the period of electing a new leader.

  • The number of active series gets increased during a period of electing a new leader and it reaches the active series limits. See AMP Quotas for more info.