Set up ingestion from an existing Prometheus server in Kubernetes - Amazon Managed Service for Prometheus

Set up ingestion from an existing Prometheus server in Kubernetes

Amazon Managed Service for Prometheus supports ingesting metrics from Prometheus servers in clusters running Amazon EKS and in self-managed Kubernetes clusters running on Amazon EC2. The detailed instructions in this section are for a Prometheus server in an Amazon EKS cluster. The steps for a self-managed Kubernetes cluster on Amazon EC2 are the same, except that you will need to set up the OIDC provider and IAM roles for service accounts yourself in the Kubernetes cluster.

The instructions in this section use Helm as the Kubernetes package manager.

Step 1: Set up IAM roles for service accounts

For the method of onboarding that we are documenting, you need to use IAM roles for service accounts in the Amazon EKS cluster where the Prometheus server is running. These roles are also called service roles.

With service roles, you can associate an IAM role with a Kubernetes service account. This service account can then provide AWS permissions to the containers in any pod that uses that service account. For more information, see IAM roles for service accounts.

If you have not already set up these roles, follow the instructions at Set up service roles for the ingestion of metrics from Amazon EKS clusters to set up the roles.

Step 2: Upgrade your existing Prometheus server using Helm

The instructions in this section includes setting up remote write and sigv4 to authenticate and authorize the Prometheus server to remote write to your AMP workspace.

Using Prometheus Helm chart

Follow these steps if you are using a Helm chart with Prometheus Server image of version 2.26.0 or later.

To set up remote write from a Prometheus server using Helm chart

  1. Create a new remote write section in your Helm configuration file:

    • Replace ${IAM_PROXY_PROMETHEUS_ROLE_ARN} with the ARN of the amp-iamproxy-ingest-role that you created in Step 1: Set up IAM roles for service accounts. The role ARN should have the format of arn:aws:iam::your account ID:role/amp-iamproxy-ingest-role.

    • Replace ${WORKSPACE_ID} with your AMP workspace ID.

    • Replace ${AWS_REGION} with the Region of the AMP workspace (i.e. us-west-2).

    ## The following is a set of default values for prometheus server helm chart which enable remoteWrite to AMP ## For the rest of prometheus helm chart values see: https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml ## serviceAccounts: server: name: amp-iamproxy-ingest-service-account annotations: eks.amazonaws.com/role-arn: ${IAM_PROXY_PROMETHEUS_ROLE_ARN} server: remoteWrite: - url: https://aps-workspaces.${AWS_REGION}.amazonaws.com/workspaces/${WORKSPACE_ID}/api/v1/remote_write sigv4: region: ${AWS_REGION} queue_config: max_samples_per_send: 1000 max_shards: 200 capacity: 2500
  2. Update your existing Prometheus Server configuration using Helm:

    • Replace prometheus-chart-name with your Prometheus release name.

    • Replace prometheus-namespace with the Kubernetes namespace where your Prometheus Server is installed.

    • Replace my_prometheus_values_yaml with the path to your Helm configuration file.

    • Replace current_helm_chart_version with the current version of your Prometheus Server Helm chart. You can find the current chart version using helm list command.

    helm upgrade prometheus-chart-name prometheus-community/prometheus \ -n prometheus-namespace \ -f my_prometheus_values_yaml \ --version current_helm_chart_version