Set up ingestion from an existing Prometheus server in Kubernetes - Amazon Managed Service for Prometheus

Set up ingestion from an existing Prometheus server in Kubernetes

Amazon Managed Service for Prometheus supports ingesting metrics from Prometheus servers in clusters running Amazon EKS and in self-managed Kubernetes clusters running on Amazon EC2. The detailed instructions in this section are for a Prometheus server in an Amazon EKS cluster. The steps for a self-managed Kubernetes cluster on Amazon EC2 are the same, except that you will need to set up the OIDC provider and IAM roles for service accounts yourself in the Kubernetes cluster.

The instructions in this section use Helm as the Kubernetes package manager.

Step 1: Set up IAM roles for service accounts

For the method of onboarding that we are documenting, you need to use IAM roles for service accounts in the Amazon EKS cluster where the Prometheus server is running. These roles are also called service roles.

With service roles, you can associate an IAM role with a Kubernetes service account. This service account can then provide AWS permissions to the containers in any pod that uses that service account. For more information, see IAM roles for service accounts.

If you have not already set up these roles, follow the instructions at Set up service roles for the ingestion of metrics from Amazon EKS clusters to set up the roles.

Step 2: Upgrade your existing Prometheus server using Helm

The instructions in this section includes setting up remote write and injecting the admission controller as a sidecar container to authenticate and authorize the Prometheus server to remote write to your AMP workspace.

Choose the appropriate section below for the version of Helm that you are using.

Using Prometheus Helm chart v13.0.0 or later

Follow these steps if you are using a Helm chart of version 13.0.0 or later.

To set up remote write from a Prometheus server

  1. On your Prometheus server, create a new remote write configuration. First, create a new update file. We will call the file amp_ingest_override_values.yaml.

    Add the following values to this YAML file.

    serviceAccounts: server: name: "amp-iamproxy-ingest-service-account" annotations: eks.amazonaws.com/role-arn: "${IAM_PROXY_PROMETHEUS_ROLE_ARN}" server: sidecarContainers: aws-sigv4-proxy-sidecar: image: public.ecr.aws/aws-observability/aws-sigv4-proxy:1.0 args: - --name - aps - --region - ${AWS_REGION} - --host - aps-workspaces.${AWS_REGION}.amazonaws.com - --port - :8005 ports: - name: aws-sigv4-proxy containerPort: 8005 statefulSet: enabled: "true" remoteWrite: - url: http://localhost:8005/workspaces/${WORKSPACE_ID}/api/v1/remote_write

    Replace ${AWS_REGION} with the Region of the AMP workspace.

    Replace ${IAM_PROXY_PROMETHEUS_ROLE_ARN} with the ARN of the amp-iamproxy-ingest-role that you created in Step 1: Set up IAM roles for service accounts. The role ARN should have the format of arn:aws:iam::your account ID:role/amp-iamproxy-ingest-role.

    Replace ${WORKSPACE_ID} with your workspace ID.

  2. Upgrade your Helm chart. First, find your Helm chart name by entering the following command. In the output from this command, look for a chart with a name that includes prometheus.

    helm ls --all-namespaces

    Then enter the following command.

    helm upgrade --install prometheus-chart-name prometheus-community/prometheus -n prometheus-namespace -f ./amp_ingest_override_values.yaml

    Replace prometheus-chart-name with the name of the Prometheus helm chart returned in the previous command.

Using Prometheus helm chart v8.4.6 to 12

Follow these steps if you are using a Helm chart of version 8.4.6 or later, but earlier than 13.0.0.

To set up remote write from a Prometheus server

  1. On your Prometheus server, create a new remote write configuration. First, create a new update file. We will call the file amp_ingest_override_values.yaml.

    Add the following values to the YAML file.

    serviceAccounts: server: name: "amp-iamproxy-ingest-service-account" annotations: eks.amazonaws.com/role-arn: "${SERVICE_ACCOUNT_IAM_INGEST_ROLE_ARN}" server: sidecarContainers: - name: aws-sigv4-proxy-sidecar image: public.ecr.aws/aws-observability/aws-sigv4-proxy:1.0 args: - --name - aps - --region - ${AWS_REGION} - --host - aps-workspaces.${AWS_REGION}.amazonaws.com - --port - :8005 ports: - name: aws-sigv4-proxy containerPort: 8005 statefulSet: enabled: "true" remoteWrite: - url: http://localhost:8005/workspaces/${WORKSPACE_ID}/api/v1/remote_write

    Replace ${AWS_REGION} with the Region of the AMP workspace.

    Replace ${SERVICE_ACCOUNT_IAM_INGEST_ROLE_ARN} with the ARN of the amp-iamproxy-ingest-role that you created in Step 1: Set up IAM roles for service accounts. The role ARN should have the format of arn:aws:iam::your account ID:role/amp-iamproxy-ingest-role.

    Replace ${WORKSPACE_ID} with your workspace ID.

  2. Upgrade your Helm chart. First, find your Helm chart name by entering the following command. In the output from this command, look for a chart with a name that includes prometheus.

    helm ls --all-namespaces

    Then enter the following command.

    helm upgrade --install helm-chart-name prometheus-community/prometheus -n prometheus-namespace -f ./amp_ingest_override_values.yaml

    Replace helm-chart-name with the name of the Prometheus helm chart returned in the previous command.

Downloading Helm charts

If you don't already have Helm charts downloaded locally, you can use the following command to download them.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm pull prometheus-community/prometheus --untar