Integrating with Amazon EKS cost monitoring - Amazon Managed Service for Prometheus

Integrating with Amazon EKS cost monitoring

Amazon Managed Service for Prometheus integrates with Amazon EKS cost monitoring (with Kubecost) to perform cost allocation calculations and provide insights into optimizing your Kubernetes clusters. Using Amazon Managed Service for Prometheus with Kubecost, you can reliably scale your cost monitoring to support larger clusters.

This section describes how to set up Kubecost and Amazon Managed Service for Prometheus to an existing Amazon EKS cluster.

Prerequisites

Before starting to integrate Kubecost and Amazon Managed Service for Prometheus with your Amazon EKS cluster, you must have the following prerequisites.

  • You must have an existing AWS account and IAM credentials to create Amazon Managed Service for Prometheus and IAM roles programmatically.

    Note

    For more information about creating an AWS account and IAM credentials, see Setting up.

  • You must have AWS CLI and eksctl installed.

  • You must have an existing Amazon EKS cluster with OpenID Connect (OIDC) enabled.

    Note

    If you do not have OIDC enabled, you can use the following command to enable it. Remember to replace the YOUR_CLUSTER_NAME and AWS_REGION with the correct values for your account.

    eksctl utils associate-iam-oidc-provider \ --cluster ${YOUR_CLUSTER_NAME} --region ${AWS_REGION} \ --approve

    For more information about using OIDC with Amazon EKS, see OIDC identity provider authentication and Creating an IAM OIDC provider in the Amazon EKS Use Guide.

Setting up Amazon Managed Service for Prometheus for integration with Kubecost

Before you set up Kubecost to monitor your Amazon EKS cluster, you must have an Amazon Managed Service for Prometheus workspace. If you do not already have an Amazon Managed Service for Prometheus workspace, you can use the following AWS CLI command to create one. Remember to replace the kubecost-amp and AWS_REGION with the appropriate values for your account.

Note

For more information about creating and managing an Amazon Managed Service for Prometheus workspace, see Create a workspace.

aws amp create-workspace --alias kubecost-amp --region $AWS_REGION

This will create a workspace for you. The output of the command will look like the following.

{ "arn": "arn:aws:aps:AWS_REGION:<AWS account>:workspace/${AMP_WORKSPACE_ID}", "status": { "statusCode": "CREATING" }, "tags": {}, "workspaceId": "${AMP_WORKSPACE_ID}" }

The values for AWS_REGION, <AWS account>, and AMP_WORKSPACE_ID will be the actual values used to create your workspace. The workspace will take a few seconds to create. To get more information about your workspace, or to view this information later, you can log into the Amazon Managed Service for Prometheus console at https://console.aws.amazon.com/prometheus/. You will need the AWS Region and Amazon Managed Service for Prometheus workspace ID for use later in this section.

Setting up Kubecost

Now you are ready to install Kubecost. This section is about how to install Kubecost to use Amazon Managed Service for Prometheus to monitor your Amazon EKS cluster. You can also refer to the instructions for Cost monitoring in the Amazon EKS User Guide.

Note

The following assumes that you have Helm (version 3.9 or later) and kubectl installed.

To set up Kubecost

  1. Install Kubecost from the Amazon ECR Public Gallery using the following command. You can replace 1.97.0 with a later version. You can see the available versions at kubecost/cost-analyzer in the Amazon ECR Public Gallery.

    helm upgrade -i kubecost \ oci://public.ecr.aws/kubecost/cost-analyzer --version 1.97.0 \ --namespace kubecost --create-namespace \ -f https://tinyurl.com/kubecost-amazon-eks
  2. Set up IAM roles with AWS managed policies for the kubecost service accounts kubecost-cost-analyzer and kubecost-prometheus-server with the following commands. Remember to replace YOUR_CLUSTER_NAME and AWS_REGION with the correct values for your account.

    eksctl create iamserviceaccount \ --name kubecost-cost-analyzer \ --namespace kubecost \ --cluster ${YOUR_CLUSTER_NAME} --region ${AWS_REGION} \ --attach-policy-arn arn:aws:iam::aws:policy/AmazonPrometheusQueryAccess \ --attach-policy-arn arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess \ --override-existing-serviceaccounts \ --approve eksctl create iamserviceaccount \ --name kubecost-prometheus-server \ --namespace kubecost \ --cluster ${YOUR_CLUSTER_NAME} --region ${AWS_REGION} \ --attach-policy-arn arn:aws:iam::aws:policy/AmazonPrometheusQueryAccess \ --attach-policy-arn arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess \ --override-existing-serviceaccounts \ --approve

    To learn more about IAM policies, see How Amazon Managed Service for Prometheus works with IAM.

  3. Now you can configure Kubecost to use Amazon Managed Service for Prometheus as the source for metrics about your cluster. You will need to create two values for this--a URL for Kubecost to query metrics and a URL for Kubecost to write calculated metrics to Amazon Managed Service for Prometheus.

    These two URLs are constructed from the workspace ID that you received when creating the Amazon Managed Service for Prometheus workspace previously. Replace the us-west-2 and AMP_WORKSPACE_ID with the correct values.

    REMOTEWRITEURL="https://aps-workspaces.us-west-2.amazonaws.com/workspaces/${AMP_WORKSPACE_ID}/api/v1/remote_write" QUERYURL="http://localhost:8005/workspaces/${AMP_WORKSPACE_ID}"

    Using those two values, you can configure Kubecost with the following command. This uses the default values for using Kubecost with Amazon Managed Service for Prometheus.

    helm upgrade -i kubecost \ oci://public.ecr.aws/kubecost/cost-analyzer --version <$VERSION> \ --namespace kubecost --create-namespace \ -f https://tinyurl.com/kubecost-amazon-eks \ -f https://tinyurl.com/kubecost-amp \ --set global.amp.prometheusServerEndpoint=${QUERYURL} \ --set global.amp.remoteWriteService=${REMOTEWRITEURL}
    Note

    For advanced configuration, you can download the values-amp.yaml file to view and edit the configuration. If you want to change the configuration, you can run the following command to configure Kubecost with your specified values.

    helm upgrade -i kubecost \ oci://public.ecr.aws/kubecost/cost-analyzer --version <$VERSION> \ --namespace kubecost --create-namespace \ -f https://tinyurl.com/kubecost-amazon-eks \ -f PATH_TO_THE_LOCAL_DIRECTORY/values-amp.yaml
  4. Finally, to reload the service account configuration, you can restart the Amazon Managed Service for Prometheus deployment with the following command.

    kubectl rollout restart deployment/kubecost-prometheus-server -n kubecost

    Your Kubecost setup is now collecting data from Amazon Managed Service for Prometheus and analyzing your metrics. You should be able to see data from your cluster within 15 minutes.

  5. [Optional] To verify that your setup is correct, you can use the following commands to view logs and verify they do not contain errors.

    To view the Prometheus logs:

    kubectl logs deploy/kubecost-prometheus-server -c prometheus-server -n kubecost --follow

    To view the sigv4proxy logs:

    kubectl logs deployment.apps/kubecost-cost-analyzer -c sigv4proxy -n kubecost --follow

    To view the cost-model logs:

    kubectl logs deployment.apps/kubecost-cost-analyzer -c cost-model -n kubecost --follow
Note

For more information about using Kubecost, including verifying that it is running properly and more advanced configuration, see Cost monitoring in the Amazon EKS User Guide.