Collecting KServe metrics with OTel Container Insights
KServe is an open-source model serving platform for Kubernetes. You describe a model in an InferenceService resource, and the KServe controller creates and maintains the pods that serve it. Container Insights collects the controller's own metrics, so you can see whether KServe is reconciling your InferenceServices.
KServe metrics help you answer operational questions such as:
-
Is the KServe controller reconciling InferenceServices, and how many reconciles fail?
-
How long do reconciles take?
-
Is the controller's admission webhook handling requests?
The KServe controller is not in the request path. Metrics about the requests that a model serves come from the model server and, in KServe Serverless mode, from the Knative queue-proxy. See Collecting vLLM metrics with OTel Container Insights and Collecting Knative Serving metrics with OTel Container Insights.
Prerequisites
KServe metrics collection is enabled by default when the following requirements are met.
-
Amazon EKS cluster running Kubernetes 1.28 or later
-
KServe installed and running in the cluster (default namespace:
kserve) -
OTel Container Insights enabled on the cluster (see Quick start)
-
The
amazon-cloudwatch-observabilityadd-on inACTIVEstatus
Configuring KServe metrics collection
KServe metrics collection is enabled by default. If you already have
OTel Container Insights enabled (otelContainerInsights.enabled: true) and
KServe is running in your cluster, metrics start flowing automatically
with no additional configuration. If KServe is not installed, the pipeline
finds no target and does nothing.
If KServe runs in a namespace other than kserve, set
otelContainerInsights.solutions.kserve.controlPlane.namespace to that
namespace. To disable KServe metrics collection, set
otelContainerInsights.solutions.kserve.controlPlane.enabled to
false. To disable all solutions at once, set
otelContainerInsights.solutions.enabled to false.
The following Helm values show the default configuration.
otelContainerInsights: solutions: enabled: true kserve: controlPlane: enabled: true namespace: kserve
To disable KServe metrics using the Amazon EKS add-on, run the following
AWS CLI command. Replace cluster-name with the name of your Amazon EKS
cluster.
aws eks update-addon \ --cluster-namecluster-name\ --addon-name amazon-cloudwatch-observability \ --configuration-values '{"otelContainerInsights":{"solutions":{"kserve":{"controlPlane":{"enabled":false}}}}}' \ --resolve-conflicts OVERWRITE
To disable KServe metrics using the Helm chart, run the following command.
helm upgrade amazon-cloudwatch-observability \ aws-observability/amazon-cloudwatch-observability \ -n amazon-cloudwatch \ --reuse-values \ --set otelContainerInsights.solutions.kserve.controlPlane.enabled=false
After you enable KServe metrics, the CloudWatch agent cluster scraper finds
the kserve-controller-manager pod by the label control-plane:
kserve-controller-manager. The controller serves its metrics only on
127.0.0.1:8080, so the agent scrapes them every 30 seconds through the
controller's kube-rbac-proxy on port 8443, using its service account
token.
KServe metrics pipeline architecture
The KServe metrics pipeline uses the following data flow:
-
The Prometheus receiver in the cluster scraper discovers the KServe controller pod using Kubernetes service discovery.
-
The receiver scrapes the controller's metrics. The pipeline keeps the controller's
controller_runtime_*andleader_election_*metrics. It drops theworkqueue_*andrest_client_*metrics, which the API server pipeline already collects. -
The OpenTelemetry processor chain attributes each metric to the controller pod and its Deployment, and adds cloud resource attributes.
-
The exporter sends the enriched metrics to CloudWatch through OTLP.
The KServe pipeline runs independently from other metric pipelines. A failure in one pipeline does not affect the others.
KServe metrics are cluster-scoped.
Available metrics
The following table lists the KServe metrics that Container Insights collects.
| Metric name | Category | Description |
|---|---|---|
controller_runtime_reconcile_total |
Reconcile | Reconciles, by controller and result |
controller_runtime_reconcile_errors_total |
Reconcile | Reconciles that returned an error, by controller |
controller_runtime_terminal_reconcile_errors_total |
Reconcile | Reconcile errors that are not retried, by controller |
controller_runtime_reconcile_panics_total |
Reconcile | Reconciles that panicked, by controller |
controller_runtime_reconcile_time_seconds |
Reconcile | Reconcile duration, by controller |
controller_runtime_active_workers |
Workers | Workers currently reconciling, by controller |
controller_runtime_max_concurrent_reconciles |
Workers | Maximum concurrent reconciles, by controller |
controller_runtime_webhook_requests_total |
Webhook | Admission webhook requests, by webhook and status code |
controller_runtime_webhook_requests_in_flight |
Webhook | Admission webhook requests in progress |
controller_runtime_webhook_latency_seconds |
Webhook | Admission webhook latency |
controller_runtime_webhook_panics_total |
Webhook | Admission webhook requests that panicked |
controller_runtime_conversion_webhook_panics_total |
Webhook | Conversion webhook requests that panicked |
leader_election_master_status |
Controller Health | 1 if this controller replica is the leader, 0 otherwise |