OpenTelemetry service identity
The CloudWatch agent sets service.name, service.namespace,
service.version, and service.instance.id the same way on all the
OpenTelemetry telemetry that it collects from a pod. So one service has one name across its
OTLP metrics, traces, and logs, its OTel Container Insights metrics and
kube-state-metrics series, its ServiceMonitor and PodMonitor metrics, and its application
logs. You can then find all the telemetry of a service by its name.
Service identity is always on, and is available in version 8.0.0 and later of the Amazon CloudWatch Observability Amazon EKS add-on and Helm chart. The agent sets each attribute only when the sender didn't set it.
How the attributes are set
The agent follows the OpenTelemetry
Kubernetes attributes convention
service.name-
The value that the sender set. Then the
resource.opentelemetry.io/service.namepod annotation. Then the name of the Deployment, ReplicaSet, StatefulSet, DaemonSet, CronJob, or Job, or the pod or container name. A value that starts withunknown_service, the SDK default, counts as not set. service.namespace-
The
resource.opentelemetry.io/service.namespacepod annotation. Then the Kubernetes namespace. service.version-
The
resource.opentelemetry.io/service.versionpod annotation. service.instance.id-
The
resource.opentelemetry.io/service.instance.idpod annotation. Then.namespace.pod.container deployment.environment.name-
cloud.platform:cluster/namespace
To give a service a name of your choice, set the
resource.opentelemetry.io/service.name annotation in the pod template of its
workload, for example:
spec: template: metadata: annotations: resource.opentelemetry.io/service.name: checkout
On KServe pods, the InferenceService name comes after the annotation. Series about the
agent's own scrape targets, such as node-exporter series and the series of control-plane
components like the Kubernetes API server and Karpenter, keep the scrape job name as
service.name, because they describe those components and not your services.
Host logs carry no service identity.
Using the app.kubernetes.io labels
The OpenTelemetry convention also lists the well-known
app.kubernetes.io/instance, app.kubernetes.io/name, and
app.kubernetes.io/version pod labels as sources. It asks for an opt-in flag,
because you might not know that your labels are used this way. The
otelServiceIdentity.useAppLabels value is that flag, and it is
false by default.
When you turn it on, the agent uses app.kubernetes.io/instance and then
app.kubernetes.io/name for service.name, after the annotation. It
uses app.kubernetes.io/version for service.version, after the
annotation.
All the pods of one Helm release usually share the same
app.kubernetes.io/instance label. With useAppLabels on, they
then share one service.name.
The following add-on configuration turns it on:
{ "otelServiceIdentity": { "useAppLabels": true } }