App Mesh observability troubleshooting

Important

End of support notice: On September 30, 2026, AWS will discontinue support for AWS App Mesh. After September 30, 2026, you will no longer be able to access the AWS App Mesh console or AWS App Mesh resources. For more information, visit this blog post Migrating from AWS App Mesh to Amazon ECS Service Connect.

This topic details common issues that you may experience with App Mesh observability.

Unable to see AWS X-Ray traces for my applications

Symptoms

Your application in App Mesh is not displaying X-Ray tracing information in the X-Ray console or APIs.

Resolution

To use X-Ray in App Mesh, you must correctly configure components to enable communication between your application, sidecar containers, and the X-Ray service. Take the following steps to confirm that X-Ray has been set up correctly:

• Make sure the App Mesh Virtual Node listener protocol is not set as TCP.

• Make sure that the X-Ray container that is deployed with your application exposes UDP port 2000 and runs as user 1337. For more information, see the Amazon ECS X-Ray example on GitHub.

• Make sure that the Envoy container has tracing enabled. If you are using the App Mesh Envoy image, you can enable X-Ray by setting the ENABLE_ENVOY_XRAY_TRACING environment variable to a value of 1 and the XRAY_DAEMON_PORT environment variable to 2000.

• If you’ve instrumented X-Ray in your application code with one of the language-specific SDKs , then make sure that it is configured correctly by following the guides for your language.

• If all of the previous items are configured correctly, then review the X-Ray container logs for errors and follow the guidance in Troubleshooting AWS X-Ray. A more detailed explanation of X-Ray integration in App Mesh can be found in Integrating X-Ray with App Mesh.

If your issue is still not resolved, then consider opening a GitHub issue or contact AWS Support.

Unable to see Envoy metrics for my applications in Amazon CloudWatch metrics

Symptoms

Your application in App Mesh is not emitting metrics generated by the Envoy proxy to CloudWatch metrics.

Resolution

When you use CloudWatch metrics in App Mesh, you must correctly configure several components to enable communication between your Envoy proxy, CloudWatch agent sidecar, and the CloudWatch metrics service. Take the following steps to confirm that CloudWatch metrics for Envoy proxy have been setup correctly:

• Make sure that you are using the CloudWatch agent image for App Mesh. For more information, see App Mesh CloudWatch agent on GitHub.

• Make sure that you have configured the CloudWatch agent for App Mesh appropriately by following the platform-specific usage instructions. For more information, see App Mesh CloudWatch agent on GitHub.

• If all of the previous items are configured correctly, then review the CloudWatch agent container logs for errors and follow the guidance provided in Troubleshooting the CloudWatch agent.

If your issue is still not resolved, then consider opening a GitHub issue or contact AWS Support.

Unable to configure custom sampling rules for AWS X-Ray traces

Symptoms

Your application is using X-Ray tracing, but you are unable to configure sampling rules for your traces.

Resolution

Since App Mesh Envoy currently does not support Dynamic X-Ray sampling configuration, the following workarounds are available.

If your Envoy version is 1.19.1 or later, you have the following options.

• To only set the sampling rate, use the XRAY_SAMPLING_RATE environment variable on the Envoy container. The value should be specified as a decimal between 0 and 1.00 (100%). For more information, see AWS X-Ray variables.

• To configure the localized custom sampling rules for the X-Ray tracer use the XRAY_SAMPLING_RULE_MANIFEST environment variable to specify a file path in the Envoy container file system. For more information, see Sampling rules in the AWS X-Ray Developer Guide.

If your Envoy version is prior to 1.19.1, then do the following.

• Use the ENVOY_TRACING_CFG_FILE environment variable to change your sampling rate. For more information, see Envoy configuration variables. Specify a custom tracing configuration and define local sampling rules. For more information, see Envoy X-Ray config.

• Custom tracing configuration for the ENVOY_TRACING_CFG_FILE environment variable example:

  tracing:
     http:
       name: envoy.tracers.xray
       typedConfig:
         "@type": type.googleapis.com/envoy.config.trace.v3.XRayConfig
         segmentName: foo/bar
         segmentFields:
           origin: AWS::AppMesh::Proxy
           aws:
             app_mesh:
               mesh_name: foo
               virtual_node_name: bar
         daemonEndpoint:
               protocol: UDP
               address: 127.0.0.1
               portValue: 2000
         samplingRuleManifest:
               filename: /tmp/sampling-rules.json

• For details on configuration for the sampling rule manifest in the samplingRuleManifest property, see Configuring the X-Ray SDK for Go.

If your issue is still not resolved, then consider opening a GitHub issue or contact AWS Support.