Troubleshooting the CloudWatch Agent - Amazon CloudWatch

Troubleshooting the CloudWatch Agent

Use the following information to help troubleshoot problems with the CloudWatch agent.

CloudWatch Agent Command Line Parameters

To see the full list of parameters supported by the CloudWatch agent, enter the following at the command line at a computer where you have it installed:

amazon-cloudwatch-agent-ctl -help

Installing the CloudWatch Agent Using Run Command Fails

To install the CloudWatch agent using Systems Manager Run Command, the SSM Agent on the target server must be version 2.2.93.0 or later. If your SSM Agent isn't the correct version, you might see errors that include the following messages:

no latest version found for package AmazonCloudWatchAgent on platform linux
failed to download installation package reliably

For information about updating your SSM Agent version, see Installing and Configuring SSM Agent in the AWS Systems Manager User Guide.

The CloudWatch Agent Won't Start

If the CloudWatch agent fails to start, there might be an issue in your configuration. Configuration information is logged in the configuration-validation.log file. This file is located in /opt/aws/amazon-cloudwatch-agent/logs/configuration-validation.log on Linux servers and in $Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\configuration-validation.log on servers running Windows Server.

Verify That the CloudWatch Agent Is Running

You can query the CloudWatch agent to find whether it's running or stopped. You can use AWS Systems Manager to do this remotely. You can also use the command line, but only to check the local server.

To query the status of the CloudWatch agent using Run Command

  1. Open the Systems Manager console at https://console.aws.amazon.com/systems-manager/.

  2. In the navigation pane, choose Run Command.

    -or-

    If the AWS Systems Manager home page opens, scroll down and choose Explore Run Command.

  3. Choose Run command.

  4. In the Command document list, choose the button next to AmazonCloudWatch-ManageAgent.

  5. In the Action list, choose status.

  6. For Optional Configuration Source choose default and keep Optional Configuration Location blank.

  7. In the Target area, choose the instance to check.

  8. Choose Run.

If the agent is running, the output resembles the following.

{ "status": "running", "starttime": "2017-12-12T18:41:18", "version": "1.73.4" }

If the agent is stopped, the "status" field displays "stopped".

To query the status of the CloudWatch agent locally using the command line

  • On a Linux server, enter the following:

    sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status

    On a server running Windows Server, enter the following in PowerShell as an administrator:

    & $Env:ProgramFiles\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1 -m ec2 -a status

The CloudWatch Agent Won't Start, and the Error Mentions an Amazon EC2 Region

If the agent doesn't start and the error message mentions an Amazon EC2 Region endpoint, you might have configured the agent to need access to the Amazon EC2 endpoint without granting that access.

For example, if you specify a value for the append_dimensions parameter in the agent configuration file that depends on Amazon EC2 metadata and you use proxies, you must make sure that the server can access the endpoint for Amazon EC2. For more information about these endpoints, see Amazon Elastic Compute Cloud (Amazon EC2) in the Amazon Web Services General Reference.

The CloudWatch Agent Won't Start on Windows Server

On Windows Server, you may see the following error:

Start-Service : Service 'Amazon CloudWatch Agent (AmazonCloudWatchAgent)' cannot be started due to the following error: Cannot start service AmazonCloudWatchAgent on computer '.'. At C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1:113 char:12 + $svc | Start-Service + ~~~~~~~~~~~~~ + CategoryInfo : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Start-Service], ServiceCommandException + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.StartServiceCommand

To fix this, first make sure that the server service is running. This error can be seen if the agent tries to start when the server service isn't running.

If the server service is already running, the following may be the issue. On some Windows Server installations, the CloudWatch agent takes more than 30 seconds to start. Because Windows Server, by default, allows only 30 seconds for services to start, this causes the agent to fail with an error similar to the following:

To fix this issue, increase the service timeout value. For more information, see A service does not start, and events 7000 and 7011 are logged in the Windows event log.

Unable to Find Credentials on Windows Server

On Windows Server, if you have credentials in a location other than $SystemDrive\\Users\\Administrator\\.aws on Windows Server 2008 R2 or Windows Server 2012, or “$SystemDrive\\Documents and Settings\\Administrator\\.aws on Windows Server 2003, you can specify your own credential path by using the shared_credential_file option in common.toml.

If you don’t have a credential file, you must create one. For more information, see (Optional) Modify the Common Configuration for Proxy or Region Information.

Where Are the Metrics?

If the CloudWatch agent has been running but you can't find metrics collected by it in the AWS Management Console or the AWS CLI, confirm that you're using the correct namespace. By default, the namespace for metrics collected by the agent is CWAgent. You can customize this namespace using the namespace field in the metrics section of the agent configuration file. If you don't see the metrics that you expect, check the configuration file to confirm the namespace being used.

When you first download the CloudWatch agent package, the agent configuration file is amazon-cloudwatch-agent.json. This file is in the directory where you ran the configuration wizard, or you might have moved it to a different directory. If you use the configuration wizard, the agent configuration file output from the wizard is named config.json. For more information about the configuration file, including the namespace field, see CloudWatch Agent Configuration File: Metrics Section.

I updated my agent configuration but don’t see the new metrics or logs in the CloudWatch console

If you update your CloudWatch agent configuration file, the next time that you start the agent, you need to use the fetch-config option. For example, if you stored the updated file on the local computer, enter the following command:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -s -m ec2 -c file:configuration-file-path

CloudWatch Agent Files and Locations

The following table lists the files installed by and used with the CloudWatch agent, along with their locations on servers running Linux or Windows Server.

File Linux Location Windows Server Location

The control script that controls starting, stopping, and restarting the agent.

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl or /usr/bin/amazon-cloudwatch-agent-ctl

$Env:ProgramFiles\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1

The log file the agent writes to. You might need to attach this when contacting AWS Support.

/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log or /var/log/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.log

$Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log

Agent configuration validation file.

/opt/aws/amazon-cloudwatch-agent/logs/configuration-validation.log or /var/log/amazon/amazon-cloudwatch-agent/configuration-validation.log

$Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\configuration-validation.log

The JSON file used to configure the agent immediately after the wizard creates it. For more information, see Create the CloudWatch Agent Configuration File.

/opt/aws/amazon-cloudwatch-agent/bin/config.json

$Env:ProgramFiles\Amazon\AmazonCloudWatchAgent\config.json

The JSON file used to configure the agent if this configuration file has been downloaded from Parameter Store.

/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json or /etc/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.json

$Env:ProgramData\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent.json

TOML file used to specify Region and credential information to be used by the agent, overriding system defaults.

/opt/aws/amazon-cloudwatch-agent/etc/common-config.toml or /etc/amazon/amazon-cloudwatch-agent/common-config.toml

$Env:ProgramData\Amazon\AmazonCloudWatchAgent\common-config.toml

Finding information about CloudWatch agent versions

To find the version number of the CloudWatch agent on a Linux server, enter the following command:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status

To find the version number of the CloudWatch agent on Windows Server, enter the following command:

& $Env:ProgramFiles\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1 -m ec2 -a status
Note

Using this command is the correct way to find the version of the CloudWatch agent. If you use Programs and Features in the Control Panel, you will see an incorrect version number.

You can also download a README file about the latest changes to the agent, and a file that indicates the version number that is currently available for download. These files are in the follow;ing locations:

  • https://s3.amazonaws.com/amazoncloudwatch-agent/info/latest/RELEASE_NOTES or https://s3.region.amazonaws.com/amazoncloudwatch-agent-Region/info/latest/RELEASE_NOTES

  • https://s3.amazonaws.com/amazoncloudwatch-agent/info/latest/CWAGENT_VERSION or https://s3.region.amazonaws.com/amazoncloudwatch-agent-Region/info/latest/CWAGENT_VERSION

Logs Generated by the CloudWatch Agent

The agent generates a log while it runs. This log includes troubleshooting information. This log is the amazon-cloudwatch-agent.log file. This file is located in /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log on Linux servers and in $Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log on servers running Windows Server.

You can configure the agent to log additional details in the amazon-cloudwatch-agent.log file. In the agent configuration file, in the agent section, set the debug field to true, then reconfigure and restart the CloudWatch agent. To disable the logging of this extra information, set the debug field to false reconfigure and restart the agent. For more information, see Manually Create or Edit the CloudWatch Agent Configuration File.

Stopping and Restarting the CloudWatch Agent

You can manually stop the CloudWatch agent using either AWS Systems Manager or the command line.

To stop the CloudWatch agent using Run Command

  1. Open the Systems Manager console at https://console.aws.amazon.com/systems-manager/.

  2. In the navigation pane, choose Run Command.

    -or-

    If the AWS Systems Manager home page opens, scroll down and choose Explore Run Command.

  3. Choose Run command.

  4. In the Command document list, choose AmazonCloudWatch-ManageAgent.

  5. In the Targets area, choose the instance where you installed the CloudWatch agent.

  6. In the Action list, choose stop.

  7. Keep Optional Configuration Source and Optional Configuration Location blank.

  8. Choose Run.

To stop the CloudWatch agent locally using the command line

  • On a Linux server, enter the following:

    sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a stop

    On a server running Windows Server, enter the following in PowerShell as an administrator:

    & $Env:ProgramFiles\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1 -m ec2 -a stop

To restart the agent, follow the instructions in Start the CloudWatch Agent.