Troubleshooting SSM Agent - AWS Systems Manager

Troubleshooting SSM Agent

If you experience problems running operations on your managed nodes, there might be a problem with AWS Systems Manager Agent (SSM Agent). Use the following information to help you view SSM Agent log files and troubleshoot the agent.

SSM Agent is out of date

An updated version of SSM Agent is released whenever new capabilities are added to Systems Manager or updates are made to existing capabilities. Failing to use the latest version of the agent can prevent your managed node from using various Systems Manager capabilities and features. For that reason, we recommend that you automate the process of keeping SSM Agent up to date on your machines. For information, see Automating updates to SSM Agent. Subscribe to the SSM Agent Release Notes page on GitHub to get notifications about SSM Agent updates.

Troubleshoot issues using SSM Agent log files

SSM Agent logs information in the following files. The information in these files can also help you troubleshoot problems. For more information about SSM Agent log files, including how to turn on debug logging, see Viewing SSM Agent logs.

Note

If you choose to view these logs by using Windows File Explorer, be sure to allow the viewing of hidden files and system files in Folder Options.

On Windows
  • %PROGRAMDATA%\Amazon\SSM\Logs\amazon-ssm-agent.log

  • %PROGRAMDATA%\Amazon\SSM\Logs\errors.log

On Linux and macOS
  • /var/log/amazon/ssm/amazon-ssm-agent.log

  • /var/log/amazon/ssm/errors.log

For Linux managed nodes, you might find more information in the messages file written to the following directory: /var/log.

For additional information about troubleshooting using agent logs, see How do I use SSM Agent logs to troubleshoot issues with SSM Agent in my managed instance? in the AWS re:Post Knowledge Center.

Agent log files don't rotate (Windows)

If you specify date-based log file rotation in the seelog.xml file (on Windows Server managed nodes) and the logs don't rotate, specify the fullname=true parameter. Here is an example of a seelog.xml configuration file with the fullname=true parameter specified.

<seelog type="adaptive" mininterval="2000000" maxinterval="100000000" critmsgcount="500" minlevel="debug"> <exceptions> <exception filepattern="test*" minlevel="error" /> </exceptions> <outputs formatid="fmtinfo"> <console formatid="fmtinfo" /> <rollingfile type="date" datepattern="200601021504" maxrolls="4" filename="C:\ProgramData\Amazon\SSM\Logs\amazon-ssm-agent.log" fullname=true /> <filter levels="error,critical" formatid="fmterror"> <rollingfile type="date" datepattern="200601021504" maxrolls="4" filename="C:\ProgramData\Amazon\SSM\Logs\errors.log" fullname=true /> </filter> </outputs> <formats> <format id="fmterror" format="%Date %Time %LEVEL [%FuncShort @ %File.%Line] %Msg%n" /> <format id="fmtdebug" format="%Date %Time %LEVEL [%FuncShort @ %File.%Line] %Msg%n" /> <format id="fmtinfo" format="%Date %Time %LEVEL %Msg%n" /> </formats> </seelog>

Unable to connect to SSM endpoints

SSM Agent must allow HTTPS (port 443) outbound traffic to the following endpoints:

  • ssm.region.amazonaws.com

  • ssmmessages.region.amazonaws.com

region represents the identifier for an AWS Region supported by AWS Systems Manager, such as us-east-2 for the US East (Ohio) Region. For a list of supported region values, see the Region column in Systems Manager service endpoints in the Amazon Web Services General Reference.

Note

Prior to 2024, ec2messages.region.amazonaws.com was also required. For AWS Regions launched before 2024, allowing traffic to ssmmessages.region.amazonaws.com is still required but optional to ec2messages.region.amazonaws.com.

For Regions launched in 2024 and later, allowing traffic to ssmmessages.region.amazonaws.com is required, but ec2messages.region.amazonaws.com endpoints are not supported for these Regions.

SSM Agent won't work if it can't communicate with the preceding endpoints, as described, even if you use AWS provided Amazon Machine Images (AMIs) such as Amazon Linux 2 or Amazon Linux 2023. Your network configuration must have open internet access or you must have custom virtual private cloud (VPC) endpoints configured. If you don't plan on creating a custom VPC endpoint, check your internet gateways or NAT gateways. For more information about how to manage VPC endpoints, see Improve the security of EC2 instances by using VPC endpoints for Systems Manager.

Verify your VPC configuration

In order to manage EC2 instances with Systems Manager, your VPC endpoints must be configured properly for ssm.region.amazonaws.com, ssmmessages.region.amazonaws.com, and in some cases explained earlier in this topic in Unable to connect to SSM endpoints, ec2messages.region.amazonaws.com. Your network configuration must have open internet access or you must have these virtual private cloud (VPC) endpoints configured.

To troubleshoot issues with your VPC endpoints, do the following:

  • Ensure that VPC endpoints are included at the VPC level. If the VPC endpoint with a specific service name is not found on the VPC, first verify that DNS support is enabled at the VPC level. Next, create a new VPC endpoint and associate it with one subnet in each Availability Zone.

  • Ensure that a private DNS name is enabled at the VPC endpoint level. Private DNS names are enabled by default but might have been manually disabled at some point.

  • Ensure that existing VPC endpoints are associated with the proper subnet. In addition, ensure that the VPCE is already associated with a subnet in that Availability Zone.

For more information, see the following topics:

Verify your VPC DNS-related attributes

As part of verifying your VPC configuration, ensure that the attributes enableDnsSupport and enableDnsHostnames are enabled.

You can enable these attributes using the Amazon EC2 ModifyVPCAttribute API action or the AWS CLI command modify-vpc-attribute.

For information about enabling these attributes in the Amazon VPC Console, see View and update DNS attributes for your VPC in the Amazon VPC User Guide.

Verify ingress rules on endpoint security groups

Ensure that any VPC endpoints you have configured (ssm, ssmmessages, and ec2messages) include an ingress rule on their security groups to allow traffic in on port 443. If necessary, you can create a new security group in the VPC with an ingress rule to allow traffic on port 443 for the Classless Inter-Domain Routing (CIDR) block for the VPC. After you create the security group, attach it to each VPC endpoint.

For more information, see the following topics:

Use ssm-cli to troubleshoot managed node availability

Starting with SSM Agent version 3.1.501.0, you can use ssm-cli to determine whether a managed node meets the primary requirements to be managed by Systems Manager, and to appear in lists of managed nodes in Fleet Manager. The ssm-cli is a standalone command line tool included in the SSM Agent installation. Preconfigured commands are included that gather the required information to help you diagnose why an Amazon EC2 instance or non-EC2 machine that you have confirmed is running isn't included in your lists of managed nodes in Systems Manager. These commands are run when you specify the get-diagnostics option.

For more information, see Troubleshooting managed node availability using ssm-cli.