Troubleshooting SSM Agent
If you experience problems running operations on your managed nodes, there might be a problem with AWS Systems Manager Agent (SSM Agent). Use the following information to help you view SSM Agent log files and troubleshoot the agent.
Topics
- SSM Agent is out of date
- Troubleshoot issues using SSM Agent log files
- Agent log files don't rotate (Windows)
- Unable to connect to SSM endpoints
- Verify your VPC configuration
- Verify your VPC DNS-related attributes
- Verify ingress rules on endpoint security groups
- Use ssm-cli to troubleshoot managed node availability
SSM Agent is out of date
An updated version of SSM Agent is released whenever new capabilities are added to Systems Manager or
updates are made to existing capabilities. Failing to use the latest version of the agent can prevent your managed node
from using various Systems Manager capabilities and features. For that reason, we recommend that you automate
the process of keeping SSM Agent up to date on your machines. For information, see Automating updates to SSM Agent. Subscribe to the SSM Agent
Release Notes
Troubleshoot issues using SSM Agent log files
SSM Agent logs information in the following files. The information in these files can also help you troubleshoot problems. For more information about SSM Agent log files, including how to turn on debug logging, see Viewing SSM Agent logs.
Note
If you choose to view these logs by using Windows File Explorer, be sure to allow the viewing of hidden files and system files in Folder Options.
On Windows
-
%PROGRAMDATA%\Amazon\SSM\Logs\amazon-ssm-agent.log
-
%PROGRAMDATA%\Amazon\SSM\Logs\errors.log
On Linux and macOS
-
/var/log/amazon/ssm/amazon-ssm-agent.log
-
/var/log/amazon/ssm/errors.log
For Linux managed nodes, you might find more information in the
messages
file written to the following directory:
/var/log
.
For additional information about troubleshooting using agent logs, see How do I use SSM Agent logs to
troubleshoot issues with SSM Agent in my managed instance?
Agent log files don't rotate (Windows)
If you specify date-based log file rotation in the seelog.xml file (on Windows Server
managed nodes) and the logs don't rotate, specify the fullname=true
parameter. Here is an example of a seelog.xml configuration file with the
fullname=true
parameter specified.
<seelog type="adaptive" mininterval="2000000" maxinterval="100000000" critmsgcount="500" minlevel="debug"> <exceptions> <exception filepattern="test*" minlevel="error" /> </exceptions> <outputs formatid="fmtinfo"> <console formatid="fmtinfo" /> <rollingfile type="date" datepattern="200601021504" maxrolls="4" filename="C:\ProgramData\Amazon\SSM\Logs\amazon-ssm-agent.log" fullname=true /> <filter levels="error,critical" formatid="fmterror"> <rollingfile type="date" datepattern="200601021504" maxrolls="4" filename="C:\ProgramData\Amazon\SSM\Logs\errors.log" fullname=true /> </filter> </outputs> <formats> <format id="fmterror" format="%Date %Time %LEVEL [%FuncShort @ %File.%Line] %Msg%n" /> <format id="fmtdebug" format="%Date %Time %LEVEL [%FuncShort @ %File.%Line] %Msg%n" /> <format id="fmtinfo" format="%Date %Time %LEVEL %Msg%n" /> </formats> </seelog>
Unable to connect to SSM endpoints
SSM Agent must allow HTTPS (port 443) outbound traffic to the following endpoints:
-
ssm.
region
.amazonaws.com -
ssmmessages.
region
.amazonaws.com
region
represents the identifier for an AWS Region
supported by AWS Systems Manager, such as us-east-2
for the US East (Ohio) Region. For a list of
supported region
values, see the Region column in Systems Manager service endpoints in the
Amazon Web Services General Reference.
Note
Prior to 2024,
ec2messages.
was
also required. For AWS Regions launched before 2024, allowing traffic to
region
.amazonaws.comssmmessages.
is still required but optional to
region
.amazonaws.comec2messages.
. region
.amazonaws.com
For Regions launched in 2024 and later, allowing traffic to
ssmmessages.
is required, but
region
.amazonaws.comec2messages.
endpoints are not supported for these Regions.region
.amazonaws.com
SSM Agent won't work if it can't communicate with the preceding endpoints, as described, even if you use AWS provided Amazon Machine Images (AMIs) such as Amazon Linux 2 or Amazon Linux 2023. Your network configuration must have open internet access or you must have custom virtual private cloud (VPC) endpoints configured. If you don't plan on creating a custom VPC endpoint, check your internet gateways or NAT gateways. For more information about how to manage VPC endpoints, see Improve the security of EC2 instances by using VPC endpoints for Systems Manager.
Verify your VPC configuration
In order to manage EC2 instances with Systems Manager, your VPC endpoints must be configured
properly for ssm.
,
region
.amazonaws.comssmmessages.
, and
in some cases explained earlier in this topic in Unable
to connect to SSM endpoints,
region
.amazonaws.comec2messages.
.
Your network configuration must have open internet access or you must have these
virtual private cloud (VPC) endpoints configured.region
.amazonaws.com
To troubleshoot issues with your VPC endpoints, do the following:
-
Ensure that VPC endpoints are included at the VPC level. If the VPC endpoint with a specific service name is not found on the VPC, first verify that DNS support is enabled at the VPC level. Next, create a new VPC endpoint and associate it with one subnet in each Availability Zone.
-
Ensure that a private DNS name is enabled at the VPC endpoint level. Private DNS names are enabled by default but might have been manually disabled at some point.
-
Ensure that existing VPC endpoints are associated with the proper subnet. In addition, ensure that the VPCE is already associated with a subnet in that Availability Zone.
For more information, see the following topics:
-
Access an AWS service using an interface VPC endpoint in the AWS PrivateLink Guide
-
Associate a private DNS name in the AWS PrivateLink Guide
-
Improve the security of EC2 instances by using VPC endpoints for Systems Manager
Verify your VPC DNS-related attributes
As part of verifying your VPC configuration, ensure that the attributes
enableDnsSupport
and enableDnsHostnames
are enabled.
You can enable these attributes using the Amazon EC2 ModifyVPCAttribute API action or the AWS CLI command modify-vpc-attribute.
For information about enabling these attributes in the Amazon VPC Console, see View and update DNS attributes for your VPC in the Amazon VPC User Guide.
Verify ingress rules on endpoint security groups
Ensure that any VPC endpoints you have configured (ssm
,
ssmmessages
, and ec2messages
) include an ingress rule
on their security groups to allow traffic in on port 443. If necessary, you can
create a new security group in the VPC with an ingress rule to allow traffic on port
443 for the Classless Inter-Domain Routing (CIDR) block for the VPC. After you
create the security group, attach it to each VPC endpoint.
For more information, see the following topics:
-
How do I create VPC endpoints so that I can use Systems Manager to manage private EC2 instances without internet access?
on AWS re:Post -
VPC CIDR blocks in the Amazon VPC User Guide
Use ssm-cli
to troubleshoot managed node
availability
Starting with SSM Agent version 3.1.501.0, you can use ssm-cli
to determine
whether a managed node meets the primary requirements to be managed by Systems Manager, and to
appear in lists of managed nodes in Fleet Manager. The ssm-cli
is a standalone command
line tool included in the SSM Agent installation. Preconfigured commands are included
that gather the required information to help you diagnose why an Amazon EC2 instance or
non-EC2 machine that you have confirmed is running isn't included in your lists of
managed nodes in Systems Manager. These commands are run when you specify the
get-diagnostics
option.
For more information, see Troubleshooting managed node availability using ssm-cli.