Troubleshooting Patch Manager - AWS Systems Manager

Troubleshooting Patch Manager

Use the following information to help you troubleshoot problems with Patch Manager, a capability of AWS Systems Manager.

Errors when running AWS-RunPatchBaseline on Linux

Issue: 'No such file or directory' error

Problem: When you run AWS-RunPatchBaseline, patching fails with one of the following errors.

IOError: [Errno 2] No such file or directory: 'patch-baseline-operations-X.XX.tar.gz'
Unable to extract tar file: /var/log/amazon/ssm/patch-baseline-operations/patch-baseline-operations-1.75.tar.gz.failed to run commands: exit status 155
Unable to load and extract the content of payload, abort.failed to run commands: exit status 152

Cause 1: Two commands to run AWS-RunPatchBaseline were running at the same time on the same instance. This creates a race condition that results in the temporary file patch-baseline-operations* not being created or accessed properly.

Cause 2: Insufficient storage space remains under the /var directory.

Solution 1: Ensure that no maintenance window has two or more Run Command tasks that run AWS-RunPatchBaseline with the same Priority level and that run on the same target IDs. If this is the case, reorder the priority. Run Command is a capability of AWS Systems Manager.

Solution 2: Ensure that only one maintenance window at a time is running Run Command tasks that use AWS-RunPatchBaseline on the same targets and on the same schedule. If this is the case, change the schedule.

Solution 3: Ensure that only one State Manager association is running AWS-RunPatchBaseline on the same schedule and targeting the same instances. State Manager is a capability of AWS Systems Manager.

Solution 4: Free up sufficient storage space under the /var directory for the update packages.

Issue: 'another process has acquired yum lock' error

Problem: When you run AWS-RunPatchBaseline, patching fails with the following error.

12/20/2019 21:41:48 root [INFO]: another process has acquired yum lock, waiting 2 s and retry.

Cause: The AWS-RunPatchBaseline document has started running on an instance where it's already running in another operation and and has acquired the package manager yum process.

Solution: Ensure that no State Manager association, maintenance window tasks, or other configurations that run AWS-RunPatchBaseline on a schedule) are targeting the same instance around the same time.

Issue: 'Permission denied / failed to run commands' error

Problem: When you run AWS-RunPatchBaseline, patching fails with the following error.

sh: 
/var/lib/amazon/ssm/instanceid/document/orchestration/commandid/PatchLinux/_script.sh: Permission denied
failed to run commands: exit status 126

Cause: /var/lib/amazon/ might be mounted with noexec permissions. This is an issue because SSM Agent downloads payload scripts to /var/lib/amazon/ssm and runs them from that location.

Solution: Ensure that you have have configured exclusive partitions to /var/log/amazon and /var/lib/amazon, and that they're mounted with exec permissions.

Issue: 'Unable to download payload' error

Problem: When you run AWS-RunPatchBaseline, patching fails with the following error.

Unable to download payload: https://s3.DOC-EXAMPLE-BUCKET.region.amazonaws.com/aws-ssm-region/patchbaselineoperations/linux/payloads/patch-baseline-operations-X.XX.tar.gz.failed to run commands: exit status 156

Cause: The instance doesn't have the required permissions to access the specified Amazon Simple Storage Service (Amazon S3) bucket.

Solution: Update your network configuration so that S3 endpoints are reachable. For more details, see information about required access to S3 buckets for Patch Manager in SSM Agent communications with AWS managed S3 buckets.

Issue: 'unsupported package manager and python version combination' error

Problem: When you run AWS-RunPatchBaseline, patching fails with the following error.

An unsupported package manager and python version combination was found. Apt requires Python3 to be installed.
failed to run commands: exit status 1

Cause: python3 isn't installed on the Ubuntu Server or Debian Server instance.

Solution: Install python3 on the server, which is required for Ubuntu Server or Debian Server instances.

Issue: Patch Manager isn't applying rules specified to exclude certain packages

Problem: You have attempted to exclude certain packages by specifying them in the /etc/yum.conf file, in the format exclude=package-name, but they aren't excluded during the Patch Manager Install operation.

Cause: Patch Manager doesn't incorporate exclusions specified in the /etc/yum.conf file.

Solution: To exclude specific packages, create a custom patch baseline and create a rule to exclude the packages you don't want installed.

Issue: Patching fails and Patch Manager reports that the Server Name Indication extension to TLS is not available

Problem: The patching operation issues the following message.

/var/log/amazon/ssm/patch-baseline-operations/urllib3/util/ssl_.py:369: 
SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension
to TLS is not available on this platform. This may cause the server to present an incorrect TLS 
certificate, which can cause validation failures. You can upgrade to a newer version of Python 
to solve this. 
For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

Cause: This message doesn't indicate an error. Instead, it's a warning that the older version of Python distributed with the operating system doesn't support TLS Server Name Indication. The Systems Manager patch payload script issues this warning when connecting to AWS APIs that support SNI.

Solution: To troubleshoot any patching failures when this message is reported, review the contents of the stdout and stderr files. If you haven't configured the patch baseline to store these files in an Amazon S3 bucket or in Amazon CloudWatch Logs, you can locate the files in the following location on your Linux instance.

/var/lib/amazon/ssm/instance-id/document/orchestration/Run-Command-execution-id/awsrunShellScript/PatchLinux

Issue: Patch Manager reports 'No more mirrors to try'

Problem: The patching operation issues the following message.

[Errno 256] No more mirrors to try.

Cause: The repositories configured on the instance are not working correctly. Possible causes for this include:

  • The yum cache is corrupted.

  • A repository URL can't be reached due to network-related issues.

Solution: Patch Manager uses the instance’s default package manager to perform patching operation. Double-check that repositories are configured and operating correctly.

Errors when running AWS-RunPatchBaseline on Windows Server

Issue: mismatched product family/product pairs

Problem: When you create a patch baseline in the Systems Manager console, you specify a product family and a product. For example, you might choose:

  • Product family: Office

    Product: Office 2016

Cause: If you attempt to create a patch baseline with a mismatched product family/product pair, an error message is displayed. The following are reasons this can occur:

  • You selected a valid product family and product pair but then removed the product family selection.

  • You chose a product from the Obsolete or mismatched options sublist instead of the Available and matching options sublist.

    Items in the product Obsolete or mismatched options sublist might have been entered in error through an SDK or AWS Command Line Interface (AWS CLI) create-patch-baseline command. This could mean a typo was introduced or a product was assigned to the wrong product family. A product is also included in the Obsolete or mismatched options sublist if it was specified for a previous patch baseline but has no patches available from Microsoft.

Solution: To avoid this issue in the console, always choose options from the Currently available options sublists.

You can also view the products that have available patches by using the describe-patch-properties command in the AWS CLI or the DescribePatchProperties API command.

Issue: AWS-RunPatchBaseline output returns an HRESULT (Windows Server)

Problem: You received an error like the following.

----------ERROR-------
Invoke-PatchBaselineOperation : Exception Details: An error occurred when 
attempting to search Windows Update.
Exception Level 1:
 Error Message: Exception from HRESULT: 0x80240437
 Stack Trace: at WUApiLib.IUpdateSearcher.Search(String criteria)..
(Windows updates)
11/22/2020 09:17:30 UTC | Info | Searching for Windows Updates.
11/22/2020 09:18:59 UTC | Error | Searching for updates resulted in error: Exception from HRESULT: 0x80240437
----------ERROR-------
failed to run commands: exit status 4294967295

Cause: This output indicates that the native Windows Update APIs were unable to run the patching operations.

Solution: Check the HResult code in the Microsoft documentation to identify troubleshooting steps for resolving the error.

Issue: Instance doesn't have access to Windows Update Catalog or WSUS

Problem: You received an error like the following.

Downloading PatchBaselineOperations PowerShell module from https://s3.amazonaws.com/path_to_module.zip to C:\Windows\TEMP\Amazon.PatchBaselineOperations-1.29.zip.

Extracting PatchBaselineOperations zip file contents to temporary folder.

Verifying SHA 256 of the PatchBaselineOperations PowerShell module files.

Successfully downloaded and installed the PatchBaselineOperations PowerShell module.

Patch Summary for

PatchGroup :

BaselineId :

Baseline : null

SnapshotId :

RebootOption : RebootIfNeeded

OwnerInformation :

OperationType : Scan

OperationStartTime : 1970-01-01T00:00:00.0000000Z

OperationEndTime : 1970-01-01T00:00:00.0000000Z

InstalledCount : -1

InstalledRejectedCount : -1

InstalledPendingRebootCount : -1

InstalledOtherCount : -1

FailedCount : -1

MissingCount : -1

NotApplicableCount : -1

UnreportedNotApplicableCount : -1

EC2AMAZ-VL3099P - PatchBaselineOperations Assessment Results - 2020-12-30T20:59:46.169

----------ERROR-------

Invoke-PatchBaselineOperation : Exception Details: An error occurred when attempting to search Windows Update.

Exception Level 1:

Error Message: Exception from HRESULT: 0x80072EE2

Stack Trace: at WUApiLib.IUpdateSearcher.Search(String criteria)

at Amazon.Patch.Baseline.Operations.PatchNow.Implementations.WindowsUpdateAgent.SearchForUpdates(String

searchCriteria)

At C:\ProgramData\Amazon\SSM\InstanceData\i-02573cafcfEXAMPLE\document\orchestration\3d2d4864-04b7-4316-84fe-eafff1ea58

e3\PatchWindows\_script.ps1:230 char:13

+ $response = Invoke-PatchBaselineOperation -Operation Install -Snapsho ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ CategoryInfo : OperationStopped: (Amazon.Patch.Ba...UpdateOperation:InstallWindowsUpdateOperation) [Inv

oke-PatchBaselineOperation], Exception

+ FullyQualifiedErrorId : Exception Level 1:

Error Message: Exception Details: An error occurred when attempting to search Windows Update.

Exception Level 1:

Error Message: Exception from HRESULT: 0x80072EE2

Stack Trace: at WUApiLib.IUpdateSearcher.Search(String criteria)

at Amazon.Patch.Baseline.Operations.PatchNow.Implementations.WindowsUpdateAgent.SearchForUpdates(String searc

---Error truncated----

Cause: This error could be related to the Windows Update components, or to a lack of connectivity to the Windows Update Catalog or Windows Server Update Services (WSUS).

Solution: Confirm that the instance has connectivity to the Microsoft Update Catalog through an internet gateway, NAT gateway, or NAT instance. If you're using WSUS, confirm that the instance has connectivity to the WSUS server in your environment. If connectivity is available to the intended destination, check the Microsoft documentation for other potential causes of HResult 0x80072EE2. This might indicate an operating system level issue.

Issue: PatchBaselineOperations PowerShell module is not downloadable

Problem: You received an error like the following.

Preparing to download PatchBaselineOperations PowerShell module from S3.
                    
Downloading PatchBaselineOperations PowerShell module from https://s3.amazonaws.com/path_to_module.zip to C:\Windows\TEMP\Amazon.PatchBaselineOperations-1.29.zip.
----------ERROR-------

C:\ProgramData\Amazon\SSM\InstanceData\i-02573cafcfEXAMPLE\document\orchestration\aaaaaaaa-bbbb-cccc-dddd-4f6ed6bd5514\

PatchWindows\_script.ps1 : An error occurred when executing PatchBaselineOperations: Unable to connect to the remote server

+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException

+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,_script.ps1

failed to run commands: exit status 4294967295

Solution: Check the instance connectivity and permissions to Amazon Simple Storage Service (Amazon S3). The instance's AWS Identity and Access Management (IAM) role must use the minimum permissions cited in SSM Agent communications with AWS managed S3 buckets. The instance must communicate with the Amazon S3 endpoint via Amazon S3 gateway endpoint, NAT gateway, or internet gateway. For more information about the VPC Endpoint requirements for AWS Systems Manager SSM Agent (SSM Agent), see Step 6: (Optional) Create a Virtual Private Cloud endpoint.

Issue: missing patches

Problem: AWS-RunPatchbaseline completed successfully, but there are some missing patches.

The following are some common causes and their solutions.

Cause 1: The baseline isn't effective.

Solution 1: To check if this is the cause, use the following procedure.

  1. Open the AWS Systems Manager console at https://console.aws.amazon.com/systems-manager/.

  2. In the navigation pane, choose Run Command.

    -or-

    If the AWS Systems Manager home page opens first, choose the menu icon ( ) to open the navigation pane, and then choose Run Command.

  3. Select the Command history tab and then select the command whose baseline you want to check.

  4. Select the instance that has missing patches.

  5. Select Step 1 - Output and find the BaselineId value.

  6. Check the assigned patch baseline configuration, that is, the operating system, product name, classification, and severity for the patch baseline.

  7. Go to the Microsoft Update Catalog.

  8. Search the Microsoft Knowledge Base (KB) article IDs (for example, KB3216916).

  9. Verify that the value under Product matches that of your instance and select the corresponding Title. A new Update Details window will open.

  10. In the Overview tab, the classification and MSRC severity must match the patch baseline configuration you found earlier.

Cause 2: The patch was replaced.

Solution 2: To check if this is true, use the following procedure.

  1. Go to the Microsoft Update Catalog.

  2. Search the Microsoft Knowledge Base (KB) article IDs (for example, KB3216916).

  3. Verify that the value under Product matches that of your instance and select the corresponding Title. A new Update Details window will open.

  4. Go to the Package Details tab. Look for an entry under the This update has been replaced by the following updates: header.

Cause 3: The same patch might have different KB numbers because the WSUS and Window online updates are handled as independent Release Channels by Microsoft.

Solution 3: Check the patch eligibility. If the package isn't available under WSUS, install OS Build 14393.3115. If the package is available for all operating system builds, install OS Builds 18362.1256 and 18363.1256.

Contacting AWS Support

If you can't find troubleshooting solutions in this section or in the Systems Manager Developer Forum, and you have a Developer, Business, or Enterprise AWS Support plan, you can create a technical support case at AWS Support.

Before you contact AWS Support, collect the following items:

  • SSM agent logs

  • Run Command command ID, maintenance window ID, or Automation execution ID

  • For Windows Server instances, also collect the following:

    • %PROGRAMDATA%\Amazon\PatchBaselineOperations\Logs as described on the Windows tab of How patches are installed

    • Windows update logs: For Windows Server 2012 R2 and older, use %windir%/WindowsUpdate.log. For Windows Server 2016 and newer, first run the PowerShell command Get-WindowsUpdateLog before using %windir%/WindowsUpdate.log

  • For Linux instances, also collect the following:

    • The contents of the file /var/lib/amazon/ssm/instance-id/document/orchestration/Run-Command-execution-id/awsrunShellScript/PatchLinux