Patch monitoring and failure remediation - AMS Accelerate Operations Plan

Patch monitoring and failure remediation

AMS Accelerate Patch add-on monitors patching and remediates failures.

Patch notification

AMS Accelerate sends notifications for the patching maintenance windows configured, see Create an SSM maintenance window for patching. The system sends CloudWatch Events as advance notice of each upcoming maintenance window and a CloudWatch Event at the end of each maintenance window. The advance notices are sent four days and one hour before the maintenance window. The advance notice CloudWatch Events are sent with the following schema:

{ "version": "0", "id": "37004d81-458d-2cef-fe1c-8afa8af30406", "detail-type": "AMS Patch Window Execution State Change", "source": "aws.managedservices", "account": "145917996532", "time": "2021-05-20T02:00:00Z", "region": "us-east-1", "resources": [ "arn:aws:ssm:us-east-1:123456789012:maintenancewindow/mw-00000001235", "arn:aws:ec2:us-east-1:123456789012:instance/i-0000000aaaaaaaaaa", "arn:aws:ec2:us-east-1:123456789012:instance/i-0000000aaaaaaaaab" ], "detail": { "State": "PREEMPTIVE", "StartTime": "2021-05-24T02:00:00.000000", "WindowArn": "arn:aws:ssm:us-east-1:123456789012:maintenancewindow/mw-00000001235", "Results": "[{\"instanceId\": \"i-0000000aaaaaaaaaa\"}, {\"instanceId\": \"i-0000000aaaaaaaaab\"}]" } }

This table describes the advance notice event schema.

Patch notification details
Property name Description Sample values

State

The state of the patching maintenance window

PREEMPTIVE – The patching window scheduled to begin soon

Status

The status of the patching maintenance window

SUCCESS – All instances were patch without failure

FAILED – At least one instance has failed to patch

StartTime

The start time, in ISO format, of the patching maintenance window

2021-02-03T22:14:05.814308

WindowArn

The unique identifier of the patching maintenance window.

arn:aws:ssm:us-east-1: 123456789012:maintenancewindow/mw-00000001235

Results

The list of instances that will be targeted by the patch window

InstanceId – the instance ID targeted

The CloudWatch Events can be used to trigger a CloudWatch rule to notify you whenever a patching maintenance window advance notice is sent. In this case, you would configure the CloudWatch rule with the following:

{"source": [ "aws.managedservices" ], "detail-type: ["AMS Patch Window Execution State Change"], "detail": { "State": ["PREEMPTIVE"] } }

The window end CloudWatch Event is sent with the following schema:

{"version": "0", "id": "0f25add5-44a9-0702-d2bc-bd2102affefe", "detail-type": "AMS Patch Window Execution State Change", "source": "aws.managedservices", "account": "123456789012", "time": "2021-02-03T22:14:06Z", "region": "us-east-1", "resources": [ "arn:aws:ssm:us-east-1:123456789012:maintenancewindow/mw-00000001235", "arn:aws:ec2:us-east-1:123456789012:instance/i-0000000aaaaaaaaaa", "arn:aws:ec2:us-east-1:123456789012:instance/i-0000000aaaaaaaaab" ], "detail": {"State": "[COMPLETED]", "Status": "SUCCESS", "StartTime": "2021-02-03T22:12:00.814308", "EndTime": "2021-02-03T22:14:05.814309", "WindowArn": "arn:aws:ssm:us-east-1:123456789012:maintenancewindow/mw-00000001235", "WindowExecutionId": "e32088eb-c05f-4c63-b766-6866e163c818", "Results": "[{\"instanceId\": \"i-0000000aaaaaaaaaa\", \"status\": \"Success\", \"missing_critical_patch_count\": 0, \"missing_total_patch_count\": 0} }, {\"instanceId\": \"i-0000000aaaaaaaaab\", \"status\": Success}, \"missing_critical_patch_count\": 0, \"missing_total_patch_count\": 0}]" } }
Patch window end details
Property name Description Sample values

State

The state of the patching maintenance window

COMPLETED – The patching window is finished

Status

The status of the patching maintenance window

SUCCESS – All instances were patch without failure

FAILED – At least one instance has failed to patch

StartTime

The start time, in ISO format, of the patching maintenance window

2021-02-03T22:14:05.814308

EndTime

The end time, in ISO format, of the patching maintenance window

2021-02-03T23:14:05.814308

WindowArn

The unique identifier of the patching maintenance window.

arn:aws:ssm:us-east-1: 123456789012:maintenancewindow/mw-00000001235

WindowExecutionId

The window execution ID, which can be seen from the SSM Maintenance Window Console

e32088eb-c05f-4c63-b766-6866e163c818

Results

The list of instances that will be targeted by the patch window

InstanceId – the instance ID targeted

status – the instance patch status

missing_critical_patch_count - the count of critical patches missing on the instance

missing_total_patch_count - the count of total patches missing on the instance

Patch remediation

AWS Managed Services (AMS) manages patching and includes patch failure remediation. When patch fails, AMS Operations is alerted and they proceed to remediate by following AWS and AMS best practices to address the issue.

AMS creates an SSM OpsItem in the account with the following title: AWS Managed Services – Patch Instance failure for instance <instance-id>. AMS Operation will resolve the OpsItem if the situation can be corrected without your intervention. If they need your intervention, a Service Request is sent to the account owner to collaborate and address the issue. Upon remediation, the OpsItem is resolved.

If no action is taken, the patching maintenance window runs on the next cycle and attempts to patch the instance again.