Patch monitoring and failure remediation

AMS Accelerate Patch add-on monitors patching and remediates failures.

Patch notification

AMS Accelerate sends notifications for the patching maintenance windows configured, see Create a patch maintenance window. The system sends CloudWatch Events as advance notice of each upcoming maintenance window and a CloudWatch Event at the end of each maintenance window. The advance notices are sent four days and one hour before the maintenance window. The advance notice CloudWatch Events are sent with the following schema:

{
    "version": "0",
    "id": "37004d81-458d-2cef-fe1c-8afa8af30406",
    "detail-type": "AMS Patch Window Execution State Change",
    "source": "aws.managedservices",
    "account": "145917996532",
    "time": "2021-05-20T02:00:00Z",
    "region": "us-east-1",
    "resources": [
        "arn:aws:ssm:us-east-1:123456789012:maintenancewindow/mw-00000001235",
        "arn:aws:ec2:us-east-1:123456789012:instance/i-0000000aaaaaaaaaa",
        "arn:aws:ec2:us-east-1:123456789012:instance/i-0000000aaaaaaaaab"
    ],
    "detail": {
        "State": "PREEMPTIVE",
        "StartTime": "2021-05-24T02:00:00.000000",
        "WindowArn": "arn:aws:ssm:us-east-1:123456789012:maintenancewindow/mw-00000001235",
        "Results": "[{\"instanceId\": \"i-0000000aaaaaaaaaa\"}, {\"instanceId\": \"i-0000000aaaaaaaaab\"}]"
    }
}

This table describes the advance notice event schema.

Patch notification details
Property name	Description	Sample values
State	The state of the patching maintenance window	PREEMPTIVE – The patching window scheduled to begin soon
Status	The status of the patching maintenance window	SUCCESS – All instances were patch without failure FAILED – At least one instance has failed to patch
StartTime	The start time, in ISO format, of the patching maintenance window	2021-02-03T22:14:05.814308
WindowArn	The unique identifier of the patching maintenance window.	arn:aws:ssm:us-east-1: 123456789012:maintenancewindow/mw-00000001235
Results	The list of instances that will be targeted by the patch window	InstanceId – the instance ID targeted

The CloudWatch Events can be used to trigger a CloudWatch rule to notify you whenever a patching maintenance window advance notice is sent. In this case, you would configure the CloudWatch rule with the following:

{"source": [
    "aws.managedservices"
  ],
  "detail-type: ["AMS Patch Window Execution State Change"],
  "detail": {
      "State": ["PREEMPTIVE"]
  }
 }

The window end CloudWatch Event is sent with the following schema:

{"version": "0",
    "id": "0f25add5-44a9-0702-d2bc-bd2102affefe",
    "detail-type": "AMS Patch Window Execution State Change",
    "source": "aws.managedservices",
    "account": "123456789012",
    "time": "2021-02-03T22:14:06Z",
    "region": "us-east-1",
    "resources": [
        "arn:aws:ssm:us-east-1:123456789012:maintenancewindow/mw-00000001235",
        "arn:aws:ec2:us-east-1:123456789012:instance/i-0000000aaaaaaaaaa",
        "arn:aws:ec2:us-east-1:123456789012:instance/i-0000000aaaaaaaaab"
    ],
    "detail": {"State": "[COMPLETED]",
        "Status": "SUCCESS",
        "StartTime": "2021-02-03T22:12:00.814308",
        "EndTime": "2021-02-03T22:14:05.814309",
        "WindowArn": "arn:aws:ssm:us-east-1:123456789012:maintenancewindow/mw-00000001235",
        "WindowExecutionId": "e32088eb-c05f-4c63-b766-6866e163c818",
        "Results": "[{\"instanceId\": \"i-0000000aaaaaaaaaa\", \"status\": \"Success\", \"missing_critical_patch_count\": 0, \"missing_total_patch_count\": 0} }, {\"instanceId\": \"i-0000000aaaaaaaaab\", \"status\": Success}, \"missing_critical_patch_count\": 0, \"missing_total_patch_count\": 0}]"
    }
}

Patch window end details
Property name	Description	Sample values
State	The state of the patching maintenance window	COMPLETED – The patching window is finished
Status	The status of the patching maintenance window	SUCCESS – All instances were patch without failure FAILED – At least one instance has failed to patch
StartTime	The start time, in ISO format, of the patching maintenance window	2021-02-03T22:14:05.814308
EndTime	The end time, in ISO format, of the patching maintenance window	2021-02-03T23:14:05.814308
WindowArn	The unique identifier of the patching maintenance window.	arn:aws:ssm:us-east-1: 123456789012:maintenancewindow/mw-00000001235
WindowExecutionId	The window execution ID, which can be seen from the SSM Maintenance Window Console	e32088eb-c05f-4c63-b766-6866e163c818
Results	The list of instances that will be targeted by the patch window	InstanceId – the instance ID targeted status – the instance patch status missing_critical_patch_count - the count of critical patches missing on the instance missing_total_patch_count - the count of total patches missing on the instance

Note

Patch failure alerts aren't created for instances that have unsupported operating systems, or that are stopped during the maintenance window.

Patch remediation

AWS Managed Services (AMS) manages patching and includes patch failure remediation. When patch fails, AMS Operations is alerted and they proceed to remediate by following AWS and AMS best practices to address the issue.

AMS creates an SSM OpsItem in the account with the following title: AWS Managed Services – Patch Instance failure for instance <instance-id>. AMS Operation will resolve the OpsItem if the situation can be corrected without your intervention. If they need your intervention, a service request is created and notice sent to the account owner to collaborate and address the issue. Upon remediation, the OpsItem is resolved.

If no action is taken, the patching maintenance window runs on the next cycle and attempts to patch the instance again.

Note

Patch failure alerts aren't created for instances that have unsupported operating systems, or that are stopped during the maintenance window.