OPS03-BP05 Design for efficient patch management
Patching keeps systems secure, supported, and current with features, but patching manually does not scale and is error-prone. Treat patch management as a designed process: decide what to patch, when, and how it moves across environments, and then automate the mechanics. Rebuild systems from a known good image where you can, and use controlled in-place patching where you cannot.
Desired outcome:
-
Your machine and container images are patched, updated, and ready to launch from a known good state.
-
You can track the patch status of all deployed images and know your compliance position.
-
You have a process to report current patch status against your compliance requirements.
-
You know about planned lifecycle events in advance and plan patching around them.
Common anti-patterns:
-
A team is told to apply every security patch within two hours, and the rushed patches cause incidentss from application incompatibility.
-
An unpatched library carries a known vulnerability that an unauthorized user can use to reach the workload.
-
A team patches developer environments automatically with no notice, and the changes break the developers' work.
-
A team never patches commercial software on a long-lived instance, then finds the vendor will not support the outdated version during an incident.
-
A team learns of a zero-day vulnerability and has to patch every environment manually because nothing is automated.
-
A team ignores planned lifecycle events, loses the lead time to prepare, and faces emergency changes and unexpected downtime.
Benefits of establishing this best practice:
-
A defined patch process lets you scale patching and report on patch levels across environments.
-
You gain assurance and visibility that known fixes are in place.
-
You adopt new features and capabilities and remove known issues quickly.
-
Automation lowers the effort to deploy patches and reduces manual errors.
Level of risk exposed if this best practice is not established: Medium
Implementation guidance
Immutable infrastructure replaces a running system with a freshly built, patched image, so the fleet never diverges from a known good state. In-place patching modifies systems where they run. With immutable patching, you roll back by redeploying the previous image. Your systems must be rebuildable with externalized state. We recommend immutable patching wherever the workload supports it, and in-place patching when a system cannot be rebuilt.
Apply a patch only when it supports an operational or business outcome, and test it in a safe environment before production. Define your patch criteria and distribution method up front: which patches qualify, how they move across environments, and how quickly. Urgent patches then have a fast path without bypassing testing.
Planned lifecycle events also require patching: mandatory version
updates, runtime deprecations, and end-of-support dates. Missing
these events turns a scheduled update into an emergency change.
AWS Health
Image pipelines make immutable patching repeatable.
Amazon EC2 Image Builder
Amazon Inspector
When you must patch in place, automate it. AWS Systems Manager Patch Manager patches managed instances against a patch baseline, and Systems Manager Maintenance Windows schedule that work for times that limit disruption. You can then respond to previously-unknown issues across every environment without patching each system manually.
When a patch or platform upgrade changes behavior in several subsystems, release it in stages and hold each stage long enough to observe sustained production load. Some defects take days to appear. See OPS04-BP01 Employ and test safe deployment strategies for staged rollout and bake time.
Implementation steps
-
Choose immutable or in-place patching per workload: For workloads you can rebuild, patch by producing a new image and redeploying. Reserve in-place patching for systems that cannot be rebuilt. Document which model each workload uses so the patch process is predictable.
-
Build image pipelines for immutable patching: Use Amazon EC2 Image Builder to create an image pipeline for machine images and a container image pipeline for containers. Define the pipeline schedule, choose a recipe with a base image and build components, set the target registry, and review the configuration before you run it. Revisit recipes regularly so base images and components stay current.
-
Automate in-place patching where rebuilds are not possible: Create a patch baseline in AWS Systems Manager Patch Manager, select a patching method, and schedule it with Maintenance Windows. Enable compliance scanning so you can report on patch levels.
-
Test patches before production: Apply patches in a safe environment and verify application compatibility before you promote them. Notify the owners of shared or developer environments before patching, so changes do not disrupt their work.
-
Track compliance and monitor lifecycle events: Use Patch Manager compliance reports to report patch status against your requirements, and review AWS Health
for planned lifecycle events that require action.
Resources
Related best practices:
Related documents:
Related examples:
Related services: