PERF05-BP03 Use automation to proactively remediate performance-related issues
Configure automated investigation as the first line of incident response so analysis starts when an alert or ticket meets your incident criteria. While an engineer responds to the page, the investigation correlates telemetry and topology and returns a root cause or evidence-backed hypotheses. Deterministic automation then handles well-understood, low-risk remediation. Reserve human judgment for uncertain or high-impact actions.
Desired outcome:
-
You start agent-assisted investigation only from alerts or tickets that meet your defined incident criteria.
-
You link duplicate signals to active investigations, skip non-actionable events, and proceed with new incidents that require analysis.
-
Your investigations correlate metrics, logs, traces, code changes, deployments, and service topology across the systems your team operates.
-
You give the agent your current application design, critical user journeys, SLOs, ownership, known constraints, maintenance windows, and escalation context.
-
Your findings and mitigation plans reach the ticketing and collaboration tools where your responders already work.
-
You limit automated remediation to well-understood cases, keep it proportionate to the problem, and govern it with approval and rollback controls.
-
Every automated action verifies that it resolved the issue before your workflow considers it complete.
-
Your incident history produces proactive recommendations that reduce recurrence and improve your observability, infrastructure, governance, and code.
-
Your reusable Skills and executable runbooks capture investigation procedures, operational practices, and remediation workflows, and you test them on a schedule.
Common anti-patterns:
-
Teams route every alarm to the operations team without starting automated investigation or remediation.
-
Teams route every alert to an agent without severity, user-impact, actionability, deduplication, maintenance-window, or skip criteria, producing irrelevant investigations and unnecessary agent-time charges.
-
Teams deploy an incident agent without connecting service topology, application design, ownership, organizational context, code and deployment history, or the telemetry sources needed for cross-system correlation.
-
Teams rely on ad hoc prompts and do not create reusable Skills for recurring investigation procedures, triage rules, tool usage, and operational practices.
-
Teams leave ticketing and collaboration tools disconnected, forcing responders to copy investigation context by hand.
-
Teams expose broad write-capable tools through custom integrations or MCP servers without least-privilege access, tool allowlisting, approval, and audit controls.
-
Teams treat agent-generated hypotheses as confirmed root causes or execute mitigation plans without validation.
-
Teams automate symptoms without addressing the underlying cause.
-
Teams apply disruptive remediation to minor problems, such as replacing an instance for a small memory increase or failing over a database for a slow query.
-
Teams trigger a fix in one component that creates a performance problem elsewhere, causing chain reactions.
-
Teams remediate an issue without verifying that the action worked.
-
Teams close incidents without reviewing history for recurring patterns and preventive improvements.
Benefits of establishing this best practice:
-
You reduce mean time to recovery because investigation begins before the responding engineer completes manual context gathering.
-
You control investigation volume and agent usage cost by separating alert signals from incidents that require analysis.
-
Responders start with correlated evidence, affected dependencies, root-cause hypotheses, and mitigation options instead of an isolated alarm.
-
You reduce user impact by acting on predictive signals for well-understood failure modes.
-
You free operations staff to concentrate on complex issues that require judgment.
-
You build institutional knowledge from playbooks, investigation history, and proactive recommendations.
-
You keep automated responses proportionate and auditable, avoiding disruptive actions for minor issues.
Level of risk exposed if this best practice is not established: Low
Implementation guidance
Proactive remediation acts on a signal that predicts a problem before it occurs. A reactive automation that restarts an instance after latency has spiked still leaves users with a bad experience during the spike. A proactive workflow watches a leading indicator, such as a KPI trending toward its limit, and acts while headroom remains. Forecast-based signals require baselines and dashboards that define normal behavior before they identify a meaningful deviation.
Use an incident agent as the first investigation path for selected alerts and tickets. AWS DevOps Agent starts an investigation from alarming or ticketing systems and uses a continuously updated application topology to trace dependencies. It correlates metrics, logs, traces, code changes, deployment history, and AWS Health events across the topology to identify root cause, assess blast radius, or return ranked hypotheses.
Define triage criteria using severity, customer impact, actionability, confidence, maintenance state, and whether an active investigation already covers the same symptom. AWS DevOps Agent triage links related events to an active investigation, skips events that match criteria defined in an Incident Triage Skill, or proceeds with a new investigation. Route critical and actionable signals automatically and sample or suppress low-value signals according to documented policy. Investigations are billed by agent-second, so indiscriminate routing adds cost, creates noise, and pollutes the incident history used for future recommendations.
Connect the agent to the systems that hold operational context. Native integrations include Amazon CloudWatch and third-party observability platforms, CI/CD and code repositories, ServiceNow, PagerDuty, and Slack. Webhooks and Amazon EventBridge connect additional incident workflows. Custom MCP servers expose proprietary telemetry, runbooks, and operational systems through structured tools. Limit investigation tools to read access, grant each Agent Space only the required tools and accounts, and record tool invocations in an audit trail.
Telemetry and topology don't contain all the context an investigation needs. Add workload purpose, critical user journeys, SLOs, service ownership, architecture decisions, known constraints, failure modes, maintenance windows, deployment practices, and escalation policy. Keep this context, topology, and integration credentials current.
Address the underlying cause. An automation that restarts a service each time memory climbs masks a leak while the defect remains. Keep the response proportionate. A database failover for a slow query, or an instance replacement for a small memory increase, causes more disruption than the original condition.
Bound each workflow with least-privilege permissions, idempotency, concurrency limits, cooldown periods, approval gates for high-impact actions, and an explicit rollback path. Verify success with a signal independent from the trigger. If memory pressure triggers a remediation, confirm customer-facing latency, error rate, and workload health. A lower memory reading alone doesn't prove recovery. Escalate to a person when a workflow cannot verify or safely revert its action.
AWS DevOps Agent Skills encode recurring investigation procedures, triage and skip criteria, architectural standards, operational practices, organizational knowledge, and instructions for using native or MCP tools. Skills replace repeated one-off prompts and guide the agent's reasoning. Deterministic runbooks remain the executable layer for approved remediation. Keep agent-generated root-cause hypotheses and mitigation plans separate from execution until evidence and policy support the action.
AWS DevOps Agent proactive incident prevention evaluates investigation history to identify recurring patterns and targeted improvements across observability, infrastructure, governance, and code. Review these recommendations on a schedule, assign owners, validate them against workload evidence, and track whether implemented changes reduce recurrence or detection and recovery time.
Amazon CloudWatch alarms initiate workflows, and CloudWatch investigations provides read-only AI-assisted telemetry analysis. AWS DevOps Agent performs topology-aware investigation and coordinates findings through integrated systems. AWS Systems Manager Automation runs approved deterministic remediation runbooks. Amazon CloudWatch Synthetics canaries and application health checks provide independent verification. AWS Fault Injection Service rehearses investigation and remediation with controlled failures in non-production environments.
Implementation steps
-
Plan incident routing and triage: Identify which alarm and ticket conditions represent an incident that requires investigation. Define severity, customer-impact, actionability, deduplication, maintenance-window, sampling, and skip criteria. Route critical and actionable signals automatically, link related events to active investigations, and suppress or sample low-value signals.
-
Provide topology and operational context: Configure the agent's AWS accounts, application topology, observability sources, code repositories, deployment pipelines, and AWS Health context. Add workload purpose, critical user journeys, SLOs, service ownership, architecture decisions, known constraints, maintenance windows, and escalation policy. Add only the integrations required for the workload.
-
Create reusable Skills: Encode recurring investigation procedures, Incident Triage skip rules, operational practices, organizational knowledge, and instructions for native and MCP tools in AWS DevOps Agent Skills. Version and review Skills when the workload, tools, ownership, or process changes.
-
Identify and prioritize remediation candidates: Use investigation history and performance telemetry to rank recurring issues by frequency, customer impact, confidence in the root cause, reversibility, and remediation blast radius. Automate the high-impact, well-understood, reversible cases first.
-
Define remediation logic and controls: Write a process that addresses the underlying cause and keeps the action proportionate. Define least-privilege permissions, idempotency, concurrency limits, cooldown periods, approval requirements, rollback criteria, and the conditions that require human escalation.
-
Automate approved remediation: Implement deterministic actions with AWS Systems Manager Automation, Amazon EventBridge targets, or equivalent tooling. Keep agent-generated mitigation plans separate from execution until the plan satisfies approval and policy requirements.
-
Verify the result independently: Define customer-facing and workload-health success criteria. Use Amazon CloudWatch Synthetics canaries, application health checks, and KPIs that are independent from the trigger. Roll back or escalate when verification fails.
-
Test investigation and remediation: Validate routing, triage, Skills, integrations, permissions, approvals, actions, verification, rollback, and escalation in a pre-production environment. Use AWS Fault Injection Service to introduce controlled conditions and confirm that automation does not create a chain reaction.
-
Document and train: Record incident-routing policy, topology and context sources, investigation logic, Skills, automated runbooks, approval points, rollback, and escalation paths. Train responders to review evidence and take over when automation reaches its boundary.
-
Review incident history and proactive recommendations: Run periodic evaluations of resolved investigations. Prioritize recommendations that improve observability, capacity, deployment governance, code, and recurring failure modes. Assign owners and measure recurrence, detection time, and recovery time after implementation.
Resources
Related best practices:
Related documents:
Related videos:
Related examples:
Related services: