REL12-BP05 Conduct game days regularly
Conduct game days to regularly exercise your procedures for responding to workload-impacting events and impairments. Involve the same teams who would be responsible for handling production scenarios. These exercises help enforce measures to prevent user impact caused by production events. When you practice your response procedures in realistic conditions, you can identify and address any gaps or weaknesses before a real event occurs.
Game days simulate events in production-like environments to test systems, processes, and team responses. The purpose is to perform the same actions the team would perform as if the event actually occurred. These exercises help you understand where improvements can be made and can help develop organizational experience in dealing with events and impairments. These should be conducted regularly so that your team knows builds ingrained habits for how to respond.
Game days prepare teams to handle production events with greater confidence. Teams that are well-practiced are more able to quickly detect and respond to various scenarios. This results in a significantly improved readiness and resilience posture.
Desired outcome: You run resilience game days on a consistent, scheduled basis. These game days are seen as a normal and expected part of doing business. Your organization has built a culture of preparedness, and when production issues occur, your teams are well-prepared to respond effectively, resolve the issues efficiently, and mitigate the impact on customers.
Common anti-patterns:
-
You document your procedures, but your never exercise them.
-
You exclude business decision makers in the test exercises.
-
You run a game day, but you don't inform all relevant stakeholders.
-
You focus solely on technical failures, but you don't involve business stakeholders.
-
You don't incorporate lessons learned from game days into your recovery processes.
-
You blame teams for failures or bugs.
Benefits of establishing this best practice:
-
Enhance response skills: On game days, teams practice their duties and test their communication mechanisms during simulated events, which creates a more coordinated and efficient response in production situations.
-
Identify and address dependencies: Complex environments often involve intricate dependencies between various systems, services, and components. Game days can help you identify and address these dependencies, and verify that your critical systems and services are properly covered by your runbook procedures and can be scaled up or recovered in a timely manner.
-
Foster a culture of resilience: Game days can help cultivate a mindset of resilience within an organization. When you involve cross-functional teams and stakeholders, these exercises promote awareness, collaboration, and a shared understanding of the importance of resilience across the entire organization.
-
Continuous improvement and adaptation: Regular game days help you to continually assess and adapt your resilience strategies, which keeps them relevant and effective in the face of changing circumstances.
-
Increase confidence in the system: Successful game days can help you build confidence in the system's ability to withstand and recover from disruptions.
Level of risk exposed if this best practice is not established: Medium
Implementation guidance
Once you have designed and implemented the necessary resilience measures, conduct a game day to validate that everything works as planned in production. A game day, especially the first one, should involve all team members, and all stakeholders and participants should be informed in advance about the date, time, and simulated scenarios.
During the game day, the involved teams simulate various events and potential scenarios according to the prescribed procedures. The participants closely monitor and assess the impact of these simulated events. If the system operates as designed, the automated detection, scaling, and self-healing mechanisms should activate and result in little to no impact on users. If the team observes any negative impact, they roll back the test and remedy the identified issues, either through automated means or manual intervention documented in the applicable runbooks.
To continuously improve resilience, it's critical to document and incorporate lessons learned. This process is a feedback loop that systematically captures insights from game days and uses them to enhance systems, processes, and team capabilities.
To help you reproduce real-world scenarios where system components or services may fail unexpectedly, inject simulated faults as a game day exercise. Teams can test the resilience and fault tolerance of their systems and simulate their incident response and recovery processes in a controlled environment.
In AWS, your game days can be carried out with replicas of your
production environment using infrastructure as code. Through this
process, you can test in a safe environment that closely resembles
your production environment. Consider
AWS Fault Injection Service
Implementation steps
-
Establish a game day program: Develop a structured program that defines the frequency, scope and objectives of game days. Involve key stakeholders and subject matter experts in planning and running these exercises.
-
Prepare the game day:
-
Identify the key business-critical services that are the focus of the game day. Catalog and map the people, processes, and technologies that support those services.
-
Set the agenda for the game day, and prepare the involved teams to participate in the event. Prepare your automation services to simulate the planned scenarios and run the appropriate recovery processes. AWS services such as AWS Fault Injection Service
, AWS Step Functions , and AWS Systems Manager can help you automate various aspects of game days, such as injection of faults and initiation of recovery actions.
-
-
Run your simulation: On the game day, run the planned scenario. Observe and document how the people, processes, and technologies react to the simulated event.
-
Conduct post-exercise reviews: After the game day, conduct a retrospective session to review the lessons learned. Identify areas for improvement and any actions needed to improve operational resilience. Document your findings, and track any necessary changes to enhance your resilience strategies and preparedness to completion.
Resources
Related best practices:
Related documents:
Related videos:
Related examples:
