View a markdown version of this page

ADVOPS03-BP01 Create runbooks for the most common operational events and incidents that can impact your advertising workload - Video Streaming Advertising Lens
Implementation guidanceKey AWS servicesResources

ADVOPS03-BP01 Create runbooks for the most common operational events and incidents that can impact your advertising workload

Develop structured procedures to manage operational events and incidents in your advertising workloads. Runbooks provide step-by-step procedures for well-understood, routine operations, while playbooks guide your response to incidents with less predictable outcomes. These documented procedures assist teams to respond consistently and effectively, reducing human error and improving operational resilience.

Implementation guidance

  1. Use automation for predictable operational responses:

    • Implement auto scaling and load balancing using AWS services to handle common traffic patterns:

      • Configure Amazon EC2 Auto Scaling groups with appropriate scaling policies based on advertising traffic patterns.

      • Set up Elastic Load Balancing to distribute traffic across healthy instances.

      • Implement predictive scaling based on historical data for cyclical advertising campaigns.

      • Configure scheduled scaling for planned high-traffic events like product launches or holiday promotions.

    • For scenarios where auto scaling may not be sufficient:

      • Create runbooks for requesting additional EC2 capacity through ODCRs before anticipated high-traffic events.

      • Implement AWS Systems Manager automation documents to execute common scaling procedures.

      • Use AWS Auto Scaling for predictive scaling based on historical data patterns.

      • Configure CloudWatch alarms to trigger automated responses for common capacity issues.

  2. Create purpose-built runbooks and playbooks for different advertising scenarios:

    • Infrastructure capacity runbooks:

      • Document procedures for submitting on-demand capacity requests (ODCRs) for anticipated high-traffic events

      • Create step-by-step guides for scaling resources up or down based on traffic patterns

      • Establish processes for capacity planning before major advertising campaigns

      • Define monitoring thresholds that trigger capacity management procedures

    • Ad fraud incident playbooks:

      • Document investigation procedures for detected fraud patterns (bot traffic, click fraud, impression fraud)

      • Establish clear escalation paths for high-impact fraud incidents with defined severity levels

      • Create detailed workflows for fraud investigation and mitigation, including evidence collection

      • Define recovery procedures to restore normal operations after fraud mitigation

      • Document procedures for ads.txt and sellers.json verification and maintenance

      • Implement AWS Marketplace solutions like HUMAN for automated fraud detection and prevention

      • Create operational workflows for real-time fraud detection using Amazon SageMaker AI ML models

    • Brand safety incident playbooks:

      • Document procedures for content moderation escalations and violations

      • Establish workflows for emergency ad creative approval and rejection

      • Create processes for brand safety incident response, including stakeholder communication

      • Define procedures for implementing emergency blocking rules in ad serving systems

      • Implement content moderation workflows using Amazon Rekognition for image/video analysis and Amazon Comprehend for text analysis

      • Create escalation procedures for high-risk content identification with automated alerts via Amazon EventBridge

      • Document processes for regular updates to content moderation models in SageMaker AI

    • Measurement anomaly runbooks:

      • Document step-by-step procedures for investigating common measurement discrepancies

      • Establish workflows for cross- data reconciliation and validation

      • Create processes for measurement system recalibration and data correction

      • Define verification steps to confirm resolution of measurement issues

    • AI measurement system playbooks:

      • Document operational procedures for AI model training, validation, and deployment

      • Create monitoring workflows for detecting model drift and performance degradation

      • Establish human oversight processes for AI-based measurement systems

      • Document recovery procedures for AI system failures

      • Implement continuous improvement workflows using AWS Step Functions to orchestrate model retraining

      • Create operational procedures for managing model versions and deployments

  3. Implement continuous improvement for runbooks and playbooks:

    • Review and update documentation after each incident or significant operational event

    • Conduct regular simulation exercises to validate playbook effectiveness

    • Incorporate lessons learned into revised procedures

    • Track key metrics on playbook execution efficiency and outcome effectiveness

Key AWS services

  • Amazon EC2 Auto Scaling

  • Elastic Load Balancing

  • AWS Systems Manager

  • Amazon CloudWatch

  • AWS Auto Scaling

  • Amazon SageMaker AI

  • Amazon Rekognition

  • Amazon Comprehend

  • AWS Lambda

  • AWS Step Functions

  • Amazon EventBridge

  • Amazon Route 53

  • AWS Marketplace solutions (like HUMAN)

Resources