AWS WAF Bot Control - AWS WAF, AWS Firewall Manager, and AWS Shield Advanced

AWS WAF Bot Control

Bot Control helps you manage bot activity to your site by categorizing and identifying common bots, verifying generally desirable bots, and detecting high confidence signatures of bots. Bot Control combines an AWS managed rule group with AWS WAF features that allow you to customize handling of your bot-related traffic. Bot Control primarily targets self-identifying, non-targeted bots, in order to give you the ability to monitor and control this category of bot traffic.

Bot Control is a managed rule group that gives you visibility and control over common and pervasive bot traffic to your applications. With Bot Control, you can easily monitor, block, or rate-limit bots such as scrapers, scanners, and crawlers. You can also allow common bots like status monitors and search engines. You can protect your applications using the Bot Control managed rule group alone, or with other AWS Managed Rules rule groups and your own custom AWS WAF rules.

Bot Control includes a console dashboard that shows how much of your current traffic is coming from bots, based on request sampling. With the Bot Control managed rule group added to your web ACL, you can take action against bot traffic and receive detailed, real-time information about common bot traffic coming to your applications.

When AWS WAF evaluates a web request against the Bot Control managed rule group, the evaluation adds labels to requests that it detects as bot related. The labels provide information, for example the category and name of the bot, which you can match against in your own custom AWS WAF rules.

The labels that are generated by the Bot Control managed rule group are included in Amazon CloudWatch metrics and your web ACL logs. You can use AWS Firewall Manager AWS WAF policies to deploy the Bot Control managed rule group across your applications in multiple accounts that are part of your organization in AWS Organizations.

Bot Control components

The main components of a Bot Control implementation are the following:

  • AWSManagedRulesBotControlRuleSet – The Bot Control managed rule group whose rules detect and handle various categories of bots. For information about the rule group's rules, see AWS WAF Bot Control rule group. You include this rule group in your web ACL using a managed rule group reference statement. This rule group add labels to web requests that it detects as bot traffic. You are charged additional fees when you use this rule group. For more information, see AWS WAF Pricing.

  • Bot Control dashboard – The bot monitoring dashboard for your web ACL, available through the web ACL Bot Control tab. Use this dashboard to monitor your traffic and understand how much of it comes from various types of bots. This can be a starting point for customizing your bot management, as described in this topic. You can also use it to verify your changes and monitor activity for various bots and bot categories.

  • Logging and metrics – You can monitor your bot traffic and understand how the Bot Control managed rule group evaluates and handles it by configuring and enabling logs and Amazon CloudWatch metrics for your web ACL. The labels that Bot Control adds to your web requests are included in the logs and in Amazon CloudWatch metrics. For information about logging and metrics, see Logging and monitoring web ACL traffic and Monitoring with Amazon CloudWatch.

    Depending on your needs and the traffic that you see, you might want to customize your Bot Control implementation. For example, you might want to exclude some traffic from Bot Control evaluation, or you might want to alter how it handles some of the bot traffic that it identifies, using AWS WAF features like scope-down statements or label matching rules.

  • Scope-down statements – You can limit the scope of the web requests that the Bot Control managed rule group evaluates by adding a scope-down statement inside the Bot Control managed rule group reference statement. A scope-down statement can be any nestable rule statement. Traffic that doesn't match the scope-down statement results as not matching the rule group, and isn't evaluated by the Bot Control managed rule group. For more information about scope-down statements, see Scope-down statements.

    Pricing for the Bot Control managed rule group is based on the number of web requests that AWS WAF evaluates using the rule group. You can help reduce these costs by using a scope-down statement to limit the requests that the rule group evaluates, such as limits by paths or content types. You might find that some parts of your application require more protection than others. For example, you may want to allow your homepage to load for everyone, including bots, but block requests to your application APIs.

  • Labels and label matching rules – You can use the AWS WAF label match rule statement to evaluate the labels that the Bot Control rule group adds to your web requests. This allows you to customize how you handle web requests that are identified by the Bot Control managed rule group. For more information about labeling and using label match statements, see Label match rule statement and AWS WAF labels on web requests.

  • Custom requests and responses – You can add custom headers to requests that you allow and you can send custom responses for requests that you block by pairing label matching with the AWS WAF custom request and response features. For more information about customizing requests and responses, see Customizing web requests and responses in AWS WAF.