Detect and resolve drift in AWS Control Tower
Identifying and resolving drift is a regular operations task for AWS Control Tower management account administrators. Resolving drift helps to ensure your compliance with governance requirements.
When you create your landing zone, the landing zone and all the organizational units (OUs), accounts, and resources are compliant with the governance rules enforced by your chosen controls. As you and your organization members use the landing zone, changes in this compliance status may occur. Some changes may be accidental, and some may be made intentionally to respond to time-sensitive operational events.
Drift detection assists you in identifying resources that need changes or configuration updates to resolve the drift.
Detecting drift
AWS Control Tower detects drift automatically. To detect drift, the
AWSControlTowerAdmin role requires persistent access to your management
account so AWS Control Tower can make read-only API calls to AWS Organizations. These API calls show up as
AWS CloudTrail events.
Drift is surfaced in the Amazon Simple Notification Service (Amazon SNS) notifications that are aggregated in the audit account. Notifications in each member account send alerts to a local Amazon SNS topic, and to a Lambda function.
For controls that are part of the AWS Security Hub Service-Managed Standard: AWS Control Tower, drift is shown on the Account and Account details pages in the AWS Control Tower console, as well as by means of an Amazon SNS notification.
Member account administrators can (and as a best practice, they should) subscribe to
the SNS drift notifications for specific accounts. For example, the
aws-controltower-AggregateSecurityNotifications SNS topic provides
drift notifications. The AWS Control Tower console indicates to management account administrators
when drift has occurred. For more information about SNS topics for drift detection and
notification, see Drift prevention and notification.
Drift notification de-duplication
If the same type of drift occurs on the same set of resources multiple times, AWS Control Tower sends an SNS notification only for the initial instance of drift. If AWS Control Tower detects that this instance of drift has been remediated, it sends another notification only if drift re-occurs for those identical resources.
Examples: Account drift and SCP drift are handled in the following manner
-
If you modify the same managed SCP multiple times, you receive a notification for the first time you modify it.
-
If you modify a managed SCP, then remediate drift, then modify it again, you'll receive two notifications.
-
If an account is moved between the same source and destination OUs multiple times, without repairing the drift first, a single notification is sent, even though the account moved between those OUs more than one time.
Types of account drift
-
Account moved between OUs
-
Account removed from organization
Note
When you move an account from one OU to another, the controls from the previous OU are not removed. If you enable any new hook-based control on the destination OU, the old hook-based control is removed from the account, and the new control replaces it. Controls implemented with SCPs and AWS Config rules always must be removed manually when an account changes OUs.
Types of policy drift
-
SCP updated
-
SCP attached to OU
-
SCP detached from OU
-
SCP attached to account
For more information, see Types of Governance Drift.
Resolving drift
Although detection is automatic, the steps to resolve drift must be done manually through the console, or for controls, by calling the ResetEnabledControl API.
-
Many types of drift can be resolved through the Landing zone settings page. You can choose the Reset button in the Versions section to resolve these types of drift.
-
If your OU has fewer than 1000 accounts, you can resolve drift in Account Factory provisioned accounts, or SCP drift, by selecting Re-register OU on the Organization page or the OU details page.
-
You may be able to resolve account drift, such as Moved member account, by updating an individual account. For more information, see Update the account in the console.
For controls, many types of drift can be resolved by calling the ResetEnabledControl API.
When you take action to resolve drift on a landing zone version, two behaviors are possible.
-
If you are on the latest landing zone version, when you choose Reset and then choose Confirm, your drifted landing zone resources are reset to the saved AWS Control Tower configuration. The landing zone version stays the same.
-
If you are not on the latest version, you must choose Update. The landing zone is upgraded to the latest landing zone version. Drift is resolved as part of this process.
Considerations about drift and SCP scans
AWS Control Tower scans your managed SCPs daily to verify that the corresponding controls are applied correctly and that they have not drifted. To retrieve the SCPs and run checks on them, AWS Control Tower calls AWS Organizations on your behalf, using a role in your management account.
If an AWS Control Tower scan discovers drift, you'll receive a notification. AWS Control Tower sends only one notification per drift issue, so if your landing zone already is in a state of drift, you won't receive additional notifications unless a new drift item is found.
AWS Organizations limits how often each of its APIs can be called. This limit is expressed in transactions per second (TPS), and known as the TPS limit, throttling rate, or API request rate. When AWS Control Tower audits your SCPs by calling AWS Organizations, the API calls that AWS Control Tower makes are counted towards your TPS limit, because AWS Control Tower uses the management account to make the calls.
In rare situations, this limit can be reached when you call the same APIs repeatedly,
whether through a third-party solution or a custom script you wrote. For example, if you
and AWS Control Tower call the same AWS Organizations APIs at the same moment in time (within 1 second),
and the TPS limits are reached, subsequent calls are throttled. That is, these calls
return an error such as Rate exceeded.
If an API request rate is exceeded
-
If AWS Control Tower hits the limit and is throttled, we pause the execution of the audit and resume it at a later time.
-
If your workload hits the limit and is throttled, the result can range from slight latency all the way to a fatal error in the workload, depending on how the workload is configured. This edge case is something to be aware of.
A daily SCP scan consists of
-
Retrieving your recently active OUs.
-
For each registered OU, retrieving all SCPs managed by AWS Control Tower that are attached to the OU. Managed SCPs have identifiers that begin with
aws-guardrails. -
For each preventive control enabled on the OU, verifying that the control's policy statement is present in the OU's managed SCPs.
An OU may have one or more managed SCPs.
Types of drift to resolve right away
Most types of drift can be resolved by administrators. A few types of drift must be resolved immediately, including deletion of an organizational unit that the AWS Control Tower landing zone requires. Here are some examples of major drift that you may wish to avoid:
-
Don't delete the Security OU: The organizational unit originally named Security during landing zone setup by AWS Control Tower should not be deleted. If you delete it, you'll see an error message instructing you to reset the landing zone immediately. You won't be able to take any other actions in AWS Control Tower until the reset is complete.
-
Don't delete required roles: AWS Control Tower checks certain AWS Identity and Access Management (IAM) roles when you log into the console for IAM role drift. If these roles are missing or inaccessible, you'll see an error page instructing you to reset your landing zone. These roles are
AWSControlTowerAdminAWSControlTowerCloudTrailRoleAWSControlTowerStackSetRole.For more information about these roles, see Permissions Required to Use the AWS Control Tower Console.
-
Don't delete all Additional OUs: If you delete the organizational unit originally named Sandbox during landing zone setup by AWS Control Tower, your landing zone will be in a state of drift, but you still can use AWS Control Tower. At least one Additional OU is required for AWS Control Tower to operate, but it doesn’t have to be the Sandbox OU.
-
Don't remove shared accounts: If you remove shared accounts from Foundational OUs, such as removing the logging account from the Security OU, your landing zone will be in a state of drift. The landing zone must be reset before you can continue using the AWS Control Tower console.
Repairable changes to resources
Here's a list of changes to AWS Control Tower resources that are permitted, although they create resolvable drift. Results of these permitted operations are viewable in the AWS Control Tower console, although a refresh may be required.
For more information about how to resolve the resulting drift, see Managing Resources Outside of AWS Control Tower.
Changes permitted outside the AWS Control Tower console
-
Change the name of a registered OU.
-
Change the name of the Security OU.
-
Change the name of member accounts in non-Foundational OUs.
-
Change the name of AWS Control Tower shared accounts in the Security OU.
-
Delete a non-Foundational OU.
-
Delete an enrolled account from a non-Foundational OU.
-
Change the email address of a shared account in the Security OU.
-
Change the email address of a member account in a registered OU.
Note
Moving accounts between OUs is considered drift, and it must be resolved.
Drift and new account provisioning
If your landing zone is in a state of drift, the Enroll account feature in AWS Control Tower will not work. In that case, you must provision new accounts through AWS Service Catalog. For instructions, see Provision accounts with AWS Service Catalog Account Factory .
In particular, if you've made certain changes to your accounts by means of Service Catalog, such as changing the name of your portfolio, the Enroll account feature will not work.
