Troubleshooting assessment and evidence collection issues - AWS Audit Manager

Troubleshooting assessment and evidence collection issues

You can use the information on this page to resolve common assessment and evidence collection issues in Audit Manager.

I created an assessment but I can’t see any evidence yet

If you can't see any evidence, it's likely that you either didn't wait at least 24 hours after you created the assessment or that there's a configuration error.

We recommend that you check the following:

  1. Make sure that 24 hours passed since you created the assessment. Automated evidence becomes available 24 hours after you create the assessment.

  2. Make sure that you’re using Audit Manager in the same AWS Region as the AWS service that you’re expecting to see evidence for.

  3. If you expect to see compliance check evidence from AWS Config and AWS Security Hub, make sure that both the AWS Config and Security Hub consoles display results for these checks. The AWS Config and Security Hub results should display in the same AWS Region that you use Audit Manager in.

If you still can't see evidence in your assessment and it's not due to one of these issues, check the other potential causes that are described on this page.

My assessment isn’t collecting compliance check evidence from AWS Security Hub

If you don't see compliance check evidence for an AWS Security Hub control, this could be due to one of the following issues.

Missing configuration in AWS Security Hub

This issue can be caused if you missed some configuration steps when you enabled AWS Security Hub.

Make sure that you enabled Security Hub and configured your settings as follows.

A Security Hub control name was entered incorrectly in your ControlMappingSource

When you use the Audit Manager API to create a custom control, you can specify a Security Hub control as a data source mapping for evidence collection. To do this, you enter a control ID as the keywordValue.

If you don't see compliance check evidence for a Security Hub control, it could be that the keywordValue was entered incorrectly in your ControlMappingSource. The keywordValue is case sensitive. If you enter it incorrectly, Audit Manager might not recognize that rule. As a result, you might not collect compliance check evidence for that control as expected.

To fix this issue, update the custom control and revise the keywordValue. The correct format of a Security Hub keyword varies. For accuracy, reference the list of supported Security Hub control keywords.

AuditManagerSecurityHubFindingsReceiver Amazon EventBridge rule is missing

When you enable Audit Manager, a rule named AuditManagerSecurityHubFindingsReceiver is automatically created and enabled in Amazon EventBridge. This rule enables Audit Manager to collect Security Hub findings as evidence.

If this rule isn't listed and enabled in the AWS Region where you use Security Hub, Audit Manager can't collect Security Hub findings for that Region.

To resolve this issue, go to the EventBridge console and confirm that the AuditManagerSecurityHubFindingsReceiver rule exists in your AWS account. If the rule doesn't exist, we recommend that you disable Audit Manager and then re-enable the service. If this action doesn’t resolve the issue, or if disabling Audit Manager isn’t an option, contact AWS Support for assistance.

Service-linked AWS Config rules created by Security Hub

Keep in mind that Audit Manager doesn’t collect evidence from the service-linked AWS Config rules that Security Hub creates. This is a specific type of managed AWS Config rule that's enabled and controlled by the Security Hub service. Security Hub creates instances of these service-linked rules in your AWS environment, even if other instances of the same rules already exist. As a result, to prevent evidence duplication, Audit Manager doesn’t support evidence collection from the service-linked rules.

My assessment isn’t collecting compliance check evidence from AWS Config

If you don't see compliance check evidence for an AWS Config rule, this could be due to one of the following issues.

The rule identifier was entered incorrectly in your ControlMappingSource

When you use the Audit Manager API to create a custom control, you can specify an AWS Config rule as a data source mapping for evidence collection. The keywordValue that you specify depends on the type of rule.

If you don't see compliance check evidence for an AWS Config rule, it could be that the keywordValue was entered incorrectly in your ControlMappingSource. The keywordValue is case sensitive. If you enter it incorrectly, Audit Manager might not recognize the rule. As a result, you might not collect compliance check evidence for that rule as intended.

To fix this issue, update the custom control and revise the keywordValue.

  • For custom rules, make sure that the keywordValue has the Custom_ prefix followed by the custom rule name. The format of the custom rule name may vary. For accuracy, visit the AWS Config console to verify your custom rule names.

  • For managed rules, make sure that the keywordValue is the rule identifier in ALL_CAPS_WITH_UNDERSCORES. For example, CLOUDWATCH_LOG_GROUP_ENCRYPTED. For accuracy, reference the list of supported managed rule keywords.

    Note

    For some managed rules, the rule identifier is different from the rule name. For example, the rule identifier for restricted-ssh is INCOMING_SSH_DISABLED. Make sure to use the rule identifier, not the rule name. To find a rule identifier, choose a rule from the list of managed rules and look for its Identifier value.

The rule is a service-linked AWS Config rule

You can use managed rules and custom rules as a data source mapping for evidence collection. However, Audit Manager doesn’t collect evidence from most service-linked rules.

There are only two types of service-linked rule that Audit Manager collects evidence from:

  • Service-linked rules from Conformance Packs

  • Service-linked rules from AWS Organizations

Audit Manager doesn't collect evidence from other service-linked rules, specifically any rules with an Amazon Resource Name (ARN) that contains the following prefix: arn:aws:config:*:*:config-rule/aws-service-rule/...

The reason that Audit Manager doesn't collect evidence from most service-linked AWS Config rules is to prevent duplicate evidence in your assessments. A service-linked rule is a specific type of managed rule that enables other AWS services to create AWS Config rules in your account. For example, some Security Hub controls use an AWS Config service-linked rule to run security checks. For each Security Hub control that uses a service-linked AWS Config rule, Security Hub creates an instance of the required AWS Config rule in your AWS environment. This happens even if the original rule already exists in your account. Therefore, to avoid collecting the same evidence from the same rule twice, Audit Manager ignores the service-linked rule and doesn't collect evidence from it.

AWS Config isn't enabled and included as a service in scope

AWS Config must be enabled in your AWS account. It must also be included as a service in scope for your assessment. After you've set up AWS Config in this way, Audit Manager collects evidence each time the evaluation of an AWS Config rule occurs.

First, make sure that you enabled AWS Config in your AWS account. For instructions, see Enable and set up AWS Config.

Next, make sure that you included AWS Config as a service in scope for your assessment. To review the current services in scope for your assessment, see Review an assessment, AWS services tab. To edit the list of services in scope for an assessment, see Edit AWS services in scope.

The AWS Config rule evaluated a resource configuration before you set up your assessment

If your AWS Config rule is set up to evaluate configuration changes for a specific resource, you might see a mismatch between the evaluation in AWS Config and the evidence in Audit Manager. This happens if the rule evaluation occurred before you set up the control in your Audit Manager assessment. In this case, Audit Manager doesn't generate evidence until the underlying resource changes state again and triggers a re-evaluation of the rule.

As a workaround, you can navigate to the rule in the AWS Config console and manually re-evaluate the rule. This invokes a new evaluation of all of the resources that pertain to that rule.

My assessment isn’t collecting user activity evidence from AWS CloudTrail

When you use the Audit Manager API to create a custom control, you can specify a CloudTrail event name as a data source mapping for evidence collection. To do so, you enter the event name as the keywordValue.

If you don't see user activity evidence for a CloudTrail event, it could be that the keywordValue was entered incorrectly in your ControlMappingSource. The keywordValue is case sensitive. If you enter it incorrectly, Audit Manager might not recognize the event name. As a result, you might not collect user activity evidence for that event as intended.

To fix this issue, update the custom control and revise the keywordValue. Make sure that the event is written as serviceprefix_ActionName. For example, cloudtrail_StartLogging. For accuracy, review the AWS service prefix and action names in the Service Authorization Reference.

My assessment isn’t collecting configuration data evidence for an AWS API call

When you use the Audit Manager API to create a custom control, you can specify an AWS API call as a data source mapping for evidence collection. To do so, you enter the API call as the keywordValue.

If you don't see configuration data evidence for an AWS API call, it could be that the keywordValue was entered incorrectly in your ControlMappingSource. The keywordValue case sensitive. If you enter it incorrectly, Audit Manager might not recognize the API call. As a result, you might not collect configuration data evidence for that API call as intended.

To fix this issue, update the custom control and revise the keywordValue. Make sure that the API call is written as serviceprefix_ActionName. For example, iam_ListGroups. For accuracy, reference the list of supported API calls.

My assessment isn’t collecting evidence from another AWS service

If an AWS service isn't selected as in scope for your assessment, Audit Manager doesn't collect evidence from resources related to that service. This is also the case if an AWS service is selected but you haven't enabled it in your environment.

If you created your assessment from a custom framework, you can edit the services in scope for your assessment. You can then specify additional AWS services that you want to collect evidence from. After you add these services, evidence becomes available after 24 hours.

Note

If you created your assessment from a standard framework, the list of AWS services in scope is preselected and can’t be edited. This is because when you create an assessment from a standard framework, Audit Manager automatically maps and selects the relevant data sources and services for you. The selection is made based on the requirements of the standard framework. Note that, for standard frameworks that contain manual controls only, no AWS services are in scope.

The workaround for editing the AWS services in scope while still creating an assessment based on a standard framework is to customize the standard framework. By using this workaround, you can use the framework that you customized to create a new assessment. In this assessment, you can then specify which AWS services are in scope.

My evidence is generated at different intervals, and I'm not sure how often it’s being collected

The controls in Audit Manager assessments are mapped to various data sources. Each data source has a different evidence collection frequency. As a result, there’s no one-size-fits-all answer for how often evidence is collected. Some data sources evaluate compliance, whereas others only capture resource state and change data without a compliance determination.

The following is a summary of the different data source types and how often they collect evidence.

Data source type Description Evidence collection frequency When this control is active in an assessment

AWS CloudTrail

Tracks a specific user activity.

Continual

Audit Manager filters your CloudTrail logs based on the keyword that you choose. The processed logs are imported as User activity evidence.

AWS Security Hub

Captures a snapshot of your resource security posture by reporting findings from Security Hub.

Based on the schedule of the Security Hub check (typically around every 12 hours)

Audit Manager retrieves the security finding directly from Security Hub. The finding is imported as Compliance check evidence.

AWS Config

Captures a snapshot of your resource security posture by reporting findings from AWS Config.

Based on the settings that are defined in the AWS Config rule Audit Manager retrieves the rule evaluation directly from AWS Config. The evaluation is imported as Compliance check evidence.
AWS API calls

Takes a snapshot of your resource configuration directly through an API call to the specified AWS service.

Daily, weekly, or monthly Audit Manager makes the API call based on the frequency that you specify. The response is imported as Configuration data evidence.

Regardless of the evidence collection frequency, new evidence is collected automatically for as long as the assessment is active. For more information, see Evidence collection frequency.

To learn more, see Supported control data sources for automated evidence and Changing the evidence collection frequency for a control.

What happens if I remove an in-scope account from my organization?

When an in-scope account is removed from your organization, Audit Manager no longer collects evidence for that account. However, the account continues to show in your assessment under the AWS accounts tab. To remove the account from the list of accounts in scope, edit the assessment. The removed account no longer shows in the list during editing, and you can save your changes without that account in scope.

I can't edit the services in scope for my assessment

When you use the Audit Manager console to create an assessment from a standard framework, the list of AWS services in scope is selected by default. This list can’t be edited. This is because Audit Manager automatically maps and selects the data sources and services for you. This selection is made according to the requirements of the standard framework. If the standard framework that you selected contains only manual controls, no AWS services are in scope for your assessment, and you can't add any services to your assessment.

If you need to edit the list of services in scope, use the UpdateAssessment API operation that's provided by Audit Manager. Alternatively, you can customize the standard framework and then create an assessment from the custom framework.

What's the difference between a service in scope and a data source type?

A service in scope is an AWS service that's specified as part of your assessment. When a service is in scope, Audit Manager collects evidence about your usage of that service and its resources.

A data source type indicates where exactly the evidence is collected from. If you upload your own evidence, the data source type is Manual. If Audit Manager collects the evidence, the data source can be one of four types.

  1. AWS Security Hub – Captures a snapshot of your resource security posture by reporting findings from Security Hub.

  2. AWS Config – Captures a snapshot of your resource security posture by reporting findings from AWS Config.

  3. AWS CloudTrail – Tracks a specific user activity for a resource.

  4. AWS API calls – Takes a snapshot of your resource configuration directly through an API call to a specific AWS service.

Here are two examples to illustrate the difference between a service in scope and a data source type.

Example 1

Let's say that you want to collect evidence for a control that's named 4.1.2 - Disallow public write access to S3 buckets. This control checks the access levels of your S3 bucket policies. For this control, Audit Manager uses a specific AWS Config rule (s3-bucket-public-write-prohibited) to look for an evaluation of your S3 buckets. In this example, the following is true:

Example 2

Let's say that you want to collect evidence for a HIPAA control that's named 164.308(a)(5)(ii)(C). This control requires a monitoring procedure for detecting inappropriate sign-ins. For this control, Audit Manager uses CloudTrail logs to look for all AWS Management Console sign-in events. In this example, the following is true:

My assessment creation failed

If your assessment creation fails, it could be because you selected too many AWS accounts in your assessment scope. If you're using AWS Organizations, Audit Manager can support up to approximately 150 member accounts in the scope of a single assessment. If you exceed this number, the assessment creation might fail. As a workaround, you can run multiple assessments with different accounts in scope for each assessment.

I disabled and then re-enabled Audit Manager, and now my pre-existing assessments are no longer collecting evidence

When you disable Audit Manager and choose not to delete your data, your existing assessments move into a dormant state and stop collecting evidence. This means that when you re-enable Audit Manager, the assessments that you previously created remain available. However, they don't automatically resume evidence collection.

To start collecting evidence again for a pre-existing assessment, edit the assessment and choose Save without making any changes.