Troubleshooting assessment and evidence collection issues

You can use the information on this page to resolve common assessment and evidence collection issues in Audit Manager.

Evidence collection issues

• I created an assessment but I can’t see any evidence yet

• My assessment isn’t collecting compliance check evidence from AWS Security Hub

• My assessment isn’t collecting compliance check evidence from AWS Config

• My assessment isn’t collecting user activity evidence from AWS CloudTrail

• My assessment isn’t collecting configuration data evidence for an AWS API call

• A common control isn’t collecting any automated evidence

• My evidence is generated at different intervals, and I'm not sure how often it’s being collected

• I disabled and then re-enabled Audit Manager, and now my pre-existing assessments are no longer collecting evidence

• On my assessment details page, I’m prompted to recreate my assessment

• What’s the difference between a data source and an evidence source?

Assessment issues

• My assessment creation failed

• What happens if I remove an in-scope account from my organization?

• I can't see the services in scope for my assessment

• I can't edit the services in scope for my assessment

• What's the difference between a service in scope and a data source type?

I created an assessment but I can’t see any evidence yet

If you can't see any evidence, it's likely that you either didn't wait at least 24 hours after you created the assessment or that there's a configuration error.

We recommend that you check the following:

1. Make sure that 24 hours passed since you created the assessment. Automated evidence becomes available 24 hours after you create the assessment.

2. Make sure that you’re using Audit Manager in the same AWS Region as the AWS service that you’re expecting to see evidence for.

3. If you expect to see compliance check evidence from AWS Config and AWS Security Hub, make sure that both the AWS Config and Security Hub consoles display results for these checks. The AWS Config and Security Hub results should display in the same AWS Region that you use Audit Manager in.

If you still can't see evidence in your assessment and it's not due to one of these issues, check the other potential causes that are described on this page.

My assessment isn’t collecting compliance check evidence from AWS Security Hub

If you don't see compliance check evidence for an AWS Security Hub control, this could be due to one of the following issues.

Missing configuration in AWS Security Hub

This issue can be caused if you missed some configuration steps when you enabled AWS Security Hub.

To fix this issue, make sure that you enabled Security Hub with the required settings for Audit Manager. For instructions, see Enable and set up AWS Security Hub (optional).

A Security Hub control name was entered incorrectly in your ControlMappingSource

When you use the Audit Manager API to create a custom control, you can specify a Security Hub control as a data source mapping for evidence collection. To do this, you enter a control ID as the keywordValue.

If you don't see compliance check evidence for a Security Hub control, it could be that the keywordValue was entered incorrectly in your ControlMappingSource. The keywordValue is case sensitive. If you enter it incorrectly, Audit Manager might not recognize that rule. As a result, you might not collect compliance check evidence for that control as expected.

To fix this issue, update the custom control and revise the keywordValue. The correct format of a Security Hub keyword varies. For accuracy, reference the list of Supported Security Hub controls .

AuditManagerSecurityHubFindingsReceiver Amazon EventBridge rule is missing

When you enable Audit Manager, a rule named AuditManagerSecurityHubFindingsReceiver is automatically created and enabled in Amazon EventBridge. This rule enables Audit Manager to collect Security Hub findings as evidence.

If this rule isn't listed and enabled in the AWS Region where you use Security Hub, Audit Manager can't collect Security Hub findings for that Region.

To resolve this issue, go to the EventBridge console and confirm that the AuditManagerSecurityHubFindingsReceiver rule exists in your AWS account. If the rule doesn't exist, we recommend that you disable Audit Manager and then re-enable the service. If this action doesn’t resolve the issue, or if disabling Audit Manager isn’t an option, contact AWS Support for assistance.

Service-linked AWS Config rules created by Security Hub

Keep in mind that Audit Manager doesn’t collect evidence from the service-linked AWS Config rules that Security Hub creates. This is a specific type of managed AWS Config rule that's enabled and controlled by the Security Hub service. Security Hub creates instances of these service-linked rules in your AWS environment, even if other instances of the same rules already exist. As a result, to prevent evidence duplication, Audit Manager doesn’t support evidence collection from the service-linked rules.

I disabled a security control in Security Hub. Does Audit Manager collect compliance check evidence for that security control?

Audit Manager doesn't collect evidence for disabled security controls.

If you set the status of a security control to disabled in Security Hub, no security checks are performed for that control in the current account and Region. As a result, no security findings are available in Security Hub, and no related evidence is collected by Audit Manager.

By respecting the disabled status that you set in Security Hub, Audit Manager ensures that your assessment accurately reflects the active security controls and findings that are relevant to your environment, excluding any controls that you intentionally disabled.

I set the status of a finding to Suppressed in Security Hub. Does Audit Manager collect compliance check evidence about that finding?

Audit Manager collects evidence for security controls that have suppressed findings.

If you set the workflow status of a finding to suppressed in Security Hub, this means that you reviewed the finding and do not believe that any action is needed. In Audit Manager, these suppressed findings are collected as evidence and attached to your assessment. The evidence details show the evaluation status of SUPPRESSED reported directly from Security Hub.

This approach ensures that your Audit Manager assessment accurately represents the findings from Security Hub, while also providing visibility into any suppressed findings that may require further review or consideration in an audit.

My assessment isn’t collecting compliance check evidence from AWS Config

If you don't see compliance check evidence for an AWS Config rule, this could be due to one of the following issues.

The rule identifier was entered incorrectly in your ControlMappingSource

When you use the Audit Manager API to create a custom control, you can specify an AWS Config rule as a data source mapping for evidence collection. The keywordValue that you specify depends on the type of rule.

If you don't see compliance check evidence for an AWS Config rule, it could be that the keywordValue was entered incorrectly in your ControlMappingSource. The keywordValue is case sensitive. If you enter it incorrectly, Audit Manager might not recognize the rule. As a result, you might not collect compliance check evidence for that rule as intended.

To fix this issue, update the custom control and revise the keywordValue.

• For custom rules, make sure that the keywordValue has the Custom_ prefix followed by the custom rule name. The format of the custom rule name may vary. For accuracy, visit the AWS Config console to verify your custom rule names.

• For managed rules, make sure that the keywordValue is the rule identifier in ALL_CAPS_WITH_UNDERSCORES. For example, CLOUDWATCH_LOG_GROUP_ENCRYPTED. For accuracy, reference the list of supported managed rule keywords.

  Note

  For some managed rules, the rule identifier is different from the rule name. For example, the rule identifier for restricted-ssh is INCOMING_SSH_DISABLED. Make sure to use the rule identifier, not the rule name. To find a rule identifier, choose a rule from the list of managed rules and look for its Identifier value.

The rule is a service-linked AWS Config rule

You can use managed rules and custom rules as a data source mapping for evidence collection. However, Audit Manager doesn’t collect evidence from most service-linked rules.

There are only two types of service-linked rule that Audit Manager collects evidence from:

• Service-linked rules from Conformance Packs

• Service-linked rules from AWS Organizations

Audit Manager doesn't collect evidence from other service-linked rules, specifically any rules with an Amazon Resource Name (ARN) that contains the following prefix: arn:aws:config:*:*:config-rule/aws-service-rule/...

The reason that Audit Manager doesn't collect evidence from most service-linked AWS Config rules is to prevent duplicate evidence in your assessments. A service-linked rule is a specific type of managed rule that enables other AWS services to create AWS Config rules in your account. For example, some Security Hub controls use an AWS Config service-linked rule to run security checks. For each Security Hub control that uses a service-linked AWS Config rule, Security Hub creates an instance of the required AWS Config rule in your AWS environment. This happens even if the original rule already exists in your account. Therefore, to avoid collecting the same evidence from the same rule twice, Audit Manager ignores the service-linked rule and doesn't collect evidence from it.

AWS Config isn't enabled

AWS Config must be enabled in your AWS account. After you've set up AWS Config in this way, Audit Manager collects evidence each time the evaluation of an AWS Config rule occurs. Make sure that you enabled AWS Config in your AWS account. For instructions, see Enable and set up AWS Config.

The AWS Config rule evaluated a resource configuration before you set up your assessment

If your AWS Config rule is set up to evaluate configuration changes for a specific resource, you might see a mismatch between the evaluation in AWS Config and the evidence in Audit Manager. This happens if the rule evaluation occurred before you set up the control in your Audit Manager assessment. In this case, Audit Manager doesn't generate evidence until the underlying resource changes state again and triggers a re-evaluation of the rule.

As a workaround, you can navigate to the rule in the AWS Config console and manually re-evaluate the rule. This invokes a new evaluation of all of the resources that pertain to that rule.

My assessment isn’t collecting user activity evidence from AWS CloudTrail

When you use the Audit Manager API to create a custom control, you can specify a CloudTrail event name as a data source mapping for evidence collection. To do so, you enter the event name as the keywordValue.

If you don't see user activity evidence for a CloudTrail event, it could be that the keywordValue was entered incorrectly in your ControlMappingSource. The keywordValue is case sensitive. If you enter it incorrectly, Audit Manager might not recognize the event name. As a result, you might not collect user activity evidence for that event as intended.

To fix this issue, update the custom control and revise the keywordValue. Make sure that the event is written as serviceprefix_ActionName. For example, cloudtrail_StartLogging. For accuracy, review the AWS service prefix and action names in the Service Authorization Reference.

My assessment isn’t collecting configuration data evidence for an AWS API call

When you use the Audit Manager API to create a custom control, you can specify an AWS API call as a data source mapping for evidence collection. To do so, you enter the API call as the keywordValue.

If you don't see configuration data evidence for an AWS API call, it could be that the keywordValue was entered incorrectly in your ControlMappingSource. The keywordValue case sensitive. If you enter it incorrectly, Audit Manager might not recognize the API call. As a result, you might not collect configuration data evidence for that API call as intended.

To fix this issue, update the custom control and revise the keywordValue. Make sure that the API call is written as serviceprefix_ActionName. For example, iam_ListGroups. For accuracy, reference the list of AWS API calls supported by AWS Audit Manager.

A common control isn’t collecting any automated evidence

When you review a common control, you might see the following message: This common control doesn’t collect automated evidence from core controls.

This means that no AWS managed evidence sources can currently support this common control. As a result, the Evidence sources tab is empty and no core controls are displayed.

When a common control doesn’t collect automated evidence, it’s referred to as a manual common control. Manual common controls typically require the provision of physical records and signatures, or details about events that occur outside of your AWS environment. For this reason, there are often no AWS data sources that can produce evidence to support the control’s requirements.

If a common control is manual, you can still use it as an evidence source for a custom control. The only difference is that the common control won’t collect any evidence automatically. Instead, you’ll need to manually upload your own evidence to support the requirements of the common control.

To add evidence to a manual common control

1. Create a custom control

   • Follow the steps to create or edit a custom control.

   • When you specify evidence sources in step 2, choose the manual common control as an evidence source.

2. Create a custom framework

   • Follow the steps to create or edit a custom framework.

   • When you specify a control set in step 2, include your new custom control.

3. Create an assessment

   • Follow the steps to create an assessment from your custom framework.

   • At this point, the manual common control is now an evidence source in an active assessment control.

4. Upload manual evidence

   • Follow the steps to add manual evidence to the control in your assessment.

Note

As more AWS data sources become available in the future, it’s possible that AWS might update the common control to include core controls as evidence sources. In this case, if the common control is an evidence source in one or more of your active assessment controls, you’ll benefit from these updates automatically. No further set up is needed from your side, and you’ll start to collect automated evidence that supports the common control.

My evidence is generated at different intervals, and I'm not sure how often it’s being collected

The controls in Audit Manager assessments are mapped to various data sources. Each data source has a different evidence collection frequency. As a result, there’s no one-size-fits-all answer for how often evidence is collected. Some data sources evaluate compliance, whereas others only capture resource state and change data without a compliance determination.

The following is a summary of the different data source types and how often they collect evidence.

Data source type	Description	Evidence collection frequency	When this control is active in an assessment
AWS CloudTrail	Tracks a specific user activity.	Continual	Audit Manager filters your CloudTrail logs based on the keyword that you choose. The processed logs are imported as User activity evidence.
AWS Security Hub	Captures a snapshot of your resource security posture by reporting findings from Security Hub.	Based on the schedule of the Security Hub check (typically around every 12 hours)	Audit Manager retrieves the security finding directly from Security Hub. The finding is imported as Compliance check evidence.
AWS Config	Captures a snapshot of your resource security posture by reporting findings from AWS Config.	Based on the settings that are defined in the AWS Config rule	Audit Manager retrieves the rule evaluation directly from AWS Config. The evaluation is imported as Compliance check evidence.
AWS API calls	Takes a snapshot of your resource configuration directly through an API call to the specified AWS service.	Daily, weekly, or monthly	Audit Manager makes the API call based on the frequency that you specify. The response is imported as Configuration data evidence.

Regardless of the evidence collection frequency, new evidence is collected automatically for as long as the assessment is active. For more information, see Evidence collection frequency.

To learn more, see Supported data source types for automated evidence and Changing how often a control collects evidence.

I disabled and then re-enabled Audit Manager, and now my pre-existing assessments are no longer collecting evidence

When you disable Audit Manager and choose not to delete your data, your existing assessments move into a dormant state and stop collecting evidence. This means that when you re-enable Audit Manager, the assessments that you previously created remain available. However, they don't automatically resume evidence collection.

To start collecting evidence again for a pre-existing assessment, edit the assessment and choose Save without making any changes.

On my assessment details page, I’m prompted to recreate my assessment

If you see a message that says Create new assessment to collect more comprehensive evidence, this indicates that Audit Manager now provides a new definition of the standard framework that your assessment was created from.

In the new framework definition, all of the framework’s standard controls can now collect evidence from AWS managed sources. This means that whenever there’s an update to the underlying data sources for a common or core control, Audit Manager automatically applies the same update to all related standard controls.

To benefit from these AWS managed sources, we recommend that you create a new assessment from the updated framework. After you do this, you can then change the old assessment status to inactive. This action helps to ensure that your new assessment collects the most accurate and comprehensive evidence that’s available from AWS managed sources. If you take no action, your assessment continues to use the old framework and control definitions to collect evidence exactly as it did before.

What’s the difference between a data source and an evidence source?

An evidence source determines where evidence is collected from. This can be an individual data source, or a predefined grouping of data sources that maps to a core control or a common control.

A data source is the most granular type of evidence source. A data source includes the following details that tell Audit Manager where exactly to collect evidence data from:

• Data source type (for example, AWS Config)

• Data source mapping (for example, a specific AWS Config rule such as s3-bucket-public-write-prohibited)

My assessment creation failed

If your assessment creation fails, it could be because you selected too many AWS accounts in your assessment scope. If you're using AWS Organizations, Audit Manager can support up to 200 member accounts in the scope of a single assessment. If you exceed this number, the assessment creation might fail. As a workaround, you can run multiple assessments with different accounts in scope for each assessment.

What happens if I remove an in-scope account from my organization?

When an in-scope account is removed from your organization, Audit Manager no longer collects evidence for that account. However, the account continues to show in your assessment under the AWS accounts tab. To remove the account from the list of accounts in scope, edit the assessment. The removed account no longer shows in the list during editing, and you can save your changes without that account in scope.

I can't see the services in scope for my assessment

If you don't see the AWS services tab, this means that the services in scope are managed for you by Audit Manager. When you create a new assessment, Audit Manager manages the services in scope for you from that point onwards.

If you have an older assessment, it’s possible that you saw this tab previously in your assessment. However, Audit Manager automatically removes this tab from your assessment and takes over the management of services in scope when either of the following events occur:

• You edit your assessment

• You edit one of the custom controls that’s used in your assessment

Audit Manager infers the services in scope by examining your assessment controls and their data sources, and then mapping this information to the corresponding AWS services. If an underlying data source changes for your assessment, we automatically update the scope as needed to reflect the correct services. This ensures that your assessment collects accurate and comprehensive evidence about all of the relevant services in your AWS environment.

I can't edit the services in scope for my assessment

The Editing an assessment in AWS Audit Manager workflow no longer has an Edit services step. This is because Audit Manager now manages which AWS services are in scope for your assessment.

If you have an older assessment, it’s possible that you manually defined the services in scope when you created that assessment. However, you can’t edit these services moving forward. Audit Manager automatically takes over the management of services in scope for your assessment when either of the following events occur:

• You edit your assessment

• You edit one of the custom controls that’s used in your assessment

Audit Manager infers the services in scope by examining your assessment controls and their data sources, and then mapping this information to the corresponding AWS services. If an underlying data source changes for your assessment, we automatically update the scope as needed to reflect the correct services. This ensures that your assessment collects accurate and comprehensive evidence about all of the relevant services in your AWS environment.

What's the difference between a service in scope and a data source type?

A service in scope is an AWS service that's included in the scope of your assessment. When a service is in scope, Audit Manager collects evidence about your usage of that service and its resources.

Note

Audit Manager manages which AWS services are in scope for your assessments. If you have an older assessment, it’s possible that you manually specified the services in scope in the past. Moving forward, you can’t specify or edit services in scope.

A data source type indicates where exactly the evidence is collected from. If you upload your own evidence, the data source type is Manual. If Audit Manager collects the evidence, the data source can be one of four types.

1. AWS Security Hub – Captures a snapshot of your resource security posture by reporting findings from Security Hub.

2. AWS Config – Captures a snapshot of your resource security posture by reporting findings from AWS Config.

3. AWS CloudTrail – Tracks a specific user activity for a resource.

4. AWS API calls – Takes a snapshot of your resource configuration directly through an API call to a specific AWS service.

Here are two examples to illustrate the difference between a service in scope and a data source type.

Example 1

Let's say that you want to collect evidence for a control that's named 4.1.2 - Disallow public write access to S3 buckets. This control checks the access levels of your S3 bucket policies. For this control, Audit Manager uses a specific AWS Config rule (s3-bucket-public-write-prohibited) to look for an evaluation of your S3 buckets. In this example, the following is true:

• The service in scope is Amazon S3

• The resources that are being assessed are your S3 buckets

• The data source type is AWS Config

• The data source mapping is a specific AWS Config rule (s3-bucket-public-write-prohibited)

Example 2

Let's say that you want to collect evidence for a HIPAA control that's named 164.308(a)(5)(ii)(C). This control requires a monitoring procedure for detecting inappropriate sign-ins. For this control, Audit Manager uses CloudTrail logs to look for all AWS Management Console sign-in events. In this example, the following is true:

• The service in scope is IAM

• The resources that are being assessed are your users

• The data source type is CloudTrail

• The data source mapping is a specific CloudTrail event (ConsoleLogin)