Troubleshooting assessment and evidence collection issues - AWS Audit Manager

Troubleshooting assessment and evidence collection issues

You can use the information on this page to resolve common assessment and evidence collection issues in Audit Manager.

I created an assessment but I can’t see any evidence yet

If you can't see any evidence, it's likely that you either didn't wait at least 24 hours after you created the assessment or that there's a configuration error.

We recommend that you check the following:

  1. Make sure that more than 24 hours passed since you created the assessment. Automated evidence becomes available 24 hours after you create the assessment.

  2. Make sure that you’re using Audit Manager in the same AWS Region as the AWS service that you’re expecting to see evidence for.

  3. If you expect to see compliance check evidence from AWS Config and AWS Security Hub, make sure that both the AWS Config and Security Hub consoles are displaying results for these checks. The AWS Config and Security Hub results should be displaying in the same AWS Region that you use Audit Manager in.

If you still can't see any evidence in your assessment and it's not because of one of these issues, consider checking for the other issues that are described on this page.

My assessment isn’t collecting compliance check evidence from AWS Security Hub

Firstly, this issue can be caused if you missed some configuration steps in your AWS Security Hub settings.

If you're using a single AWS account, you must enable AWS Config and the PCI DSS security standard for your account.

If you're using Organizations, you must do the following:

  • Enable AWS Config and the PCI DSS security standard for every member account.

  • Designate the same administrator account in Security Hub and in Audit Manager.

Make sure that you configured your Security Hub settings as follows.

Before you enable any security standards in Security Hub, make sure that you enabled AWS Config and configured resource recording. For more information, see Enabling and configuring AWS Config in the AWS Security Hub User Guide. Then, follow this procedure to configure your Security Hub settings for Audit Manager.

To configure Security Hub settings for a single account

  1. Sign in to the AWS Management Console and open the AWS Security Hub console at https://console.aws.amazon.com/securityhub/.

  2. In the left navigation pane, choose Security standards.

  3. Under PCI DSS v3.2.1, choose Enable to enable the PCI DSS security standard for your account. By default, the AWS CIS Foundations Benchmark standard and the AWS Foundational Best Practices standard are already enabled. For more information, see Enabling a security standard in the AWS Security Hub User Guide.

Before you enable any security standards in Security Hub, make sure that you enabled AWS Config and configured resource recording for your organization. For more information, see Enabling and configuring AWS Config in the AWS Security Hub User Guide. Then, follow this procedure to configure your Security Hub settings for Audit Manager.

To configure Security Hub settings for an organization

  1. Sign in to the AWS Management Console and open the AWS Security Hub console at https://console.aws.amazon.com/securityhub/.

  2. Using your AWS Organizations management account, designate an account as the delegated administrator for Security Hub. Make sure that the delegated administrator account that you designate in Security Hub is the same one that you designated in Audit Manager. For more information, see Designating a Security Hub administrator account in the AWS Security Hub User Guide.

  3. Using your Organizations delegated administrator account, go to Settings, Accounts and enable your organization accounts as Security Hub member accounts. For more instructions, see Enabling member accounts from your organization in the AWS Security Hub User Guide.

  4. Enable the PCI DSS security standard for every member account of the organization. By default, the AWS CIS Foundations Benchmark standard and the AWS Foundational Best Practices standard are already enabled. For more information, see Enabling a security standard in the AWS Security Hub User Guide.

Secondly, keep in mind that Audit Manager doesn’t collect evidence from the service-linked AWS Config rules that Security Hub creates. This is a specific type of managed AWS Config rule that's enabled and controlled by the Security Hub service. Security Hub creates instances of these service-linked rules in your AWS environment, even if other instances of the same rules already exist. As a result, to prevent evidence duplication, Audit Manager doesn’t support evidence collection from the service-linked rules.

My assessment isn’t collecting compliance check evidence from an AWS Config rule

You can use managed AWS Config rules and custom AWS Config rules as a data source mapping for evidence collection. However, Audit Manager doesn’t collect evidence from most service-linked AWS Config rules.

There are only two types of service-linked rule that Audit Manager collects evidence from:

  • Service-linked rules from Conformance Packs

  • Service-linked rules from AWS Organizations

Audit Manager doesn't collect evidence from other service-linked rules, specifically any rules with an Amazon Resource Name (ARN) that contains the following prefix: arn:aws:config:*:*:config-rule/aws-service-rule/...

The reason that Audit Manager doesn't collect evidence from most service-linked AWS Config rules is to prevent duplicate evidence in your assessments. A service-linked rule is a specific type of managed rule that enables other AWS services to create AWS Config rules in your account. For example, some Security Hub controls use an AWS Config service-linked rule to run security checks. For each Security Hub control that uses a service-linked AWS Config rule, Security Hub creates an instance of the required AWS Config rule in your AWS environment. This happens even if the original rule already exists in your account.

Therefore, to avoid collecting the same evidence from the same rule twice, Audit Manager ignores the service-linked rule and doesn't collect evidence from it.

My assessment isn’t collecting evidence from another AWS service

If an AWS service isn't selected as in scope for your assessment, Audit Manager doesn't collect evidence from resources related to that service. This is also the case if an AWS service is selected but you haven't enabled it in your environment.

If you created your assessment from a custom framework, you can edit the services in scope for your assessment. You can then specify additional AWS services that you want to collect evidence from. After you add these services, evidence becomes available after 24 hours.

Note

If you created your assessment from a standard framework, the list of AWS services in scope is preselected and can’t be edited. This is because when you create an assessment from a standard framework, Audit Manager automatically maps and selects the relevant data sources and services for you. The selection is made based on the requirements of the standard framework. Note that, for standard frameworks that contain manual controls only, no AWS services are in scope.

The workaround for editing the AWS services in scope while still creating an assessment based on a standard framework is to customize the standard framework. By using this workaround, you can use the framework that you customized to create a new assessment. In this assessment, you can then specify which AWS services are in scope.

My evidence is generated at different intervals, and I don’t understand how often it’s being collected

The controls in Audit Manager assessments are mapped to a combination of data sources. Each data source has a different evidence collection frequency. As a result, there’s no one-size-fits-all answer for how often evidence is collected. Some data sources evaluate compliance, whereas others only capture the resource state and change data without a compliance determination.

The following is a summary of the different data source types and their evidence collection frequency.

Data source type Description Evidence collection frequency When this control is active in an assessment

AWS CloudTrail

Tracks a particular user activity that's needed in your audit.

Continual

Audit Manager filters your CloudTrail logs based on the keyword that you choose. The processed logs are imported as User activity evidence.

AWS Security Hub

Captures a snapshot of your resource security posture by reporting the result of a security check from Security Hub.

Based on the schedule of the Security Hub check (typically around every 12 hours)

Audit Manager retrieves the result of the security check directly from Security Hub. The result is imported as Compliance check evidence.

AWS Config

Captures a snapshot of your resource security posture by reporting findings from AWS Config.

Based on the settings that are defined in the AWS Config rule Audit Manager retrieves the findings for this rule directly from AWS Config. The result is imported as Compliance check evidence.
AWS API calls

Takes a snapshot of your resource configuration directly through an API call to the specified AWS service.

Daily, weekly, or monthly Audit Manager makes the API call based on the frequency that you specify. The response is imported as Configuration data evidence.

Regardless of the evidence collection frequency, new evidence is collected automatically for as long as the assessment is active. For more information, see Evidence collection frequency.

To learn more about control data sources, see Supported control data sources for automated evidence and Changing the evidence collection frequency for a control.

What happens if I remove an in-scope account from my organization?

When an in-scope account is removed from your organization, AWS Audit Manager no longer collects evidence for that account. However, the account continues to show in your assessment under the AWS accounts tab. To remove the account from the list of accounts in scope, you can edit the assessment. The removed account no longer shows in the list during editing, and you can save your changes without that account in scope.

I can't edit the services in scope for my assessment

When you use the Audit Manager console to create an assessment from a standard framework, the list of AWS services in scope is selected by default. This list can’t be edited. This is because Audit Manager automatically maps and selects the data sources and services for you. This selection is made according to the requirements of the standard framework. If the standard framework that you selected contains only manual controls, no AWS services are in scope for your assessment, and you can't add any services to your assessment.

If you need to edit the list of services in scope, you can do so by using the CreateAssessment API operation that's provided by Audit Manager. Alternatively, you can customize the standard framework and then create an assessment from the custom framework.

What's the difference between a service in scope and a data source type?

A service in scope is an AWS service that's specified as part of your assessment. When a service is in scope, Audit Manager assesses the resources of that service and collects the findings as evidence.

A data source type indicates where exactly the evidence comes from for a specific control in your assessment. If you upload your own evidence, the data source type is Manual. If Audit Manager collects the evidence, the data source can be one of four types.

  1. AWS Security Hub – Captures a snapshot of your resource security posture by reporting findings from Security Hub.

  2. AWS Config – Captures a snapshot of your resource security posture by reporting findings from AWS Config.

  3. AWS CloudTrail – Tracks a specific user activity for a resource.

  4. AWS API calls – Takes a snapshot of your resource configuration directly through an API call to a specific AWS service.

Here are two examples to illustrate the difference between a service in scope and a data source type.

Example 1

Let's say that you want to collect evidence for a control that's named 4.1.2 - Disallow public write access to S3 buckets. This control checks if the access levels of your S3 bucket policies are too lenient. For this control, Audit Manager uses a specific AWS Config rule (s3-bucket-public-write-prohibited) to look for an evaluation of your S3 buckets. In this example, the following is true:

Example 2

Let's say that you want to collect evidence for a HIPAA control that's named 164.308(a)(5)(ii)(C). This control requires a monitoring procedure for detecting inappropriate sign-ins. For this control, Audit Manager uses CloudTrail logs to look for all AWS Management Console sign-in events. In this example, the following is true:

My assessment creation failed

If your assessment creation fails, it could be because you selected too many services in the scope of your assessment. If you're using AWS Organizations, Audit Manager can support up to approximately 150 member accounts in the scope of a single assessment. If you exceed this number, the assessment creation might fail. As a workaround, you can run multiple assessments with different accounts in scope for each assessment.