Appendix B: Detailed Solution Workflow - AWS Trusted Advisor Explorer

Appendix B: Detailed Solution Workflow

The following section describes the workflow for extracting Trusted Advisor cost recommendations and resource tags.

Extract Trusted Advisor recommendations and resource tag data workflow

Figure 2: Extract Trusted Advisor recommendations and resource tag data workflow

The GetAccountsLambda AWS Lambda function is triggered on a schedule using an CloudWatch Events rule. The cron schedule is defined by the user while deploying the solution. This Lambda function retrieves all active accounts from AWS Organizations, batches them into groups of fifty, and then executes the MapOrganizations step function once per batch.

If resource tags are specified by the user as part of input parameters of the AWS CloudFormation template, the GetAccountsLambda Lambda function also executes the TagMapOrganizations Step Functions once per batch of 50 accounts.

Note

You can provide a file input instead of using AWS Organizations. For more information, see Appendix A.

The MapOrganizations and TagMapOrganizations Step Functions are executed once per batch of 50 accounts. Each account entry in the batch contains of the following parameters:

  • Account ID

  • Account Name

  • Account Email

  • Date (for example, 12-01-2019)

  • DateTime (for example, 2019-12-01 09:00:13)

MapOrganizations Step Functions

The MapOrganizations Step Functions are composed of the GetTAChecks Lambda function that runs the DescribeTrustedAdvisorChecks API and extracts the Cost Optimization Check Ids, Check Names, and Categories text fields. It appends these extracted fields to each of the entries in the input batch and then invokes the MapTACheck Step Functions once per Account.

The following is an example of the input batch passed on to the MapTACheck Step Functions:

  • Account ID

  • Account Name

  • Account Email

  • Date (for example, 12-01-2019)

  • DateTime (for example, 2019-12-01 09:00:13)

  • CheckID (for example, Qch7DwouX1)

  • CheckName (for example, Low Utilization Amazon EC2 Instances)

  • Category (for example, cost_optimizing)

  • Language (for example, en)

TagMapOrganizations Step Functions

The TagMapOrganizations step function is composed of one Lambda function, (GetTAChecks) that runs the DescribeRegions API and extracts all the AWS Regions. It appends the regions and resource types to the batch and invokes the ExtractTags Step Functions.

The following is an example of the input batch passed on to ExtractTags Step Functions:

  • Account ID

  • Account Name

  • Account Email

  • Date (for example, 12-01-2019)

  • DateTime (for example, 2019-12-01 09:00:13)

  • ResourceType (for example, Amazon RDS:db)

  • Region (for example, eu-north-1)

MapTACheck Step Functions

The MapTACheck Step Functions contains three AWS Lambda functions: RefreshTACheck, VerifyTACheckStatus, and ExtractTAData. The Step Functions starts off with first running RefreshTACheck AWS Lambda function that runs a RefreshTrustedAdvisorCheck API call to refresh the Trusted Advisor checks in all of the member accounts.

The VerifyTACheckStatus AWS Lambda function runs the DescribeTrustedAdvisorCheckRefreshStatuses API call and determines the wait duration for the check refresh to complete.

Note

The Step Functions only waits for 3600 secs. If the check takes more than 3600 seconds to refresh, the solution ignores the wait time and proceeds to extracting the recommendations data.

The ExtractTAData AWS Lambda function runs the DescribeTrustedAdvisorCheckResult API call for extracting the Trusted Advisor check data, write it into a csv file and pushes the csv file to an Amazon S3 bucket.

ExtractTags Step Function

The ExtractTags Step Functions contains one Lambda function. The TagExtractor Lambda function runs the ResourceGroupsTaggingAPI’s GetResources API and is responsible for extracting the associated resource tags for the given resource type in the input batch. The output is stored in a CSV file and is pushed to an Amazon S3 bucket.

The following section describes the workflow for creating the Trusted Advisor recommendations data lake.

Create Trusted Advisor Recommendations data lake workflow

Figure 3: Create Trusted Advisor recommendations data lake workflow

Note

The create data lake workflow must be triggered at least two hours after the extract Trusted Advisor recommendations and resource tag data workflow is triggered.

AWSTrustedAdvisorExplorer_Tags_Crawler is triggered on a schedule based on the cron defined by the user at the time of deploying the AWS CloudFormationtemplate. This crawler populates the AWS Glue Data Catalog with the Resource Tag table.

An event based CloudWatch Events rule triggers as result of successful completion of AWSTrustedAdvisorExplorer_Tags_Crawler. This CloudWatch Event rule invokes the StartGlueCrawlerLambda Lambda function, which triggers the AWSTrustedAdvisorExplorer_Crawler crawler.

AWSTrustedAdvisorExplorer_Crawler populates the AWS Glue Data Catalog with Trusted Advisor check data tables. Another CloudWatch Events rule triggers after AWSTrustedAdvisorExplorer_Crawler finishes. This CloudWatch Event rule invokes the CreateAthenaViewLambda Lambda function, which creates the required Athena views and posts an Amazon Simple Notification Service (SNS) notification to the AWSTrustedAdvisorExplorer-DataRefresh topic.

The user can now access the Athena console and run queries against the populated data. The user can also import the views into Amazon QuickSightto build Amazon QuickSight dashboards for visualization.