Data Collection Monitor

Introduction

The Data Collection Lab tooling provides instrumentation log data to help you monitor the executions of the various modules of the Data Collection Framework. Starting with version 3.11, most Data Collection modules emit basic log data to track module execution and potential errors encountered. This dashboard reads that instrumentation data to present multiple views to track historical executions as well as troubleshoot any issues.

As of this version, the monitoring is based on Step Function execution. Failures in individual Lambda functions are caught by the module’s Step Function and, in some circumstances, the Lambda error is available through the dashboard. Errors for any given execution will be listed on the dashboard with a link to the Step Function execution instance pertaining to it. You can investigate the error by clicking on the link. Errors are also reported to CloudWatch metrics under the namespace "CID-DataCollection", for which you can optionally setup an alarm for more active monitoring.

Future releases will convey more granular Lambda error details and links to those execution logs.

Preview

Prerequisites

Deploy or update the Data Collection Lab to version 3.11 or higher and include whatever modules you intend to use. Once deployed, any modules installed or updated will automatically be enabled for logging and picked up by the dashboard with the exception of the Feeds and Pricing modules. They will be enabled for instrumentation at a later time.

Deployment

CloudFormation

Note

Prerequisite: To install this dashboard using CloudFormation, you need to install Foundational Dashboards CFN with version v4.0.0 or above as described here

1. Log in to to your Data Collection Account. 1. Click the Launch Stack button below to open the pre-populated stack template in your CloudFormation.

   

2. You can change Stack name for your template if you wish.

3. Leave Parameters values as it is.

4. Review the configuration and click Create stack.

5. You will see the stack will start in CREATE_IN_PROGRESS. Once complete, the stack will show CREATE_COMPLETE

6. You can check the stack output for dashboard URLs.

   Note

   Troubleshooting: If you see error "No export named cid-CidExecArn found" during stack deployment, make sure you have completed prerequisite steps.

Command Line

Alternative method to install dashboards is the cid-cmd tool.

1. Log in to to your Data Collection Account.

2. Open up a command-line interface with permissions to run API requests in your AWS account. We recommend to use CloudShell.

3. In your command-line interface run the following command to download and install the CID CLI tool:

    pip3 install --upgrade cid-cmd

4. In your command-line interface run the following command to deploy the dashboard:

    cid-cmd deploy --dashboard-id dc-monitor

   Please follow the instructions from the deployment wizard. More info about command line options are in the Readme or cid-cmd --help.

Update

Please note that dashboards are not updated with update of CloudFormation Stack. When new version of the dashboard template is released, you can update your dashboard by running the following command in your command-line interface:

 cid-cmd update --dashboard-id dc-monitor

Authors

• Eric Christensen, Senior Technical Account Manager

Contributors

• Iakov Gan, Senior Solution Architect

• Yuriy Prykhodko, Principal Technical Account Manager

Feedback & Support

Follow Feedback & Support guide

Note

These dashboards and their content: (a) are for informational purposes only, (b) represent current AWS product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from AWS and its affiliates, suppliers or licensors. AWS content, products or services are provided "as is" without warranties, representations, or conditions of any kind, whether express or implied. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.