Appendix F: Logging - Ops Automator

Appendix F: Logging

Ops Automator creates a log group (<ops-automator-stackname>-logs). The solution logs information for every step of a task that is run, which may result in many log streams. The log streams this solutions creates have structured names to help make it easier to find logs for a specific task or component of the framework.

The solution creates the following general log streams:

  • SetupHelperHandler-YYYYMMDD - Logs the output of the setup process when you deploy or update the Ops Automator stack. Check this stream if deploying or updating the stack fails.

  • OpsAutomatorMain-YYYMMMDD - Logs errors that are handled in specific modules and logged in their log streams. This stream is only created if errors occur.

  • ScheduleHandler-YYYYMMDD - Logs the output from the Lambda function that schedules configured tasks. This stream contains information that shows which tasks are run and the next time the task will run. Check this stream if a task does not run at the expected time.

  • CompletionHandler-YYYYMMDD - Logs the output from the Lambda function that runs the completion logic. The output of the completion logic is logged in the stream for the specific task.

  • TaskConfigAdminAPI-YYYYMMDD - Logs the output from configuration steps performed through the Configuration Admin API.

  • ConfigurationResourcesHandler-YYYYMMDD - Logs the output from the custom resource handler that creates tasks. Check this log when creating a task stack fails because it includes all parameter validation errors.

  • TaskTrackingHandler-YYYYMMDD - Logs detailed information from core task processing. This steam is created if you enable detailed debugging.

  • Ec2StateEventHandler-<account>-<region>-YYYYMMDD - Logs information about processing Amazon EC2 state events for a specific account and/or Region.

  • Ec2TagEventHandler-<account>-<region>-YYYYMMDD - Logs information about processing Amazon EC2 tag modification events for a specific account and/or Region.

  • EbsSnapshotEventHandler-<account>-<region>-YYYYMMDD - Logs information about processing Amazon EBS snapshot events for a specific account and/or Region.

The solution also creates the following task-specific log streams:

  • SelectResourcesHandler-<taskname>-<account>-<region>-YYYYMMDD

  • SelectResourcesHandler-<taskname>-<account>-YYYYMMDD

  • SelectResourcesHandler-<taskname>-YYYYMMDD

These log streams log the output from the AWS Lambda function that selects the resources for a task. Check these logs if the expected resources are not selected for a task. When detailed logging is enabled, these streams contain detailed information about found resources and the filtering used to include or exclude the resources from the action execution. These streams are named based on the aggregation level of the resources.

  • <taskname>-<account>-<region>-<resource>-<task-id>-YYYYMMDD

  • <taskname>-<account>-<region>-<task-id>-YYYYMMDD

  • <taskname>-<account>-<task-id>-YYYYMMDD

  • <taskname>-<task-id>-YYYYMMDD

These log streams log the output from the execution and, if enabled, completion logic of the action. Check these logs if an action fails to run or complete. These streams are named based on the aggregation level of the resources. For example, if the task is for a specific resource, the log stream name will include the name of the specific resource. If the task is for a specific Region, the stream name will include the Region name.

The solution also creates the following Lambda service log groups:

  • /aws/lambda/<stackname>-Standard

  • /aws/lambda/<stackname>-Medium

  • /aws/lambda/<stackname>-Large

  • /aws/lambda/<stackname>-XLarge

  • /aws/lambda/<stackname>-XXLarge

  • /aws/lambda/<stackname>-XXXLarge

These log groups contain the default log streams for the Lambda functions. These streams do not contain any Ops Automator-specific information.

Messages

This solution also logs error, warning, and debugging messages. Each message has the format: yyyy-mm-dd – hh:mm:ss.mmm - <category> - <text>. The category can have the value INFO for informational messages, DEBUG for detailed debugging messages, WARNING for warning messages, or ERROR for error messages.

Issues SNS Topic

Ops Automator creates an Amazon Simple Notification Service (Amazon SNS) topic where all warnings and errors are posted. Messages include the error, and the name of the log group and the log stream where the error is logged. The name of the Amazon SNS topic is located in the Ops Automator stack output named IssueSNSTopic.

The messages posted to the topic contain the following attributes:

Attribute Description
log-group The name of the log group of the stream where the message was logged
log-stream The name of the log stream where the message was logged
category ERROR or WARNING
message The error or warning message

Notifications SNS Topic

If enabled, Ops Automator creates an Amazon SNS topic where notifications for every started or completed task are posted. Use the Task Notifications parameter to enable this feature. The name of the Amazon SNS topic is located in the Ops Automator stack output named NotificationsSNSTopic.

The messages posted to the topic contain the following attributes:

Attribute Description
Id The task ID.
TaskName The name of the task.
Account The account in which the action is run.
Resources The task resources.
Parameters The parameters of the task.
Time The date and time in ISO format.
Type Can be either task-started or task-ended.
Status The status of the implemented task: completed, timed-out, or failed. Included when the Type is task-ended.
ActionResult The result output of a completed task. Included when the Status is completed.
Error The error reported by a failed task. Included when the Status is failed.

Out-of-Band Logging

Ops Automator uses Amazon CloudWatch logs to log all output. To help deal with high volume and concurrency, the solution includes a layer on top of CloudWatch logging that enables the solution to log out-of-band information. Logged messages are buffered and written to an Amazon Simple Queue Service (Amazon SQS) queue. A Lambda function (<stackname>-OpsAutomatorCloudWatchQueueHandler) processes the queued messages and writes them to the log streams. Because of the buffering and queuing, there might be a delay between when messages are generated and when they are written to the log stream.