Menu
AWS Greengrass
Developer Guide

Monitoring with AWS Greengrass Logs

AWS Greengrass consists of the cloud service and the AWS Greengrass core software. The core software can write logs to CloudWatch and to the local file system of your core device. Logging is configured at the group level.

All AWS Greengrass log entries include a time stamp, log level, and information about the event.

CloudWatch Logs

If you configure CloudWatch logging, you can view the logs on the Logs page of the Amazon CloudWatch console. Log groups for AWS Greengrass logs use the following naming conventions:

/aws/greengrass/GreengrassSystem/greengrass-system-component-name /aws/greengrass/Lambda/aws-region/account-id/lambda-function-name

Under each log group, you see log streams with the following structure:

date/account-id/greengrass-group-id/name-of-core-that-generated-log

Be aware of the following considerations when using CloudWatch Logs:

  • To enable logging to CloudWatch Logs, the following actions must be allowed in the AWS Greengrass group role:

    • logs:PutLogEvents

    • logs:CreateLogGroup

    • logs:CreateLogStream

    • logs:DescribeLogStreams

  • Logs are sent to CloudWatch Logs with a limited number of retries in case there's no internet connectivity. After the retries are exhausted, the event is dropped.

  • Transaction, memory, and other limitations apply. For more information, see Logging Limitations.

File System Logs

If you configure file system logging, the log files are stored under greengrass-root/ggc/var/log on the core device, with the following high-level directory structure:

greengrass-root/ggc/var/log - crash.log - system - log files for each Greengrass system component - user - log files generated by each user-defined Lambda function

Note

By default, greengrass-root is the /greengrass directory. If a write directory is configured, then the logs are under that directory.

Be aware of the following considerations when using file system logs:

  • Reading AWS Greengrass logs on the file system requires root privileges.

  • AWS Greengrass supports size-based rotation and automatic cleanup when the amount of log data is close to the configured limit.

  • The crash.log file is available in file system logs only. This log isn't written to CloudWatch Logs.

  • Disk usage limitations apply. For more information, see Logging Limitations.

Note

Logs for AWS Greengrass Core Software v1.0.0 are stored under the greengrass-root/var/log directory.

Default Logging Configuration

If logging settings aren't explicitly configured, AWS Greengrass uses the following default logging configuration after the first group deployment.

AWS Greengrass System Components
  • Type - FileSystem

  • Component - GreengrassSystem

  • Level - INFO

  • Space - 128 KB

User-defined Lambda Functions
  • Type - FileSystem

  • Component - Lambda

  • Level - INFO

  • Space - 128 KB

Note

Before the first deployment, only system components write logs to the file system because no user-defined Lambda functions are deployed.

Configure Logging for AWS Greengrass

You can use the AWS IoT console or the AWS Greengrass APIs to configure AWS Greengrass logging.

Note

To allow AWS Greengrass to write logs to CloudWatch Logs, your group role must allow the required CloudWatch Logs actions.

Configure Logging (Console)

You can configure logging on the group's Settings page.

  1. In the AWS IoT console, choose Greengrass, and then choose Groups.

  2. Choose the group where you want to configure logging.

  3. On the group configuration page, choose Settings.

  4. Choose the logging location, as follows:

    • To configure CloudWatch logging, for CloudWatch logs configuration, choose Edit.

    • To configure file system logging, for Local logs configuration, choose Edit.

    You can configure logging for one location or both locations.

  5. On the Configure Group logging page, choose Add another log type.

  6. Choose the event source, as follows:

    • To log events from user-defined Lambda functions, choose User Lambdas.

    • To log events from AWS Greengrass system components, choose Greengrass system.

    You can choose one component or both components.

  7. Choose Update.

  8. Choose the lowest level of events that you want to log. Events below this threshold are filtered out and aren't stored.

  9. For file system logs, specify a disk space limit.

  10. Choose Save.

Configure Logging (API)

You can use AWS Greengrass logger APIs to configure logging programmatically. For example, use the CreateLoggerDefinition action to create a logger definition based on a LoggerDefinitionVersion payload, which uses the following syntax:

{ "Loggers": [ { "Id": "string", "Type": "FileSystem|AWSCloudWatch", "Component": "GreengrassSystem|Lambda", "Level": "DEBUG|INFO|WARN|ERROR|FATAL", "Space": "integer" }, { "Id": "string", ... } ] }

LoggerDefinitionVersion is an array of one or more Logger objects that have the following properties:

Id

An identifier for the logger.

Type

The storage mechanism for log events. When AWSCloudWatch is used, log events are sent to CloudWatch Logs. When FileSystem is used, log events are stored on the local file system.

Valid values: AWSCloudWatch, FileSystem

Component

The source of the log event. When GreengrassSystem is used, events from Greengrass system components are logged. When Lambda is used, events from user-defined Lambda functions are logged.

Valid values: GreengrassSystem, Lambda

Level

The log-level threshold. Log events below this threshold are filtered out and aren't stored.

Valid values: DEBUG, INFO (recommended), WARN, ERROR, FATAL

Space

The maximum amount of local storage, in KB, to use for storing logs. This field applies only when Type is set to FileSystem.

Configuration Example

The following LoggerDefinitionVersion example specifies a logging configuration that:

  • Turns on file system ERROR (and above) logging for AWS Greengrass system components.

  • Turns on file system INFO (and above) logging for user-defined Lambda functions.

  • Turns on CloudWatch INFO (and above) logging for user-defined Lambda functions.

{ "Name": "LoggingExample", "InitialVersion": { "Loggers": [ { "Id": "1", "Component": "GreengrassSystem", "Level": "ERROR", "Space": 10240, "Type": "FileSystem" }, { "Id": "2", "Component": "Lambda", "Level": "INFO", "Space": 10240, "Type": "FileSystem" }, { "Id": "3", "Component": "Lambda", "Level": "INFO", "Type": "AWSCloudWatch" } ] } }

After you create a logger definition version, you can use its version ARN to create a group version before deploying the group.

Logging Limitations

AWS Greengrass has the following logging limitations.

Transactions per Second

When logging to CloudWatch is enabled, the logging component batches log events locally before sending them to CloudWatch, so you can log at a rate higher than five requests per second per log stream.

Memory

If AWS Greengrass is configured to send logs to CloudWatch and a Lambda function logs more than 5 MB/second for a prolonged period of time, the internal processing pipeline eventually fills up. The theoretical worst case is 6 MB per Lambda function.

Clock Skew

When logging to CloudWatch is enabled, the logging component signs requests to CloudWatch using the normal Signature Version 4 signing process. If the system time on the AWS Greengrass core device is out of sync by more than 15 minutes, then the requests are rejected.

Disk Usage

Use the following formula to calculate the total maximum amount of disk usage for logging.

greengrass-system-component-space * 8 // 7 if automatic IP detection is disabled + 128KB // the internal log for the local logging component + lambda-space * lambda-count // different versions of a Lambda function are treated as one

Where:

greengrass-system-component-space

The maximum amount of local storage for the AWS Greengrass system component logs.

lambda-space

The maximum amount of local storage for Lambda logs.

lambda-count

The number of deployed Lambda functions.

Log Loss

If your AWS Greengrass core device is configured to log only to CloudWatch and there's no internet connectivity, you have no way to retrieve the logs currently in the memory.

When Lambda functions are terminated (for example, during deployment), a few seconds' worth of logs are not written to CloudWatch.