Menu
AWS Greengrass
Developer Guide

Monitoring with AWS Greengrass Logs

GGC v1.0.0GGC v1.1.0
GGC v1.0.0

AWS Greengrass consists of the cloud service and the AWS Greengrass core software. The AWS Greengrass core software can write logs to CloudWatch Logs or to the local file system of your AWS Greengrass core device. If the logging component is configured to log events to the local file system, the log files are stored under /greengrass/var/log with the following directory structure:

Copy
/greengrass/var - log - system - log files for each Greengrass system components - user - log files generated by each Lambda function

Currently, for file system logging, we support size-based rotation and automatic cleanup when the amount of log data is close to the configured limit.

GGC v1.1.0

AWS Greengrass consists of the cloud service and the AWS Greengrass core software. The AWS Greengrass core software can write logs to CloudWatch Logs or to the local file system of your AWS Greengrass core device. If the logging component is configured to log events to the local file system, the log files are stored under /greengrass/ggc/var/log with the following directory structure:

Copy
/greengrass/ggc/var - crash.log - system - log files for each Greengrass system components - user - log files generated by each Lambda function

Currently, for file system logging, we support size-based rotation and automatic cleanup when the amount of log data is close to the configured limit.

Configuring Greengrass Logging

The logging component can be configured through the Greengrass logging list APIs with a payload similar to the following:

Copy
{ "loggingModelList": [ { "Type": "log-storage-type", "Component": component-type, "Level": logging-level, "Space": max-storage }, { "Type": ... , "Level": ... , ... } ] }
Type

AWSCloudWatch and FileSystem. This field specifies which storage mechanism is used for log events. When AWSCloudWatch is used, log events are sent to CloudWatch with a limited number of retries in case there's no internet connectivity. After the retries are exhausted, the event is dropped. When FileSystem is used, log events are stored on the local file system.

Component

Supported values are GreengrassSystem and Lambda. When GreengrassSystem is used, log events from Greengrass system components are filtered based on the log level threshold and stored to the designated location. Depends on the Type. When Lambda is used, log events from a user's Lambda functions are filtered based on the log level and stored to the designated location. Again, depends on the Type.

Level

Log events below this threshold are filtered out and aren't stored.

Space

The maximum amount of local storage, in KB, to use for storing logs. This field applies only when Type is set to FileSystem.

If logging is not configured, before the first deployment, the AWS Greengrass core software does not contain any logging configuration information. It uses the following settings:

Copy
[ { "Type": "FileSystem", "Component": "GreengrassSystem", "Level": "INFO", "Space": 128 } ]

After the first deployment, if you explicitly configure the Greengrass software to emit no logs, none will be emitted. If you don't configure logging, the following settings are used by default:

Copy
[ { "Type": "FileSystem", "Component": "GreengrassSystem", "Level": "INFO", "Space": 128 }, { "Type": "FileSystem", "Component": "Lambda", "Level": "INFO", "Space": 128 } ]

Required IAM Permissions

To enable logging to CloudWatch, the following CloudWatch Logs actions must be present in the AWS Greengrass group role:

  • logs:PutLogEvents

  • logs:CreateLogGroup

  • logs:CreateLogStream

  • logs:DescribeLogStreams

Logging Output

If the logging component is configured to write logs to CloudWatch, the following log groups are displayed:

Copy
/aws/greengrass/GreengrassSystem/GreengrassSystemComponentName /aws/greengrass/Lambda/aws-region/accountId/lambda-function-name

Under each log group, you see log streams with the following structure:

Copy
date/accountId/greengrass-group-id/name-of-core-that-generated-log

The AWS Greengrass core logs look the same whether they are written to local storage or to CloudWatch Logs. The format for AWS Greengrass system logs is:

Copy
[<timestamp>][<Log level>]-<error message>

The format for Lambda logs is:

Copy
[<timestamp>][<Log level>]-<error message>

AWS Greengrass Logging Limitations

Transactions per Second

When logging to CloudWatch is enabled, the logging component batches log events locally before sending them to CloudWatch, so you can log at a rate higher than five requests per second per log stream.

Memory

If AWS Greengrass is configured to send logs to CloudWatch and a Lambda function logs more than 5 MB/second for a prolonged period of time, the internal processing pipeline eventually fills up. The theoretical worst case is 6 MB per Lambda function.

Clock Skew

When logging to CloudWatch is enabled, the logging component signs requests to CloudWatch using the normal Signature Version 4 process. If the system time on the AWS Greengrass core device is out of sync by more than 15 minutes, then the requests are rejected.

Disk Usage

Use the following formula to calculate the total maximum amount of disk usage for logging.

Copy
greengrass-storage * 7 // 6 if IP Detector is not used + 128KB // localwatch internal log + lambda-storage * lambda-count // different versions of a lambda are treated as one.

Where:

greengrass-storage

The maximum amount of local storage for AWS Greengrass logs.

lambda-storage

The maximum amount of local storage for Lambda logs.

lambda-count

The number of deployed Lambda functions.

Log Loss

If your AWS Greengrass core device is configured to log only to CloudWatch and there's no internet connectivity, you have no way to retrieve the logs currently in the memory.

When Lambda functions are terminated (for example, during deployment), a few seconds worth of logs are not written to CloudWatch.