Monitoring with AWS IoT Greengrass logs - AWS IoT Greengrass

You are viewing the documentation for AWS IoT Greengrass Version 1. AWS IoT Greengrass Version 2 is the latest major version of AWS IoT Greengrass. For more information about using AWS IoT Greengrass V2, see the AWS IoT Greengrass Version 2 Developer Guide.

Monitoring with AWS IoT Greengrass logs

AWS IoT Greengrass consists of the cloud service and the AWS IoT Greengrass Core software. The AWS IoT Greengrass Core software can write logs to Amazon CloudWatch and to the local file system of your core device. Lambda functions and connectors running on the core can also write logs to CloudWatch Logs and the local file system. You can use logs to monitor events and troubleshoot issues. All AWS IoT Greengrass log entries include a timestamp, log level, and information about the event. Changes to logging settings take effect after you deploy the group.

Logging is configured at the group level. For steps that show how to configure logging for a Greengrass group, see Configure logging for AWS IoT Greengrass.

Accessing CloudWatch Logs

If you configure CloudWatch logging, you can view the logs on the Logs page of the Amazon CloudWatch console. Log groups for AWS IoT Greengrass logs use the following naming conventions:

/aws/greengrass/GreengrassSystem/greengrass-system-component-name /aws/greengrass/Lambda/aws-region/account-id/lambda-function-name

Each log group contains log streams that use the following naming convention:

date/account-id/greengrass-group-id/name-of-core-that-generated-log

The following considerations apply when you use CloudWatch Logs:

  • Logs are sent to CloudWatch Logs with a limited number of retries in case there's no internet connectivity. After the retries are exhausted, the event is dropped.

  • Transaction, memory, and other limitations apply. For more information, see Logging limitations.

  • Your Greengrass group role must allow AWS IoT Greengrass to write to CloudWatch Logs. To grant permissions, embed the following inline policy in your group role.

    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "logs:DescribeLogStreams" ], "Resource": [ "arn:aws:logs:*:*:*" ] } ] }
    Note

    You can grant more granular access to your log resources. For more information, see Using identity-based policies (IAM policies) for CloudWatch Logs in the Amazon CloudWatch User Guide.

    The group role is an IAM role that you create and attach to your Greengrass group. You can use the console or the AWS IoT Greengrass API to manage the group role.

     

    Using the console
    1. In the AWS IoT console, in the navigation pane, choose Greengrass, Classic (V1), Groups.

    2. Choose the target group.

    3. Choose Settings. Under Group Role, you can view, attach, or remove the group role.

      For steps that show you how to attach the group role, see group role.

     

    Using the CLI

     

    To learn how to get the group ID to use with these commands, see Getting the group ID.

Accessing file system logs

If you configure file system logging, the log files are stored under greengrass-root/ggc/var/log on the core device. The following is the high-level directory structure:

greengrass-root/ggc/var/log - crash.log - system - log files for each Greengrass system component - user - region - account-id - log files generated by each user-defined Lambda function - aws - log files generated by each connector
Note

By default, greengrass-root is the /greengrass directory. If a write directory is configured, then the logs are under that directory.

The following considerations apply when you use file system logs:

  • Reading AWS IoT Greengrass logs on the file system requires root permissions.

  • AWS IoT Greengrass supports size-based rotation and automatic cleanup when the amount of log data is close to the configured limit.

  • The crash.log file is available in file system logs only. This log isn't written to CloudWatch Logs.

  • Disk usage limitations apply. For more information, see Logging limitations.

Note

Logs for AWS IoT Greengrass Core software v1.0 are stored under the greengrass-root/var/log directory.

Default logging configuration

If logging settings aren't explicitly configured, AWS IoT Greengrass uses the following default logging configuration after the first group deployment.

AWS IoT Greengrass System Components
  • Type - FileSystem

  • Component - GreengrassSystem

  • Level - INFO

  • Space - 128 KB

User-defined Lambda Functions
  • Type - FileSystem

  • Component - Lambda

  • Level - INFO

  • Space - 128 KB

Note

Before the first deployment, only system components write logs to the file system because no user-defined Lambda functions are deployed.

Configure logging for AWS IoT Greengrass

You can use the AWS IoT console or the AWS IoT Greengrass APIs to configure AWS IoT Greengrass logging.

Note

To allow AWS IoT Greengrass to write logs to CloudWatch Logs, your group role must allow the required CloudWatch Logs actions.

Configure logging (console)

You can configure logging on the group's Settings page.

  1. In the AWS IoT console, in the navigation pane, choose Greengrass, Classic (V1), Groups.

  2. Choose the group where you want to configure logging.

  3. On the group configuration page, choose Settings.

  4. Choose the logging location, as follows:

    • To configure CloudWatch logging, for CloudWatch logs configuration, choose Edit.

    • To configure file system logging, for Local logs configuration, choose Edit.

    You can configure logging for one location or both locations.

  5. On the Configure Group logging page, choose Add another log type.

  6. Choose the event source, as follows:

    • To log events from user-defined Lambda functions, choose User Lambdas.

    • To log events from AWS IoT Greengrass system components, choose Greengrass system.

    You can choose one component or both components.

  7. Choose Update.

  8. Choose the lowest level of events that you want to log. Events below this threshold are filtered out and aren't stored.

  9. For file system logs, specify a disk space limit.

  10. Choose Save. Changes take effect after you deploy the group.

Configure logging (API)

You can use AWS IoT Greengrass logger APIs to configure logging programmatically. For example, use the CreateLoggerDefinition action to create a logger definition based on a LoggerDefinitionVersion payload, which uses the following syntax:

{ "Loggers": [ { "Id": "string", "Type": "FileSystem|AWSCloudWatch", "Component": "GreengrassSystem|Lambda", "Level": "DEBUG|INFO|WARN|ERROR|FATAL", "Space": "integer" }, { "Id": "string", ... } ] }

LoggerDefinitionVersion is an array of one or more Logger objects that have the following properties:

Id

An identifier for the logger.

Type

The storage mechanism for log events. When AWSCloudWatch is used, log events are sent to CloudWatch Logs. When FileSystem is used, log events are stored on the local file system.

Valid values: AWSCloudWatch, FileSystem

Component

The source of the log event. When GreengrassSystem is used, events from Greengrass system components are logged. When Lambda is used, events from user-defined Lambda functions are logged.

Valid values: GreengrassSystem, Lambda

Level

The log-level threshold. Log events below this threshold are filtered out and aren't stored.

Valid values: DEBUG, INFO (recommended), WARN, ERROR, FATAL

Space

The maximum amount of local storage, in KB, to use for storing logs. This field applies only when Type is set to FileSystem.

Configuration example

The following LoggerDefinitionVersion example specifies a logging configuration that:

  • Turns on file system ERROR and above logging for AWS IoT Greengrass system components.

  • Turns on file system INFO (and above) logging for user-defined Lambda functions.

  • Turns on CloudWatch INFO (and above) logging for user-defined Lambda functions.

{ "Name": "LoggingExample", "InitialVersion": { "Loggers": [ { "Id": "1", "Component": "GreengrassSystem", "Level": "ERROR", "Space": 10240, "Type": "FileSystem" }, { "Id": "2", "Component": "Lambda", "Level": "INFO", "Space": 10240, "Type": "FileSystem" }, { "Id": "3", "Component": "Lambda", "Level": "INFO", "Type": "AWSCloudWatch" } ] } }

After you create a logger definition version, you can use its version ARN to create a group version before deploying the group.

Logging limitations

AWS IoT Greengrass has the following logging limitations.

Transactions per second

When logging to CloudWatch is enabled, the logging component batches log events locally before sending them to CloudWatch, so you can log at a rate higher than five requests per second per log stream.

Memory

If AWS IoT Greengrass is configured to send logs to CloudWatch and a Lambda function logs more than 5 MB/second for a prolonged period of time, the internal processing pipeline eventually fills up. The theoretical worst case is 6 MB per Lambda function.

Clock skew

When logging to CloudWatch is enabled, the logging component signs requests to CloudWatch using the normal Signature Version 4 signing process. If the system time on the AWS IoT Greengrass core device is out of sync by more than 15 minutes, then the requests are rejected.

Disk usage

Use the following formula to calculate the total maximum amount of disk usage for logging.

greengrass-system-component-space * 8 // 7 if automatic IP detection is disabled + 128KB // the internal log for the local logging component + lambda-space * lambda-count // different versions of a Lambda function are treated as one

Where:

greengrass-system-component-space

The maximum amount of local storage for the AWS IoT Greengrass system component logs.

lambda-space

The maximum amount of local storage for Lambda function logs.

lambda-count

The number of deployed Lambda functions.

Log loss

If your AWS IoT Greengrass core device is configured to log only to CloudWatch and there's no internet connectivity, you have no way to retrieve the logs currently in the memory.

When Lambda functions are terminated (for example, during deployment), a few seconds' worth of logs are not written to CloudWatch.

CloudTrail logs

AWS IoT Greengrass runs with AWS CloudTrail, a service that provides a record of actions taken by a user, role, or an AWS service in AWS IoT Greengrass. For more information, see Logging AWS IoT Greengrass API calls with AWS CloudTrail.