AWS IoT Greengrass Version 1 entered the extended life phase on June 30, 2023. For more information, see the AWS IoT Greengrass V1 maintenance policy. After this date, AWS IoT Greengrass V1 won't release updates that provide features, enhancements, bug fixes, or security patches. Devices that run on AWS IoT Greengrass V1 won't be disrupted and will continue to operate and to connect to the cloud. We strongly recommend that you migrate to AWS IoT Greengrass Version 2, which adds significant new features and support for additional platforms.

Troubleshooting AWS IoT Greengrass

This section provides troubleshooting information and possible solutions to help resolve issues with AWS IoT Greengrass.

For information about AWS IoT Greengrass quotas (limits), see Service Quotas in the Amazon Web Services General Reference.

AWS IoT Greengrass Core issues

If the AWS IoT Greengrass Core software does not start, try the following general troubleshooting steps:

Search the following symptoms and errors to find information to help troubleshoot issues with an AWS IoT Greengrass core.

 

Error: The configuration file is missing the CaPath, CertPath or KeyPath. The Greengrass daemon process with [pid = <pid>] died.

Solution: You might see this error in crash.log when the AWS IoT Greengrass Core software does not start. This can occur if you're running v1.6 or earlier. Do one of the following:

 

Error: Failed to parse /<greengrass-root>/config/config.json.

Solution: You might see this error when the AWS IoT Greengrass Core software does not start. Make sure the Greengrass configuration file is using valid JSON format.

Open config.json (located in /greengrass-root/config) and validate the JSON format. For example, make sure that commas are used correctly.

 

Error: Error occurred while generating TLS config: ErrUnknownURIScheme

Solution: You might see this error when the AWS IoT Greengrass Core software does not start. Make sure the properties in the crypto section of the Greengrass configuration file are valid. The error message should provide more information.

Open config.json (located in /greengrass-root/config) and check the crypto section. For example, certificate and key paths must use the correct URI format and point to the correct location.

 

Error: Runtime failed to start: unable to start workers: container test timed out.

Solution: You might see this error when the AWS IoT Greengrass Core software does not start. Set the postStartHealthCheckTimeout property in the Greengrass configuration file. This optional property configures the amount of time (in milliseconds) that the Greengrass daemon waits for the post-start health check to finish. The default value is 30 seconds (30000 ms).

Open config.json (located in /greengrass-root/config). In the runtime object, add the postStartHealthCheckTimeout property and set the value to a number greater than 30000. Add a comma where needed to create a valid JSON document. For example:

  ...
  "runtime" : {
    "cgroup" : {
      "useSystemd" : "yes"
    },
    "postStartHealthCheckTimeout" : 40000
  },
  ...

 

Error: Failed to invoke PutLogEvents on local Cloudwatch, logGroup: /GreengrassSystem/connection_manager, error: RequestError: send request failed caused by: Post http://<path>/cloudwatch/logs/: dial tcp <address>: getsockopt: connection refused, response: { }.

Solution: You might see this error when the AWS IoT Greengrass Core software does not start. This can occur if you're running AWS IoT Greengrass on a Raspberry Pi and the required memory setup has not been completed. For more information, see this step.

 

Error: Unable to create server due to: failed to load group: chmod /<greengrass-root>/ggc/deployment/lambda/arn:aws:lambda:<region>:<account-id>:function:<function-name>:<version>/<file-name>: no such file or directory.

Solution: You might see this error when the AWS IoT Greengrass Core software does not start. If you deployed a Lambda executable to the core, check the function's Handler property in the group.json file (located in /greengrass-root/ggc/deployment/group). If the handler is not the exact name of your compiled executable, replace the contents of the group.json file with an empty JSON object ({}), and run the following commands to start AWS IoT Greengrass:

cd /greengrass/ggc/core/
sudo ./greengrassd start

Then, use the AWS Lambda API to update the function configuration's handler parameter, publish a new function version, and update the alias. For more information, see AWS Lambda function versioning and aliases.

Assuming that you added the function to your Greengrass group by alias (recommended), you can now redeploy your group. (If not, you must point to the new function version or alias in your group definition and subscriptions before you deploy the group.)

 

The AWS IoT Greengrass Core software doesn't start after you changed from running with no containerization to running in a Greengrass container.

Solution: Check that you are not missing any container dependencies.

 

Error: Spool size should be at least 262144 bytes.

Solution: You might see this error when the AWS IoT Greengrass Core software does not start. Open the group.json file (located in /greengrass-root/ggc/deployment/group), replace the contents of the file with an empty JSON object ({}), and run the following commands to start AWS IoT Greengrass:

cd /greengrass/ggc/core/
sudo ./greengrassd start

Then follow the steps in the To cache messages in local storage procedure. For the GGCloudSpooler function, make sure to specify a GG_CONFIG_MAX_SIZE_BYTES value that's greater than or equal to 262144.

 

Error: [ERROR]-Cloud messaging error: Error occurred while trying to publish a message. {"errorString": "operation timed out"}

Solution: You might see this error in GGCloudSpooler.log when the Greengrass core is unable to send MQTT messages to AWS IoT Core. This can occur if the core environment has limited bandwidth and high latency. If you're running AWS IoT Greengrass v1.10.2 or later, try increasing the mqttOperationTimeout value in the config.json file. If the property is not present, add it to the coreThing object. For example:

{
  "coreThing": {
    "mqttOperationTimeout": 10,
    "caPath": "root-ca.pem",
    "certPath": "hash.cert.pem",
    "keyPath": "hash.private.key",
    ...
  },
  ...
}

The default value is 5 and the minimum value is 5.

 

Error: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:64: mounting \\\"/greengrass/ggc/socket/greengrass_ipc.sock\\\" to rootfs \\\"/greengrass/ggc/packages/<version>/rootfs/merged\\\" at \\\"/greengrass_ipc.sock\\\" caused \\\"stat /greengrass/ggc/socket/greengrass_ipc.sock: permission denied\\\"\"".

Solution: You might see this error in runtime.log when the AWS IoT Greengrass Core software does not start. This occurs if your umask is higher than 0022. To resolve this issue, you must set the umask to 0022 or lower. A value of 0022 grants everyone read permission to new files by default.

 

Error: Greengrass daemon running with PID: <process-id>. Some system components failed to start. Check 'runtime.log' for errors.

Solution: You might see this error when the AWS IoT Greengrass Core software does not start. Check runtime.log and crash.log for specific error information. For more information, see Troubleshooting with logs.

 

Device shadow does not sync with the cloud.

Solution: Make sure that AWS IoT Greengrass has permissions for iot:UpdateThingShadow and iot:GetThingShadow actions in the Greengrass service role. If the service role uses the AWSGreengrassResourceAccessRolePolicy managed policy, these permissions are included by default.

See Troubleshooting shadow synchronization timeout issues.

 

ERROR: unable to accept TCP connection. accept tcp [::]:8000: accept4: too many open files.

Solution: You might see this error in the greengrassd script output. This can occur if the file descriptor limit for the AWS IoT Greengrass Core software has reached the threshold and must be increased.

Use the following command and then restart the AWS IoT Greengrass Core software.

ulimit -n 2048

 

Error: Runtime execution error: unable to start lambda container. container_linux.go:259: starting container process caused "process_linux.go:345: container init caused \"rootfs_linux.go:50: preparing rootfs caused \\\"permission denied\\\"\"".

Solution: Either install AWS IoT Greengrass directly under the root directory, or make sure that the directory where the AWS IoT Greengrass Core software is installed and its parent directories have execute permissions for everyone.

 

Warning: [WARN]-[5]GK Remote: Error retrieving public key data: ErrPrincipalNotConfigured: private key for MqttCertificate is not set.

Solution: AWS IoT Greengrass uses a common handler to validate the properties of all security principals. This warning in runtime.log is expected unless you specified a custom private key for the local MQTT server. For more information, see AWS IoT Greengrass core security principals.

 

Error: Permission denied when attempting to use role arn:aws:iam::<account-id>:role/<role-name> to access s3 url https://<region>-greengrass-updates.s3.<region>.amazonaws.com/core/<architecture>/greengrass-core-<distribution-version>.tar.gz.

Solution: You might see this error when an over-the-air (OTA) update fails. In the signer role policy, add the target AWS Region as a Resource. This signer role is used to presign the S3 URL for the AWS IoT Greengrass software update. For more information, see S3 URL signer role.

 

The AWS IoT Greengrass core is configured to use a network proxy and your Lambda function can't make outgoing connections.

Solution: Depending on your runtime and the executables used by the Lambda function to create connections, you might also receive connection timeout errors. Make sure your Lambda functions use the appropriate proxy configuration to connect through the network proxy. AWS IoT Greengrass passes the proxy configuration to user-defined Lambda functions through the http_proxy, https_proxy, and no_proxy environment variables. They can be accessed as shown in the following Python snippet.

import os
print(os.environ['http_proxy'])

Use the same case as the variable defined in your environment, for example, all lower case http_proxy or all upper case HTTP_PROXY. For these variables, AWS IoT Greengrass supports both.

 

The core is in an infinite connect-disconnect loop. The runtime.log file contains a continuous series of connect and disconnect entries.

Solution: This can happen when another device is hard-coded to use the core thing name as the client ID for MQTT connections to AWS IoT. Simultaneous connections in the same AWS Region and AWS account must use unique client IDs. By default, the core uses the core thing name as the client ID for these connections.

To resolve this issue, you can change the client ID used by the other device for the connection (recommended) or override the default value for the core.

 

Error: unable to start lambda container. container_linux.go:259: starting container process caused "process_linux.go:345: container init caused \"rootfs_linux.go:62: mounting \\\"proc\\\" to rootfs \\\"

Solution: On some platforms, you might see this error in runtime.log when AWS IoT Greengrass tries to mount the /proc file system to create a Lambda container. Or, you might see similar errors, such as operation not permitted or EPERM. These errors can occur even if tests run on the platform by the dependency checker script pass.

Try one of the following possible solutions:

 

[ERROR]-runtime execution error: unable to start lambda container. {"errorString": "failed to initialize container mounts: failed to mask greengrass root in overlay upper dir: failed to create mask device at directory <ggc-path>: file exists"}

Solution: You might see this error in runtime.log when the deployment fails. This error occurs if a Lambda function in the AWS IoT Greengrass group cannot access the /usr directory in the core's file system.

To resolve this issue, add a local volume resource to the group and then deploy the group. This resource must:

 

[ERROR]-Deployment failed. {"deploymentId": "<deployment-id>", "errorString": "container test process with pid <pid> failed: container process state: exit status 1"}

Solution: You might see this error in runtime.log when the deployment fails. This error occurs if a Lambda function in the AWS IoT Greengrass group cannot access the /usr directory in the core's file system.

You can confirm that this is the case by checking GGCanary.log for additional errors. If the Lambda function cannot access the /usr directory, GGCanary.log will contain the following error:

[ERROR]-standard_init_linux.go:207: exec user process caused "no such file or directory"

To resolve this issue, add a local volume resource to the group and then deploy the group. This resource must:

 

Error: [ERROR]-runtime execution error: unable to start lambda container. {"errorString": "failed to initialize container mounts: failed to create overlay fs for container: mounting overlay at /greengrass/ggc/packages/<ggc-version>/rootfs/merged failed: failed to mount with args source=\"no_source\" dest=\"/greengrass/ggc/packages/<ggc-version>/rootfs/merged\" fstype=\"overlay\" flags=\"0\" data=\"lowerdir=/greengrass/ggc/packages/<ggc-version>/dns:/,upperdir=/greengrass/ggc/packages/<ggc-version>/rootfs/upper,workdir=/greengrass/ggc/packages/<ggc-version>/rootfs/work\": too many levels of symbolic links"}

Solution: You might see this error in the runtime.log file when the AWS IoT Greengrass Core software doesn't start. This issue might be more common on Debian operating systems.

To resolve this issue, do the following:

 

Error: [DEBUG]-Failed to get routes. Discarding message.

Solution: Check the subscriptions in your group and make sure that the subscription listed in the [DEBUG] message exists.

 

Error: [Errno 24] Too many open <lambda-function>,[Errno 24] Too many open files

Solution: You might see this error in your Lambda function log file if the function instantiates StreamManagerClient in the function handler. We recommend that you create the client outside the handler. For more information, see Use StreamManagerClient to work with streams.

 

Error: ds server failed to start listening to socket: listen unix <ggc-path>/ggc/socket/greengrass_ipc.sock: bind: invalid argument

Solution: You might see this error when the AWS IoT Greengrass Core software doesn't start. This error occurs when the AWS IoT Greengrass Core software is installed to a folder with a long file path. Reinstall the AWS IoT Greengrass Core software to a folder with a file path that has fewer than 79 bytes, if you don't use a write directory, or 83 bytes, if you do use a write directory.

[INFO] (Copier) aws.greengrass.StreamManager: stdout. Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instant exceeds minimum or maximum instant

When you upgrade AWS IoT Greengrass core software to v1.11.3, you might see the following error in the stream manager logs if stream manager fails to start.

2021-07-16T00:54:58.568Z [INFO] (Copier) aws.greengrass.StreamManager: stdout. Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instant exceeds minimum or maximum instant (through reference chain: com.amazonaws.iot.greengrass.streammanager.export.PersistedSuccessExportStatesV1["lastExportTime"]). {scriptName=services.aws.greengrass.StreamManager.lifecycle.startup.script, serviceName=aws.greengrass.StreamManager, currentState=STARTING}
2021-07-16T00:54:58.579Z [INFO] (Copier) aws.greengrass.StreamManager: stdout. Caused by: java.time.DateTimeException: Instant exceeds minimum or maximum instant. {scriptName=services.aws.greengrass.StreamManager.lifecycle.startup.script, serviceName=aws.greengrass.StreamManager, currentState=STARTING}

If you're using a version of AWS IoT Greengrass core software older than v1.11.3, and you want to upgrade to a later version, use an OTA update to upgrade to v1.11.4.

GPG error: https://dnw9lb6lzp2d8.cloudfront.net stable InRelease: The following signatures were invalid: EXPKEYSIG 68D644ABD2327D47 AWS Greengrass Master Key

When you run apt update on a device where you installed the AWS IoT Greengrass core software from an APT repository, you might see the following error.

Err:4 https://dnw9lb6lzp2d8.cloudfront.net stable InRelease
  The following signatures were invalid: EXPKEYSIG 68D644ABD2327D47 AWS Greengrass Master Key
Reading package lists... Done                             
W: GPG error: https://dnw9lb6lzp2d8.cloudfront.net stable InRelease: The following signatures were invalid: EXPKEYSIG 68D644ABD2327D47 AWS Greengrass Master Key

This error occurs because AWS IoT Greengrass no longer offers the option to install or update the AWS IoT Greengrass core software from the APT repository. To successfully run apt update, remove the AWS IoT Greengrass repository from the device's sources list.

sudo rm /etc/apt/sources.list.d/greengrass.list 
sudo apt update

Deployment issues

Use the following information to help troubleshoot deployment issues.

 

Your current deployment does not work and you want to revert to a previous working deployment.

Solution: Use the AWS IoT console or AWS IoT Greengrass API to redeploy a previous working deployment. This deploys the corresponding group version to your core device.

 

You see a 403 Forbidden error on deployment in the logs.

Solution: Make sure the policy of the AWS IoT Greengrass core in the cloud includes "greengrass:*" as an allowed action.

 

A ConcurrentDeployment error occurs when you run the create-deployment command for the first time.

Solution: A deployment might be in progress. You can run get-deployment-status to see if a deployment was created. If not, try creating the deployment again.

 

Error: Greengrass is not authorized to assume the Service Role associated with this account, or the error: Failed: TES service role is not associated with this account.

Solution: You might see this error when the deployment fails. Check that a Greengrass service role is associated with your AWS account in the current AWS Region. For more information, see Managing the Greengrass service role (CLI) or Managing the Greengrass service role (console).

 

Error: unable to execute download step in deployment. error while downloading: error while downloading the Group definition file: ... x509: certificate has expired or is not yet valid

Solution: You might see this error in runtime.log when the deployment fails. If you receive a Deployment failed error that contains the message x509: certificate has expired or is not yet valid, check the device clock. TLS and X.509 certificates provide a secure foundation for building IoT systems, but they require accurate times on servers and clients. IoT devices should have the correct time (within 15 minutes) before they attempt to connect to AWS IoT Greengrass or other TLS services that use server certificates. For more information, see Using Device Time to Validate AWS IoT Server Certificates on The Internet of Things on AWS Official Blog.

 

The deployment doesn't finish.

Solution: Do the following:

 

Error: Unable to find java or java8 executables, or the error: Deployment <deployment-id> of type NewDeployment for group <group-id> failed error: worker with <worker-id> failed to initialize with reason Installed Java version must be greater than or equal to 8

Solution: If stream manager is enabled for the AWS IoT Greengrass core, you must install the Java 8 runtime on the core device before you deploy the group. For more information, see the requirements for stream manager. Stream manager is enabled by default when you use the Default Group creation workflow in the AWS IoT console to create a group.

Or, disable stream manager and then deploy the group. For more information, see Configure stream manager settings (console).

 

The deployment doesn't finish, and runtime.log contains multiple "wait 1s for container to stop" entries.

Solution: Run the following commands in your core device terminal to restart the AWS IoT Greengrass daemon.

cd /greengrass/ggc/core/
sudo ./greengrassd stop
sudo ./greengrassd start

 

The deployment doesn't finish, and runtime.log contains "[ERROR]-Greengrass deployment error: failed to report deployment status back to cloud {"deploymentId": "<deployment-id>", "errorString": "Failed to initiate PUT, endpoint: https://<deployment-status>, error: Put https://<deployment-status>: proxyconnect tcp: x509: certificate signed by unknown authority"}"

Solution: You might see this error in runtime.log when the Greengrass core is configured to use an HTTPS proxy connection and the proxy server certificate chain isn't trusted on the system. To try to resolve this issue, add the certificate chain to the root CA certificate. The Greengrass core adds the certificates from this file to the certificate pool used for TLS authentication in HTTPS and MQTT connections with AWS IoT Greengrass.

The following example shows a proxy server CA certificate added to the root CA certificate file:

# My proxy CA
-----BEGIN CERTIFICATE-----
MIIEFTCCAv2gAwIQWgIVAMHSAzWG/5YVRYtRQOxXUTEpHuEmApzGCSqGSIb3DQEK
\nCwUAhuL9MQswCQwJVUzEPMAVUzEYMBYGA1UECgwP1hem9uLmNvbSBJbmMuMRww
... content of proxy CA certificate ...
+vHIRlt0e5JAm5\noTIZGoFbK82A0/nO7f/t5PSIDAim9V3Gc3pSXxCCAQoFYnui
GaPUlGk1gCE84a0X\n7Rp/lND/PuMZ/s8YjlkY2NmYmNjMCAXDTE5MTEyN2cM216
gJMIADggEPADf2/m45hzEXAMPLE=
-----END CERTIFICATE-----

# Amazon Root CA 1
-----BEGIN CERTIFICATE-----
MIIDQTCCAimgF6AwIBAgITBmyfz/5mjAo54vB4ikPmljZKyjANJmApzyMZFo6qBg
ADA5MQswCQYDVQQGEwJVUzEPMA0tMVT8QtPHRh8jrdkGA1UEChMGDV3QQDExBBKW
... content of root CA certificate ...
o/ufQJQWUCyziar1hem9uMRkwFwYVPSHCb2XV4cdFyQzR1KldZwgJcIQ6XUDgHaa
5MsI+yMRQ+hDaXJiobldXgjUka642M4UwtBV8oK2xJNDd2ZhwLnoQdeXeGADKkpy
rqXRfKoQnoZsG4q5WTP46EXAMPLE
-----END CERTIFICATE-----

By default, the root CA certificate file is located in /greengrass-root/certs/root.ca.pem. To find the location on your core device, check the crypto.caPath property in config.json.

 

Error: Deployment <deployment-id> of type NewDeployment for group <group-id> failed error: Error while processing. group config is invalid: 112 or [119 0] don't have rw permission on the file: <path>.

Solution: Make sure that the owner group of the <path> directory has read and write permissions to the directory.

 

Error: <list-of-function-arns> are configured to run as root but Greengrass is not configured to run Lambda functions with root permissions.

Solution: You might see this error in runtime.log when the deployment fails. Make sure that you have configured AWS IoT Greengrass to allow Lambda functions to run with root permissions. Either change the value of allowFunctionsToRunAsRoot in greengrass_root/config/config.json to yes or change the Lambda function to run as another user/group. For more information, see Running a Lambda function as root.

 

Error: Deployment <deployment-id> of type NewDeployment for group <group-id> failed error: Greengrass deployment error: unable to execute download step in deployment. error while processing: unable to load the group file downloaded: could not find UID based on user name, userName: ggc_user: user: unknown user ggc_user.

Solution: If the default access identity of the AWS IoT Greengrass group uses the standard system accounts, the ggc_user user and ggc_group group must be present on the device. For instructions that show how to add the user and group, see this step. Make sure to enter the names exactly as shown.

 

Error: [ERROR]-runtime execution error: unable to start lambda container. {"errorString": "failed to initialize container mounts: failed to mask greengrass root in overlay upper dir: failed to create mask device at directory <ggc-path>: file exists"}

Solution: You might see this error in runtime.log when the deployment fails. This error occurs if a Lambda function in the Greengrass group can't access the /usr directory in the core's file system. To resolve this issue, add a local volume resource to the group and then deploy the group. The resource must:

 

Error: Deployment <deployment-id> of type NewDeployment for group <group-id> failed error: process start failed: container_linux.go:259: starting container process caused "process_linux.go:250: running exec setns process for init caused \"wait: no child processes\"".

Solution: You might see this error when the deployment fails. Retry the deployment.

 

Error: [WARN]-MQTT[client] dial tcp: lookup <host-prefix>-ats.iot.<region>.amazonaws.com: no such host ... [ERROR]-Greengrass deployment error: failed to report deployment status back to cloud ... net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Solution: You might see this error if you're using systemd-resolved, which enables the DNSSEC setting by default. As a result, many public domains are not recognized. Attempts to reach the AWS IoT Greengrass endpoint fail to find the host, so your deployments remain in the In Progress state.

You can use the following commands and output to test for this issue. Replace the region placeholder in the endpoints with your AWS Region.

$ ping greengrass-ats.iot.region.amazonaws.com
ping: greengrass-ats.iot.region.amazonaws.com: Name or service not known

$ systemd-resolve greengrass-ats.iot.region.amazonaws.com
greengrass-ats.iot.region.amazonaws.com: resolve call failed: DNSSEC validation failed: failed-auxiliary

One possible solution is to disable DNSSEC. When DNSSEC is false, DNS lookups are not DNSSEC validated. For more information, see this known issue for systemd.

For information about resolved.conf and DNSSEC, run man resolved.conf in your terminal.

 

Create group and create function issues

Use the following information to help troubleshoot issues with creating an AWS IoT Greengrass group or Greengrass Lambda function.

 

Error: Your 'IsolationMode' configuration for the group is invalid.

Solution: This error occurs when the IsolationMode value in the DefaultConfig of function-definition-version is not supported. Supported values are GreengrassContainer and NoContainer.

 

Error: Your 'IsolationMode' configuration for function with arn <function-arn> is invalid.

Solution: This error occurs when the IsolationMode value in the <function-arn> of the function-definition-version is not supported. Supported values are GreengrassContainer and NoContainer.

 

Error: MemorySize configuration for function with arn <function-arn> is not allowed in IsolationMode=NoContainer.

Solution: This error occurs when you specify a MemorySize value and you choose to run without containerization. Lambda functions that are run without containerization cannot have memory limits. You can either remove the limit or you can change the Lambda function to run in an AWS IoT Greengrass container.

 

Error: Access Sysfs configuration for function with arn <function-arn> is not allowed in IsolationMode=NoContainer.

Solution: This error occurs when you specify true for AccessSysfs and you choose to run without containerization. Lambda functions run without containerization must have their code updated to access the file system directly and cannot use AccessSysfs. You can either specify a value of false for AccessSysfs or you can change the Lambda function to run in an AWS IoT Greengrass container.

 

Error: MemorySize configuration for function with arn <function-arn> is required in IsolationMode=GreengrassContainer.

Solution: This error occurs because you did not specify a MemorySize limit for a Lambda function that you are running in an AWS IoT Greengrass container. Specify a MemorySize value to resolve the error.

 

Error: Function <function-arn> refers to resource of type <resource-type> that is not allowed in IsolationMode=NoContainer.

Solution: You cannot access Local.Device, Local.Volume, ML_Model.SageMaker.Job, ML_Model.S3_Object, or S3_Object.Generic_Archive resource types when you run a Lambda function without containerization. If you need those resource types, you must run in an AWS IoT Greengrass container. You can also access local devices directly when running without containerization by changing the code in your Lambda function.

 

Error: Execution configuration for function with arn <function-arn> is not allowed.

Solution: This error occurs when you create a system Lambda function with GGIPDetector or GGCloudSpooler and you specified IsolationMode or RunAs configuration. You must omit the Execution parameters for this system Lambda function.

 

Discovery issues

Use the following information to help troubleshoot issues with the AWS IoT Greengrass Discovery service.

 

Error: Device is a member of too many groups, devices may not be in more than 10 groups

Solution: This is a known limitation. A client device can be a member of up to 10 groups.

 

Machine learning resource issues

Use the following information to help troubleshoot issues with machine learning resources.

 

InvalidMLModelOwner - GroupOwnerSetting is provided in ML model resource, but GroupOwner or GroupPermission is not present

Solution: You receive this error if a machine learning resource contains the ResourceDownloadOwnerSetting object but the required GroupOwner or GroupPermission property isn't defined. To resolve this issue, define the missing property.

 

NoContainer function cannot configure permission when attaching Machine Learning resources. <function-arn> refers to Machine Learnin resource <resource-id> with permission <ro/rw> in resource access policy.

Solution: You receive this error if a non-containerized Lambda function specifies function-level permissions to a machine learning resource. Non-containerized functions must inherit permissions from the resource owner permissions defined on the machine learning resource. To resolve this issue, choose to inherit resource owner permissions (console) or remove the permissions from the Lambda function's resource access policy (API).

 

Function <function-arn> refers to Machine Learning resource <resource-id> with missing permission in both ResourceAccessPolicy and resource OwnerSetting.

Solution: You receive this error if permissions to the machine learning resource aren't configured for the attached Lambda function or the resource. To resolve this issue, configure permissions in the ResourceAccessPolicy property for the Lambda function or the OwnerSetting property for the resource.

 

Function <function-arn> refers to Machine Learning resource <resource-id> with permission \"rw\", while resource owner setting GroupPermission only allows \"ro\".

Solution: You receive this error if the access permissions defined for the attached Lambda function exceed the resource owner permissions defined for the machine learning resource. To resolve this issue, set more restrictive permissions for the Lambda function or less restrictive permissions for the resource owner.

 

NoContainer Function <function-arn> refers to resources of nested destination path.

Solution: You receive this error if multiple machine learning resources attached to a non-containerized Lambda function use the same destination path or a nested destination path. To resolve this issue, specify separate destination paths for the resources.

 

Lambda <function-arn> gains access to resource <resource-id> by sharing the same group owner id

Solution: You receive this error in runtime.log if the same OS group is specified as the Lambda function's Run as identity and the resource owner for a machine learning resource, but the resource is not attached to the Lambda function. This configuration gives the Lambda function implicit permissions that it can use to access the resource without AWS IoT Greengrass authorization.

To resolve this issue, use a different OS group for one of the properties or attach the machine learning resource to the Lambda function.

AWS IoT Greengrass core in Docker issues

Use the following information to help troubleshoot issues with running an AWS IoT Greengrass core in a Docker container.

 

Error: Unknown options: -no-include-email.

Solution: This error can occur when you run the aws ecr get-login command. Make sure that you have the latest AWS CLI version installed (for example, run: pip install awscli --upgrade --user). If you're using Windows and you installed the CLI using the MSI installer, you must repeat the installation process. For more information, see Installing the AWS Command Line Interface on Microsoft Windows in the AWS Command Line Interface User Guide.

 

Warning: IPv4 is disabled. Networking will not work.

Solution: You might receive this warning or a similar message when running AWS IoT Greengrass on a Linux computer. Enable IPv4 network forwarding as described in this step. AWS IoT Greengrass cloud deployment and MQTT communications don't work when IPv4 forwarding isn't enabled. For more information, see Configure namespaced kernel parameters (sysctls) at runtime in the Docker documentation.

 

Error: A firewall is blocking file Sharing between windows and the containers.

Solution: You might receive this error or a Firewall Detected message when running Docker on a Windows computer. This can also occur if you are signed in on a virtual private network (VPN) and your network settings are preventing the shared drive from being mounted. In that situation, turn off VPN and re-run the Docker container.

 

Error: An error occurred (AccessDeniedException) when calling the GetAuthorizationToken operation: User: arn:aws:iam::<account-id>:user/<user-name> is not authorized to perform: ecr:GetAuthorizationToken on resource: *

You might receive this error when running the aws ecr get-login-password command if you don't have sufficient permissions to access an Amazon ECR repository. For more information, see Amazon ECR Repository Policy Examples and Accessing One Amazon ECR Repository in the Amazon ECR User Guide.

 

Error: Cannot create container for the service greengrass: Conflict. The container name "/aws-iot-greengrass" is already in use.

Solution: This can occur when the container name is used by an older container. To resolve this issue, run the following command to remove the old Docker container:

docker rm -f $(docker ps -a -q -f "name=aws-iot-greengrass")

 

Error: [FATAL]-Failed to reset thread's mount namespace due to an unexpected error: "operation not permitted". To maintain consistency, GGC will crash and need to be manually restarted.

Solution: This error in runtime.log can occur when you try to deploy a GreengrassContainer Lambda function to an AWS IoT Greengrass core running in a Docker container. Currently, only NoContainer Lambda functions can be deployed to a Greengrass Docker container.

To resolve this issue, make sure that all Lambda functions are in NoContainer mode and start a new deployment. Then, when starting the container, don't bind-mount the existing deployment directory onto the AWS IoT Greengrass core Docker container. Instead, create an empty deployment directory in its place and bind-mount that in the Docker container. This allows the new Docker container to receive the latest deployment with Lambda functions running in NoContainer mode.

For more information, see Running AWS IoT Greengrass in a Docker container.

Troubleshooting with logs

You can configure logging settings for a Greengrass group, such as whether to send logs to CloudWatch Logs, store logs on the local file system, or both. To get detailed information when troubleshooting issues, you can temporarily change the logging level to DEBUG. Changes to logging settings take effect when you deploy the group. For more information, see Configure logging for AWS IoT Greengrass.

On the local file system, AWS IoT Greengrass stores logs in the following locations. Reading the logs on the file system requires root permissions.

If the logs are configured to be stored on the cloud, use CloudWatch Logs to view log messages. crash.log is found only in file system logs on the AWS IoT Greengrass core device.

If AWS IoT is configured to write logs to CloudWatch, check those logs if connection errors occur when system components attempt to connect to AWS IoT.

For more information about AWS IoT Greengrass logging, see Monitoring with AWS IoT Greengrass logs.

Troubleshooting storage issues

When the local file storage is full, some components might start failing:

You should always be aware of the amount of free space available locally. You can calculate free space based on the sizes of deployed Lambda functions, the logging configuration (see Troubleshooting with logs), and the number of shadows stored locally.

Troubleshooting messages

All messages sent locally in AWS IoT Greengrass are sent with QoS 0. By default, AWS IoT Greengrass stores messages in an in-memory queue. Therefore, unprocessed messages are lost when the Greengrass core restarts; for example, after a group deployment or device reboot. However, you can configure AWS IoT Greengrass (v1.6 or later) to cache messages to the file system so they persist across core restarts. You can also configure the queue size. If you configure a queue size, make sure that it's greater than or equal to 262144 bytes (256 KB). Otherwise, AWS IoT Greengrass might not start properly. For more information, see MQTT message queue for cloud targets.

You can also configure the core to establish persistent sessions with AWS IoT. This allows the core to receive messages sent from the AWS Cloud while the core is offline. For more information, see MQTT persistent sessions with AWS IoT Core.

Troubleshooting shadow synchronization timeout issues

Significant delays in communication between a Greengrass core device and the cloud might cause shadow synchronization to fail because of a timeout. In this case, you should see log entries similar to the following:

[2017-07-20T10:01:58.006Z][ERROR]-cloud_shadow_client.go:57,Cloud shadow client error: unable to get cloud shadow what_the_thing_is_named for synchronization. Get https://1234567890abcd.iot.us-west-2.amazonaws.com:8443/things/what_the_thing_is_named/shadow: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[2017-07-20T10:01:58.006Z][WARN]-sync_manager.go:263,Failed to get cloud copy: Get https://1234567890abcd.iot.us-west-2.amazonaws.com:8443/things/what_the_thing_is_named/shadow: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[2017-07-20T10:01:58.006Z][ERROR]-sync_manager.go:375,Failed to execute sync operation {what_the_thing_is_named VersionDiscontinued []}"

A possible fix is to configure the amount of time that the core device waits for a host response. Open the config.json file in greengrass-root/config and add a system.shadowSyncTimeout field with a timeout value in seconds. For example:

{
  "system": {
    "shadowSyncTimeout": 10
  },
  "coreThing": {
    "caPath": "root-ca.pem",
    "certPath": "cloud.pem.crt",
    "keyPath": "cloud.pem.key",
    ...
  },
  ...
}

If no shadowSyncTimeout value is specified in config.json, the default is 5 seconds.

Check AWS re:Post

If you're unable to resolve your issue using the troubleshooting information in this topic, you can search the Troubleshooting AWS IoT Greengrass or check the AWS IoT Greengrass tag on AWS re:Post for related issues or post a new question. Members of the AWS IoT Greengrass team actively monitor AWS re:Post.