Troubleshooting AWS IoT Greengrass V2 - AWS IoT Greengrass

Troubleshooting AWS IoT Greengrass V2

Use the troubleshooting information and solutions in this section to help resolve issues with AWS IoT Greengrass Version 2.

View AWS IoT Greengrass Core software and component logs

The AWS IoT Greengrass Core software writes logs to the local file system that you can use to view real-time information about the core device. You can also configure core devices to write logs to CloudWatch Logs, so you can remotely troubleshoot core devices. These logs can help you identify issues with components, deployments, and core devices. For more information, see Monitor AWS IoT Greengrass logs.

AWS IoT Greengrass Core software issues

Troubleshoot AWS IoT Greengrass Core software issues.

Topics

Unable to set up core device

If the AWS IoT Greengrass Core software installer fails and you aren't able to set up a core device, you might need to uninstall the software and try again. For more information, see Uninstall the AWS IoT Greengrass Core software.

Unable to start the AWS IoT Greengrass Core software as a system service

If the AWS IoT Greengrass Core software fails to start, check the system service logs to identify the issue. One common issue is where Java isn't available on the PATH environment variable (Linux) or PATH system variable (Windows).

Unable to set up nucleus as a system service

You might see this error when the AWS IoT Greengrass Core software installer fails to set up AWS IoT Greengrass as a system service. On Linux devices, this error typically occurs if the core device doesn't have the systemd init system. The installer can successfully set up the AWS IoT Greengrass Core software even if it fails to set up the system service.

Do one of the following:

Unable to connect to AWS IoT Core

You might see this error when the AWS IoT Greengrass Core software can't connect to AWS IoT Core to retrieve deployment jobs, for example. Do the following:

  • Check that your core device can connect to the internet and AWS IoT Core. For more information about the AWS IoT Core endpoint to which your device connects, see Configure the AWS IoT Greengrass Core software.

  • Check that your core device's AWS IoT thing uses a certificate that allows the iot:Connect, iot:Publish, iot:Receive, and iot:Subscribe permissions.

  • If your core device uses a network proxy, check that your core device has a device role and that its role allows the iot:Connect, iot:Publish, iot:Receive, and iot:Subscribe permissions.

Out of memory error

This error typically occurs if your device doesn't have sufficient memory to allocate an object in the Java heap. On devices with limited memory, you might need to specify a maximum heap size to control memory allocation. For more information, see Control memory allocation with JVM options.

Unable to install Greengrass CLI

You might see the following console message when you use the --deploy-dev-tools argument in your installation command for AWS IoT Greengrass Core.

Thing group exists, it could have existing deployment and devices, hence NOT creating deployment for Greengrass first party dev tools, please manually create a deployment if you wish to

This occurs when the Greengrass CLI component is not installed because your core device is a member of a thing group that has an existing deployment. If you see this message, you can manually deploy the Greengrass CLI component (aws.greengrass.Cli) to the device to install the Greengrass CLI. For more information, see Install the Greengrass CLI.

User root is not allowed to execute

You might see this error when the user that runs the AWS IoT Greengrass Core software (typically root) doesn't have permission to run sudo with any user and any group. For the default ggc_user system user, this error looks like the following:

Sorry, user root is not allowed to execute <command> as ggc_user:ggc_group.

Check that your /etc/sudoers file gives the user permission to run sudo as other groups. The permission for the user in /etc/sudoers should look like the following example.

root ALL=(ALL:ALL) ALL

com.aws.greengrass.lifecyclemanager.GenericExternalService: Could not determine user/group to run with

You might see this error when the core device tries to run a component, and the Greengrass nucleus doesn't specify a default system user to use to run components.

To fix this issue, configure the Greengrass nucleus to specify the default system user that runs components. For more information, see Configure the user that runs components and Configure the default component user.

Failed to map segment from shared object: operation not permitted

You might see this error when the AWS IoT Greengrass Core software fails to start because the /tmp folder is mounted with noexec permissions. The AWS Common Runtime (CRT) library uses the /tmp folder by default.

Do one of the following:

  • Run the following command to remount the /tmp folder with exec permissions and try again.

    sudo mount -o remount,exec /tmp
  • If you run Greengrass nucleus v2.5.0 or later, you can set a JVM option to change the folder that the AWS CRT library uses. You can specify the jvmOptions parameter in the Greengrass nucleus component configuration in a deployment or when you install the AWS IoT Greengrass Core software. Replace /path/to/use with the path to a folder that the AWS CRT library can use.

    { "jvmOptions": "-Daws.crt.lib.dir=\"/path/to/use\"" }

Failed to set up Windows service

You might see this error if you install the AWS IoT Greengrass Core software on a Microsoft Windows 2016 device. The AWS IoT Greengrass Core software is not supported on Windows 2016, for a list of supported operating systems, see Supported platforms.

If you must use Windows 2016, you can do the following:

  1. Unzip the downloaded AWS IoT Greengrass Core installation archive

  2. In the Greengrass directory open the bin/greengrass.xml.template file.

  3. Add the <autoRefresh> tag to the end of the file just before the </service> tag.

    </log> <autoRefresh>false</autoRefresh> </service>

com.aws.greengrass.util.exceptions.TLSAuthException: Failed to get trust manager

You might see this error when you install the AWS IoT Greengrass Core software without a root certificate authority (CA) file.

2022-06-05T10:00:39.556Z [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: service-loaded. {serviceName=DeploymentService} 2022-06-05T10:00:39.943Z [WARN] (main) com.aws.greengrass.componentmanager.ClientConfigurationUtils: configure-greengrass-mutual-auth. Error during configure greengrass client mutual auth. {} com.aws.greengrass.util.exceptions.TLSAuthException: Failed to get trust manager

Check that you specify a valid root CA file with the rootCaPath parameter in the configuration file that you provide to the installer. For more information, see Install the AWS IoT Greengrass Core software.

com.aws.greengrass.deployment.IotJobsHelper: No connection available during subscribing to Iot Jobs descriptions topic. Will retry in sometime

You might see this warning message when the core device can't connect to AWS IoT Core to subscribe to deployment job notifications. Do the following:

  • Check that the core device is connected to the internet and can reach the AWS IoT data endpoint that you configured. For more information about endpoints that core devices use, see Allow device traffic through a proxy or firewall.

  • Check the Greengrass logs for other errors that reveal other root causes.

software.amazon.awssdk.services.iam.model.IamException: The security token included in the request is invalid

You might see this error when you install the AWS IoT Greengrass Core software with automatic provisioning, and the installer uses an AWS session token that isn't valid. Do the following:

  • If you use temporary security credentials, check that the session token is correct and that you are copying and pasting the complete session token.

  • If you use long-term security credentials, check that the device doesn't have a session token from a time where you previously used temporary credentials. Do the following:

    1. Run the following command to unset the session token environment variable.

      Linux or Unix
      unset AWS_SESSION_TOKEN
      Windows Command Prompt (CMD)
      set AWS_SESSION_TOKEN=
      PowerShell
      Remove-Item Env:\AWS_SESSION_TOKEN
    2. Check if the AWS credentials file, ~/.aws/credentials, contains a session token, aws_session_token. If so, remove that line from the file.

      aws_session_token = AQoEXAMPLEH4aoAH0gNCAPyJxz4BlCFFxWNE1OPTgk5TthT+FvwqnKwRcOIfrRh3c/LTo6UDdyJwOOvEVPvLXCrrrUtdnniCEXAMPLE/IvU1dYUg2RVAJBanLiHb4IgRmpRV3zrkuWJOgQs8IZZaIv2BXIa2R4Olgk

You can also install the AWS IoT Greengrass Core software without providing AWS credentials. For more information, see Install AWS IoT Greengrass Core software with manual resource provisioning or Install AWS IoT Greengrass Core software with AWS IoT fleet provisioning.

software.amazon.awssdk.services.iot.model.IotException: User: <user> is not authorized to perform: iot:GetPolicy

You might see this error when you install the AWS IoT Greengrass Core software with automatic provisioning, and the installer uses AWS credentials that don't have the required permissions. For more information about the permissions that are required, see Minimal IAM policy for installer to provision resources.

Check the permissions for the credentials' IAM identity, and grant the IAM identity any required permissions that are missing.

Error: com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest: Could not execute cloud shadow get request

You might see this error when you use the shadow manager component to sync device shadows with AWS IoT Core. The HTTP 403 status code indicates that this error occurred because the core device's AWS IoT policy doesn't grant permission to call GetThingShadow.

com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest: Could not execute cloud shadow get request. {thing name=MyGreengrassCore, shadow name=MyShadow} 2021-07-14T21:09:02.456Z [ERROR] (pool-2-thread-109) com.aws.greengrass.shadowmanager.sync.SyncHandler: sync. Skipping sync request. {thing name=MyGreengrassCore, shadow name=MyShadow} com.aws.greengrass.shadowmanager.exception.SkipSyncRequestException: software.amazon.awssdk.services.iotdataplane.model.IotDataPlaneException: null (Service: IotDataPlane, Status Code: 403, Request ID: f6e713ba-1b01-414c-7b78-5beb3f3ad8f6, Extended Request ID: null)

To sync local shadows with AWS IoT Core, the core device's AWS IoT policy must grant the following permissions:

  • iot:GetThingShadow

  • iot:UpdateThingShadow

  • iot:DeleteThingShadow

Check the core device's AWS IoT policy, and add any required permissions that are missing. For more information, see the following:

Operation aws.greengrass#<operation> is not supported by Greengrass

You might see this error when you use an interprocess communication (IPC) operation in a custom Greengrass component, and the required AWS-provided component isn't installed on the core device.

To fix this issue, add the required component as a dependency in your component recipe, so the AWS IoT Greengrass Core software installed the required component when you deploy your component.

java.io.FileNotFoundException: <stream-manager-store-root-dir>/stream_manager_metadata_store (Permission denied)

You might see this error in the stream manager log file (aws.greengrass.StreamManager.log) when you configure stream manager to use a root folder that doesn't exist or have the correct permissions. For more information about how to configure this folder, see stream manager configuration.

com.aws.greengrass.security.provider.pkcs11.PKCS11CryptoKeyService: Private key or certificate with label <label> does not exist

This error occurs when the PKCS#11 provider component can't find or load the private key or certificate that you specify when you configure the AWS IoT Greengrass Core software to use a hardware security module (HSM). Do the following:

  • Check that the private key and certificate are stored in the HSM using the slot, user PIN, and object label that you configure the AWS IoT Greengrass Core software to use.

  • Check that the private key and certificate use the same object label in the HSM.

  • If your HSM supports object IDs, check that the private key and certificate use the same object ID in the HSM.

Check the documentation for your HSM to learn how to query details about the security tokens in the HSM. If you need to change the slot, object label, or object ID for a security token, check the documentation for your HSM to learn how to do so.

software.amazon.awssdk.services.secretsmanager.model.SecretsManagerException: User: <user> is not authorized to perform: secretsmanager:GetSecretValue on resource: <arn>

This error can occur when you use the secret manager component to deploy an AWS Secrets Manager secret. If the core device's token exchange IAM role doesn't grant permission to get the secret, the deployment fails and the Greengrass logs include this error.

To authorize a core device to download a secret

  1. Add the secretsmanager:GetSecretValue permission to the core device's token exchange role. The following example policy statement grants permission to get the value of a secret.

    { "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue" ], "Resource": [ "arn:aws:secretsmanager:us-west-2:123456789012:secret:MyGreengrassSecret-abcdef" ] }

    For more information, see Authorize core devices to interact with AWS services.

  2. Reapply the deployment to the core device. Do one of the following:

    • Revise the deployment without any changes. The core device tries to download the secret again when it receives the revised deployment. For more information, see Revise deployments.

    • Restart the AWS IoT Greengrass Core software to retry the deployment. For more information, see Run the AWS IoT Greengrass Core software

    The deployment succeeds if secret manager downloads the secret successfully.

software.amazon.awssdk.services.secretsmanager.model.SecretsManagerException: Access to KMS is not allowed

This error can occur when you use the secret manager component to deploy an AWS Secrets Manager secret that is encrypted by an AWS Key Management Service key. If the core device's token exchange IAM role doesn't grant permission to decrypt the secret, the deployment fails and the Greengrass logs include this error.

To fix the issue, add the kms:Decrypt permission to the core device's token exchange role. For more information, see the following:

java.lang.NoClassDefFoundError: com/aws/greengrass/security/CryptoKeySpi

You might see this error when you try to install the AWS IoT Greengrass Core software with hardware security and you use an earlier Greengrass nucleus version that doesn't support hardware security integration. To use hardware security integration, you must use Greengrass nucleus v2.5.3 or later.

AWS IoT Greengrass cloud issues

Use the following information to troubleshoot issues with the AWS IoT Greengrass console and API. Each entry corresponds to an error message that you might see when you perform an action.

An error occurred (AccessDeniedException) when calling the CreateComponentVersion operation: User: arn:aws:iam::123456789012:user/<username> is not authorized to perform: null

You might see this error when you create a component version from the AWS IoT Greengrass console or with the CreateComponentVersion operation.

This error indicates that your recipe isn't valid JSON or YAML. Check the syntax of your recipe, fix any syntax issues, and try again. You can use an online JSON or YAML syntax checker to identify syntax issues in your recipe.

Invalid Input: Encountered following errors in Artifacts: {<s3ArtifactUri> = Specified artifact resource cannot be accessed}

You might see this error when you create a component version from the AWS IoT Greengrass console or with the CreateComponentVersion operation. This error indicates that an S3 artifact in the component recipe isn't valid.

Do the following:

  • Check that the S3 bucket is in the same AWS Region where you create the component. AWS IoT Greengrass doesn't support cross-Region requests for component artifacts.

  • Check that the artifact URI is a valid S3 object URL, and check the artifact exists at that S3 object URL.

  • Check that your AWS account has permission to access the artifact at its S3 object URL.

Core device deployment issues

Troubleshoot deployment issues on Greengrass core devices. Each entry corresponds to a log message that you might see on your core device.

Error: com.aws.greengrass.componentmanager.exceptions.PackageDownloadException: Failed to download artifact

You might see this error when the AWS IoT Greengrass Core software fails to download a component artifact when the core device applies a deployment. The deployment fails as a result of this error.

When you receive this error, the log also includes a stack trace that you can use to identify the specific issue. Each of the following entries corresponds to a message that you might see in the stack trace of the Failed to download artifact error message.

software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 403, Request ID: null, ...)

The PackageDownloadException error might include this stack trace in the following cases:

  • The component artifact isn't available at the S3 object URL that you specify in the component's recipe. Check that you uploaded the artifact to the S3 bucket and that the artifact URI matches the S3 object URL of the artifact in the bucket.

  • The core device's token exchange role doesn't allow the AWS IoT Greengrass Core software to download the component artifact from the S3 object URL that you specify in the component's recipe. Check that the token exchange role allows s3:GetObject for the S3 object URL where the artifact is available.

software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: <requestID>

The PackageDownloadException error might include this stack trace when the core device doesn't have permission to call s3:GetBucketLocation. The error message also includes the following message.

reason: Failed to determine S3 bucket location

Check that the core device's token exchange role allows s3:GetBucketLocation for the S3 bucket where the artifact is available.

Error: com.aws.greengrass.componentmanager.exceptions.ArtifactChecksumMismatchException: Integrity check for downloaded artifact failed. Probably due to file corruption.

You might see this error when the AWS IoT Greengrass Core software fails to download a component artifact when the core device applies a deployment. The deployment fails because the downloaded artifact file's checksum doesn't match the checksum that AWS IoT Greengrass calculated when you created the component.

Do the following:

  • Check if the artifact file changed in the S3 bucket where you host it. If the file changed since you created the component, restore it to the previous version that the core device expects. If you can't restore the file to its previous version, or if you want to use the new version of the file, create a new version of the component with the artifact file.

  • Check your core device's internet connection. This error can occur if the artifact file becomes corrupted while it downloads. Create a new deployment and try again.

Error: com.aws.greengrass.componentmanager.exceptions.NoAvailableComponentVersionException: Failed to negotiate component <name> version with cloud and no local applicable version satisfying requirement <requirements>

You might see this error when a core device can't find a component version that meets the requirements of the deployments for that core device. The core device checks for the component in the AWS IoT Greengrass service and on the local device. The error message includes each deployment's target and that deployment's version requirements for the component. The deployment target can be a thing, a thing group, or LOCAL_DEPLOYMENT, which represents the local deployment on the core device.

This issue can occur in the following cases:

  • The core device is the target of multiple deployments that have conflicting component version requirements. For example, the core device might be the target of multiple deployments that include a com.example.HelloWorld component, where one deployment requires version 1.0.0 and the other requires version 1.0.1. It's impossible to have a component that meets both requirements, so the deployment fails.

  • The component version doesn't exist in the AWS IoT Greengrass service or on the local device. The component might have been deleted, for example.

  • There exists component versions that meet the version requirements, but none are compatible with the core device's platform.

  • The core device's AWS IoT policy doesn't grant the greengrass:ResolveComponentCandidates permission. Look for Status Code: 403 in the error log to identify this issue. To resolve this issue, add the greengrass:ResolveComponentCandidates permission to the core device's AWS IoT policy. For more information, see Minimal AWS IoT policy for AWS IoT Greengrass V2 core devices.

To resolve this issue, revise the deployments to include compatible component versions or remove incompatible ones. For more information about how to revise cloud deployments, see Revise deployments. For more information about how to revise local deployments, see the AWS IoT Greengrass CLI deployment create command.

software.amazon.awssdk.services.greengrassv2data.model.ResourceNotFoundException: The latest version of Component <componentName> doesn't claim platform <coreDevicePlatform> compatibility

You might see this error when you deploy a component to a core device, and the component doesn't list a platform that is compatible with the core device's platform. Do one of the following:

  • If the component is a custom Greengrass component, you can update the component to be compatible with the core device. Add a new manifest that matches the core device's platform, or update an existing manifest to match the core device's platform. For more information, see AWS IoT Greengrass component recipe reference.

  • If the component is provided by AWS, check if another version of the component is compatible with the core device. If no version is compatible, contact us on AWS re:Post using the AWS IoT Greengrass tag, or contact AWS Support.

com.aws.greengrass.componentmanager.exceptions.PackagingException: The deployment attempts to update the nucleus from aws.greengrass.Nucleus-<version> to aws.greengrass.Nucleus-<version> but no component of type nucleus was included as target component

You might see this error when you deploy a component that depends on the Greengrass nucleus, and the core device runs an earlier Greengrass nucleus version than the latest minor version available. This error occurs because the AWS IoT Greengrass Core software tries to automatically update components to the latest compatible version. However, the AWS IoT Greengrass Core software prevents the Greengrass nucleus from updating to a new minor version, because several AWS-provided components depend on specific minor versions of the Greengrass nucleus. For more information, see Greengrass nucleus update behavior.

You must revise the deployment to specify the Greengrass nucleus version that you want to use. Do one of the following:

  • Revise the deployment to specify the Greengrass nucleus version that the core device currently runs.

  • Revise the deployment to specify a later minor version of the Greengrass nucleus. If you choose this option, you must also update the versions of all AWS-provided components that depend on specific minor versions of the Greengrass nucleus. For more information, see AWS-provided components.

Info: com.aws.greengrass.deployment.exceptions.RetryableDeploymentDocumentDownloadException: Greengrass Cloud Service returned an error when getting full deployment configuration

You might see this error when the core device receives a large deployment document, which is a deployment document larger than 7 KB (for deployments that target things) or 31 KB (for deployments that target thing groups). To retrieve a large deployment document, a core device's AWS IoT policy must allow the greengrass:GetDeploymentConfiguration permission. This error can occur when the core device doesn't have this permission. When this error occurs, the deployment retries indefinitely, and its status is In progress (IN_PROGRESS).

To resolve this issue, add the greengrass:GetDeploymentConfiguration permission to the core device's AWS IoT policy. For more information, see Update a core device's AWS IoT policy.

Warn: com.aws.greengrass.deployment.DeploymentService: Failed to get thing group hierarchy

You might see this warning when the core device receives a deployment and the core device's AWS IoT policy doesn't allow the greengrass:ListThingGroupsForCoreDevice permission. When you create a deployment, the core device uses this permission to identify its thing groups and remove components for any thing groups from which you removed the core device. If the core device runs Greengrass nucleus v2.5.0, the deployment fails. If the core device runs Greengrass nucleus v2.5.1 or later, the deployment proceeds but doesn't remove components. For more information about thing group removal behavior, see Deploy AWS IoT Greengrass components to devices.

To update the core device's behavior to remove components for thing groups from which you remove the core device, add the greengrass:ListThingGroupsForCoreDevice permission to the core device's AWS IoT policy. For more information, see Update a core device's AWS IoT policy.

Info: com.aws.greengrass.deployment.DeploymentDocumentDownloader: Calling Greengrass cloud to get full deployment configuration

You might see this information message printed multiple times without an error, because the core device logs the error at the DEBUG log level. This issue can occur when the core device receives a large deployment document. When this issue occurs, the deployment retries indefinitely, and its status is In progress (IN_PROGRESS). For more information about how to resolve this issue, see this troubleshooting entry.

Core device component issues

Troubleshoot Greengrass component issues on core devices.

Warn: '<command>' is not recognized as an internal or external command

You might see this error in a Greengrass component's logs when the AWS IoT Greengrass Core software fails to run a command in the component's lifecycle script. The component's state becomes BROKEN as a result of this error. This error can occur if the system user that runs the component, such as ggc_user, can't find the command's executable in the folders in the PATH.

On Windows devices, check that the folder that contains the executable is in the PATH for the system user that runs the component. If it's missing from the PATH, do one of the following:

  • Add the executable's folder to the PATH system variable, which is available to all users. Then, restart the component.

    If you run Greengrass nucleus 2.5.0, after you update the PATH system variable, you must restart the AWS IoT Greengrass Core software to run components with the updated PATH. If the AWS IoT Greengrass Core software doesn't use the updated PATH after you restart the software, restart the device and try again. For more information, see Run the AWS IoT Greengrass Core software.

  • Add the executable's folder to the PATH user variable for the system user that runs the component.

Python script doesn't log messages

Greengrass core devices collect logs that you can use to identify issues with components. If your Python script's stdout and stderr messages don't appear in your component logs, you might need to flush the buffer or disable buffering for these standard output streams in Python. Do any of the following:

  • Run Python with the -u argument to disable buffering on stdout and stderr.

    Linux or Unix
    python3 -u hello_world.py
    Windows
    py -3 -u hello_world.py
  • Use Setenv in your component's recipe to set the PYTHONUNBUFFERED environment variable to a non-empty string. This environment variable disables buffering on stdout and stderr.

  • Flush the buffer for the stdout or stderr streams. Do one of the following:

    • Flush a message when you print.

      import sys print('Hello, error!', file=sys.stderr, flush=True)
    • Flush a message after you print. You can send multiple messages before you flush the stream.

      import sys print('Hello, error!', file=sys.stderr) sys.stderr.flush()

For more information about how to verify that your Python script outputs log messages, see Monitor AWS IoT Greengrass logs.

Component configuration doesn't update when changing default configuration

When you change the DefaultConfiguration in a component's recipe, the new default configuration won't replace the component's existing configuration during a deployment. To apply the new default configuration, you must reset the component's configuration to its default settings. When you deploy the component, specify a single empty string as the reset update.

Console
Reset paths
[""]
AWS CLI

The following command creates a deployment to a core device.

aws greengrassv2 create-deployment --cli-input-json file://reset-configuration-deployment.json

The reset-configuration-deployment.json file contains the following JSON document.

{ "targetArn": "arn:aws:iot:us-west-2:123456789012:thing/MyGreengrassCore", "deploymentName": "Deployment for MyGreengrassCore", "components": { "com.example.HelloWorld": { "componentVersion": "1.0.0", "configurationUpdate": {, "reset": [""] } } } }
Greengrass CLI

The following Greengrass CLI command creates a local deployment on a core device.

sudo greengrass-cli deployment create \ --recipeDir recipes \ --artifactDir artifacts \ --merge "com.example.HelloWorld=1.0.0" \ --update-config reset-configuration-deployment.json

The reset-configuration-deployment.json file contains the following JSON document.

{ "com.example.HelloWorld": { "RESET": [""] } }

awsiot.greengrasscoreipc.model.UnauthorizedError

You might see this error in a Greengrass component's logs when the component doesn't have permission to perform an IPC operation on a resource. To grant a component permission to call an IPC operation, define an IPC authorization policy in the component's configuration. For more information, see Authorize components to perform IPC operations.

Tip

If you change the DefaultConfiguration in a component's recipe, you must reset the component's configuration to its new default configuration. When you deploy the component, specify a single empty string as the reset update. For more information, see Component configuration doesn't update when changing default configuration.

com.aws.greengrass.authorization.exceptions.AuthorizationException: Duplicate policy ID "<id>" for principal "<componentList>"

You might see this error if multiple IPC authorization policies, including across all components on the core device, use the same policy ID.

Check your components' IPC authorization policies, fix any duplicates, and try again. To create unique policy IDs, we recommend that you combine the component name, IPC service name, and a counter. For more information, see Authorize components to perform IPC operations.

Tip

If you change the DefaultConfiguration in a component's recipe, you must reset the component's configuration to its new default configuration. When you deploy the component, specify a single empty string as the reset update. For more information, see Component configuration doesn't update when changing default configuration.

com.aws.greengrass.tes.CredentialRequestHandler: Error in retrieving AwsCredentials from TES (HTTP 400)

You might see this error when a core device can't get AWS credentials from the token exchange service. The HTTP 400 status code indicates that this error occurred because the core device's token exchange IAM role doesn't exist or doesn't have a trust relationship that allows the AWS IoT credentials provider to assume it.

Do the following:

  1. Identify the token exchange role that the core device uses. The error message includes the core device's AWS IoT role alias, which points to the token exchange role. Run the following command on your development computer, and replace MyGreengrassCoreTokenExchangeRoleAlias with the name of the AWS IoT role alias from the error message.

    aws iot describe-role-alias --role-alias MyGreengrassCoreTokenExchangeRoleAlias

    The response includes the Amazon Resource Name (ARN) of the token exchange IAM role.

    { "roleAliasDescription": { "roleAlias": "MyGreengrassCoreTokenExchangeRoleAlias", "roleAliasArn": "arn:aws:iot:us-west-2:123456789012:rolealias/MyGreengrassCoreTokenExchangeRoleAlias", "roleArn": "arn:aws:iam::123456789012:role/MyGreengrassV2TokenExchangeRole", "owner": "123456789012", "credentialDurationSeconds": 3600, "creationDate": "2021-02-05T16:46:18.042000-08:00", "lastModifiedDate": "2021-02-05T16:46:18.042000-08:00" } }
  2. Check that the role exists. Run the following command, and replace MyGreengrassV2TokenExchangeRole with the name of the token exchange role.

    aws iam get-role --role-name MyGreengrassV2TokenExchangeRole

    If the command returns a NoSuchEntity error, the role doesn't exist, and you must create it. For more information about how to create and configure this role, see Authorize core devices to interact with AWS services.

  3. Check that the role has a trust relationship that allows the AWS IoT credentials provider to assume it. The response from the previous step contains an AssumeRolePolicyDocument, which defines the role's trust relationships. The role must define a trust relationship that allows credentials.iot.amazonaws.com to assume it. This document should look similar to the following example.

    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "credentials.iot.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }

    If the role's trust relationships don't allow credentials.iot.amazonaws.com to assume it, you must add this trust relationship to the role. For more information, see Modifying a role in the AWS Identity and Access Management User Guide.

com.aws.greengrass.tes.CredentialRequestHandler: Error in retrieving AwsCredentials from TES (HTTP 403)

You might see this error when a core device can't get AWS credentials from the token exchange service. The HTTP 403 status code indicates that this error occurred because the core device's AWS IoT policies don't grant the iot:AssumeRoleWithCertificate permission for the core device's AWS IoT role alias.

Review the core device's AWS IoT policies, and add the iot:AssumeRoleWithCertificate permission for the core device's AWS IoT role alias. The error message includes the core device's current AWS IoT role alias. For more information about this permission and how to update the core device's AWS IoT policies, see Minimal AWS IoT policy for AWS IoT Greengrass V2 core devices and Update a core device's AWS IoT policy.

Received error when attempting to retrieve ECS metadata: Could not connect to the endpoint URL: "<tokenExchangeServiceEndpoint>"

You might see this error when the component isn't running the token exchange service and a component tries to request AWS credentials.

Do the following:

  • Check that the component declares a dependency on the token exchange service component, aws.greengrass.TokenExchangeService. If it doesn't, add the dependency, and redeploy the component.

  • Check whether the component uses AWS credentials in its install lifecycle. AWS IoT Greengrass doesn't guarantee that the token exchange service is available during the install lifecycle. Update the component to move the code that uses AWS credentials into the startup or run lifecycle, and redeploy the component.

copyFrom: <configurationPath> is already a container, not a leaf

You might see this error when you change a configuration value from a container type (a list or object) to a non-container type (a string, number, or Boolean). Do the following:

  1. Check the component's recipe to see whether its default configuration sets that configuration value to a list or an object. If so, remove or change that configuration value.

  2. Create a deployment to reset that configuration value to its default value. For more information, see Create deployments and Update component configurations.

Then, you can set that configuration value to a string, number, or Boolean.

com.aws.greengrass.componentmanager.plugins.docker.exceptions.DockerLoginException: Error logging into the registry using credentials - 'The stub received bad data.'

You might see this error in the Greengrass nucleus logs when the Docker application manager component tries to download a Docker image from a private repository in Amazon Elastic Container Registry (Amazon ECR). This error occurs if you use the wincred Docker credential helper (docker-credential-wincred). As a result, Amazon ECR is unable to store the login credentials.

Take one of the following actions:

  • If you don't use the wincred Docker credential helper, remove the docker-credential-wincred program from the core device.

  • If you use the wincred Docker credential helper, do the following:

    1. Rename the docker-credential-wincred program on the core device. Replace wincred with a new name for the Windows Docker credential helper. For example, you can rename it to docker-credential-wincredreal.

    2. Update the credsStore option in the Docker configuration file (.docker/config.json) to use the new name for the Windows Docker credential helper. For example, if you renamed the program to docker-credential-wincredreal, update the credsStore option to wincredreal.

      { "credsStore": "wincredreal" }

java.io.IOException: Cannot run program "cmd" ...: [LogonUser] The password for this account has expired.

You might see this error on a Windows core device when the system user that runs the component's processes, such as ggc_user, has an expired password. As a result, the AWS IoT Greengrass Core software is unable to run component processes as that system user.

To update a Greengrass system user's password

  1. Run the following command as an administrator to set the user's password. Replace ggc_user with the system user, and replace password with the password to set.

    net user ggc_user password
  2. Use the PsExec utility to store the user's new password in the Credential Manager instance for the LocalSystem account. Replace password with the user's password that you set.

    psexec -s cmd /c cmdkey /generic:ggc_user /user:ggc_user /pass:password
Tip

Depending on your Windows configuration, the user's password might be set to expire at a date in the future. To ensure your Greengrass applications continue to operate, track when the password expires, and update it before it expires. You can also set the user's password to never expire.

  • To check when a user and its password expire, run the following command.

    net user ggc_user | findstr /C:expires
  • To set a user's password to never expire, run the following command.

    wmic UserAccount where "Name='ggc_user'" set PasswordExpires=False

aws.greengrass.StreamManager: Instant exceeds minimum or maximum instant

When you upgrade stream manager v2.0.7 to a version between v2.0.8 and v2.0.11, you might see the following error in the stream manager component's logs if the component fails to start.

2021-07-16T00:54:58.568Z [INFO] (Copier) aws.greengrass.StreamManager: stdout. Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instant exceeds minimum or maximum instant (through reference chain: com.amazonaws.iot.greengrass.streammanager.export.PersistedSuccessExportStatesV1["lastExportTime"]). {scriptName=services.aws.greengrass.StreamManager.lifecycle.startup.script, serviceName=aws.greengrass.StreamManager, currentState=STARTING} 2021-07-16T00:54:58.579Z [INFO] (Copier) aws.greengrass.StreamManager: stdout. Caused by: java.time.DateTimeException: Instant exceeds minimum or maximum instant. {scriptName=services.aws.greengrass.StreamManager.lifecycle.startup.script, serviceName=aws.greengrass.StreamManager, currentState=STARTING}

If you deployed stream manager v2.0.7 and you want to upgrade to a later version, you must upgrade to stream manager v2.0.12 directly. For more information about the stream manager component, see Stream manager.

Core device Lambda function component issues

Troubleshoot Lambda function component issues on core devices.

The following cgroup subsystems are not mounted: devices, memory

You might see this error when you run a containerized Lambda function in the following cases:

  • The core device doesn't have cgroup v1 enabled for the memory or device cgroups.

  • The core device has cgroups v2 enabled. Greengrass Lambda functions require cgroups v1, and cgroups v1 and v2 are mutually exclusive.

To enable cgroups v1, boot the device with the following Linux kernel parameters.

cgroup_enable=memory cgroup_memory=1 systemd.unified_cgroup_hierarchy=0
Tip

On a Raspberry Pi, edit the /boot/cmdline.txt file to set the device's kernel parameters.

ipc_client.py:64,HTTP Error 400:Bad Request, b'No subscription exists for the source <label-or-lambda-arn> and subject <label-or-lambda-arn>

You might see this error when you run a V1 Lambda function, which uses the AWS IoT Greengrass Core SDK, on a V2 core device without specifying a subscription in the legacy subscription router component. To fix this issue, deploy and configure the legacy subscription router to specify the required subscriptions. For more information, see Import V1 Lambda functions.

Greengrass Command Line Interface issues

Troubleshoot issues with the Greengrass CLI.

java.lang.RuntimeException: Unable to create ipc client

You might see this error when you run a Greengrass CLI command and you specify a different root folder than where the AWS IoT Greengrass Core software is installed.

Do one of the following to set the root path, and replace /greengrass/v2 with the path to your AWS IoT Greengrass Core software installation:

  • Set the GGC_ROOT_PATH environment variable to /greengrass/v2.

  • Add the --ggcRootPath /greengrass/v2 argument to your command as shown in the following example.

    greengrass-cli --ggcRootPath /greengrass/v2 <command> <subcommand> [arguments]

AWS Command Line Interface issues

Troubleshoot AWS CLI issues for AWS IoT Greengrass V2.

Error: Invalid choice: 'greengrassv2'

You might see this error when you run an AWS IoT Greengrass V2 command with the AWS CLI (for example, aws greengrassv2 list-core-devices).

This error indicates that you have a version of the AWS CLI that doesn't support AWS IoT Greengrass V2. To use AWS IoT Greengrass V2 with the AWS CLI, you must have one of the following versions or later:

  • Minimum AWS CLI V1 version: v1.18.197

  • Minimum AWS CLI V2 version: v2.1.11

Tip

You can run the following command to check the version of the AWS CLI that you have.

aws --version

To resolve this issue, update the AWS CLI to a later version that supports AWS IoT Greengrass V2. For more information, see Installing, updating, and uninstalling the AWS CLI in the AWS Command Line Interface User Guide.