IDT for AWS IoT Greengrass troubleshooting - AWS IoT Greengrass
Error codesResolving IDT for AWS IoT Greengrass errors

AWS IoT Greengrass Version 1 entered the extended life phase on June 30, 2023. For more information, see the AWS IoT Greengrass V1 maintenance policy. After this date, AWS IoT Greengrass V1 won't release updates that provide features, enhancements, bug fixes, or security patches. Devices that run on AWS IoT Greengrass V1 won't be disrupted and will continue to operate and to connect to the cloud. We strongly recommend that you migrate to AWS IoT Greengrass Version 2, which adds significant new features and support for additional platforms.

IDT for AWS IoT Greengrass troubleshooting

IDT for AWS IoT Greengrass writes these errors to various locations based on the type of errors. Errors are written to the console, log files, and test reports.

Error codes

The following table lists the error codes generated by IDT for AWS IoT Greengrass.

Error code Error code name Possible root cause Troubleshooting

101

InternalError

An internal error occurred.

Check logs under the <device-tester-extract-location>/results directory. If you cannot debug the issue, contact AWS Developer Support.

102

TimeoutError

The test cannot be completed in a limited time range. This can happen if:

  • There is a slow network connection between the test machine and device (for example, if you are using a VPN network).

  • A slow network delays the communication between the device and cloud.

  • The timeout field in test configuration files (test.json) has been mistakenly modified.

  • Check the network connection and speed.

  • Make sure that you did not modify any file under the /test directory.

  • Try running the failed test group manually with "--group-id" flag.

  • Try running the test suite by increasing the test timeouts. For more information, see Timeout errors.

103

PlatformNotSupportError

Incorrect OS/architecture combination specified in device.json.

Change your configuration to one of the supported combinations:

  • Linux, x86_64

  • Linux, ARMv6l

  • Linux, ARMv7l

  • Linux, AArch64

  • Ubuntu, x86_64

  • OpenWRT, ARMv7l

  • OpenWRT, AArch64

For more information, see Configure device.json.

104

VersionNotSupportError

The AWS IoT Greengrass Core software version is not supported by the version of IDT you are using.

Use the device_tester_bin version command to find the supported version of the AWS IoT Greengrass Core software. For example, if you are using macOS, use ./devicetester_mac_x86_64 version.

To find the version of AWS IoT Greengrass Core software that you are using:

  • If you are running tests with preinstalled AWS IoT Greengrass Core software, use SSH to connect to your AWS IoT Greengrass core device and run <path-to-preinstalled-greengrass-location>/greengrass/ggc/core/greengrassd --version

  • If you are running tests with a different version of the AWS IoT Greengrass Core software, go to the devicetester_greengrass_ <os>/products/greengrass/gcc directory. The AWS IoT Greengrass Core software version is part of the .zip file name.

You can test a different version of the AWS IoT Greengrass Core software. For more information, see Getting started with AWS IoT Greengrass.

105

LanguageNotSupportError

IDT supports Python for AWS IoT Greengrass libraries and SDKs only.

Make sure:

  • The SDK package under devicetester_greengrass_<os>/products/greengrass/ggsdk is the Python SDK.

  • The contents of the bin folder under devicetester_greengrass_<os> /tests/GGQ_1.0.0/suite/resources/run.runtimefarm/bin have not been changed.

106

ValidationError

Some fields in device.json or config.json are invalid.

Check the error message on the right side of the error code in the report.

  • Invalid auth type for device: Specify the correct method to connect to your device. For more information, see Configure device.json.

  • Invalid private key path: Specify the correct path to your private key. For more information, see Configure device.json.

  • Invalid AWS Region: Specify a valid AWS Region in your config.json file. For more information, see AWS service endpoints.

  • AWS credentials: Set valid AWS credentials on your test machine (by using environment variables or the credentials file). Verify that the auth field is configured correctly. For more information, see Create and configure an AWS account.

  • Invalid HSM input: Check your p11Provider, privateKeyLabel, slotLabel, slotUserPin, and openSSLEngine fields in device.json.

107

SSHConnectionFailed

The test machine cannot connect to the configured device.

Verify that the following fields in your device.json file are correct:

  • ip

  • user

  • privKeyPath

  • password

For more information, see Configure device.json.

108

RunCommandError

A test failed to execute a command on the device under test.

Verify that root access is allowed for the configured user in device.json.

A password is required by some devices when executing commands with root access. Make sure root access is allowed without a password. For more information, see the documentation for your device.

Try running the failing command manually on your device to see if an error occurs.

109

PermissionDeniedError

No root access.

Set root access for the configured user on your device.

110

CreateFileError

Unable to create a file.

Check your device's disk space and directory permissions.

111

CreateDirError

Unable to create a directory.

Check your device's disk space and directory permissions.

112

InvalidPathError

The path to the AWS IoT Greengrass Core software is incorrect.

Verify that the path in the error message is valid. Do not edit any files under the devicetester_greengrass_<os> directory.

113

InvalidFileError

A file is invalid.

Verify that the file in the error message is valid.

114

ReadFileError

The specified file cannot be read.

Verify the following:

  • File permissions are correct.

  • limits.config allows enough files to be open.

  • The file specified in the error message exists and is valid.

If you are testing on macOS, increase the open files limit. The default limit is 256, which is enough for testing.

115

FileNotFoundError

A required file was not found.

Verify the following:

  • A compressed Greengrass file exists under devicetester_greengrass_<os>/products/greengrass/ggc. You can download the AWS IoT Greengrass Core tar file from the AWS IoT Greengrass Core Software downloads page.

  • The SDK package exists under devicetester_greengrass_<os>/products/greengrass/ggsdk.

  • The files under devicetester_greengrass_<os>/tests have not been modified.

116

OpenFileFailed

Unable to open the specified file.

Verify the following:

  • The file specified in the error message exists and is valid.

  • limits.config allows enough files to be open.

If you are testing on macOS, increase the open files limit. The default limit is 256, which is enough for testing.

117

WriteFileFailed

Failed to write file (can be the DUT or test machine).

Verify that the directory specified in the error message exists and that you have write permission.

118

FileCleanUpError

A test failed to remove the specified file or directory or to umount the specified file on the remote device.

If the binary file is still running, the file might be locked. End the process and delete the specified file.

119

InvalidInputError

Invalid configuration.

Verify that your suite.json file is valid.

120

InvalidCredentialError

Invalid AWS credentials.

  • Verify your AWS credentials. For more information, see Configure your AWS credentials.

  • Check your network connection and rerun the test group. Network problems can also casue this error.

121

AWSSessionError

Failed to create an AWS session.

This error can occur if AWS credentials are invalid or the internet connection is unstable. Try using the AWS CLI to call an AWS API operation.

122

AWSApiCallError

An AWS API error occurred.

This error might be due to a network issue. Check your network before retrying the test group.

123

IpNotExistError

IP address is not included in connectivity information.

Check your internet connection. You can use the AWS IoT Greengrass console to check the connectivity information for the AWS IoT Greengrass core thing that is being used by the test. If there are 10 endpoints included in the connectivity information, you can remove some or all of them and rerun the test. For more information, see Connectivity information.

124

OTAJobNotCompleteError

An OTA job did not complete.

Check your internet connection and retry the OTA test group.

125

CreateGreengrassServiceRoleError

One of the following occurred:

  • An error occurred while creating a role.

  • An error occurred while attaching a policy to the AWS IoT Greengrass service role.

  • The policy associated with the service role is invalid.

  • An error occurred when associating a role with an AWS account.

Configure the AWS IoT Greengrass service role. For more information, see Greengrass service role.

126

DependenciesNotPresentError

One or more dependencies required for the specific test are not present on the device.

Check the test log to see which dependencies are missing on your device: <device-tester-extract-location>/results/<execution-id>/logs/<test-case-name.log>

127

InvalidHSMConfiguration

The provided HSM/PKCS configuration is incorrect.

In your device.json file, provide the configuration required to interact with the HSM using PKCS#11.

128

OTAJobNotSuccededError

The OTA job did not succeed.

  • If you ran the ota test group individually, run the ggcdependencies test group to verify that all dependencies (such as wget) are present. Then retry the ota test group.

  • Review the detailed logs under <device-tester-extract-location>/results/<execution-id>/logs/ for troubleshooting and error information. Specifically, check the following logs:

    • Console log (test_manager.log)

    • OTA test case log (ota_test.log)

    • GGC daemon log (ota_test_ggc_logs.tar.gz)

    • OTA agent log (ota_test_ota_logs.tar.gz)

  • Check your internet connectivity and retry the ota test group.

  • If the problem persists, contact AWS Developer Support.

129

NoConnectivityError

The host agent is failing to connect to internet.

Check your network connection and firewall settings. Retry the test group after the connectivity issue is resolved.

130

NoPermissionError

The IAM user you are using to run IDT for AWS IoT Greengrass does not have permission to create the AWS resources required to run IDT.

See Permissions policy template for the policy template that grants the permissions required to run IDT for AWS IoT Greengrass.

131

LeftoverAgentExistError

Your device is running AWS IoT Greengrass processes when you attempt to start IDT for AWS IoT Greengrass.

Make sure there is no existing Greengrass daemon running on your device.

  • You can use this command to stop daemon: sudo ./<absolute-path-to-greengrass-daemon>/greengrassd stop.

  • You can also terminate the Greengrass daemon by PID.

Note

If you are using an existing installation of AWS IoT Greengrass configured to start automatically after reboot, you must stop the daemon after reboot and before running the test suite.

132

DeviceTimeOffsetError

The device has the incorrect time.

Set your device to the correct time.

133

InvalidMLConfiguration

The provided ML configuration is incorrect.

In your device.json file, provide the correct configuration required to run ML inference tests. For more information, see Optional: Configuring your device for ML qualification.

Resolving IDT for AWS IoT Greengrass errors

When you use IDT, you must get the correct configuration files in place before you run IDT for AWS IoT Greengrass. If you are getting parsing and configuration errors, your first step is to locate and use a configuration template appropriate for your environment.

If you are still having issues, see the following debugging process.

Where do I look for errors?

High-level errors are displayed on the console during execution, and a summary of the failed tests with the error is displayed when all tests are complete. awsiotdevicetester_report.xml contains a summary of all the errors that caused a test to fail. The log files for each test run are stored in a directory named with an UUID for the test execution that was displayed on the console during the test run.

The test logs directory is located in <device-tester-extract-location>/results/<execution-id>/logs/. This directory contains the following files, which are useful for debugging.

File Description
test_manager.log

All of the logs that were written to the console during the test execution. A summary of the results is located at the end of this file, which includes a list of which tests failed.

The warning and error logs in this file can give you some information about the failures.

<test-group-id>__<test-name>.log Detailed logs for the specific test.
<test-name>_ggc_logs.tar.gz A compressed collection of all the logs the AWS IoT Greengrass core daemon generated during the test. For more information, see Troubleshooting AWS IoT Greengrass.
<test-name>_ota_logs.tar.gz A compressed collection of logs generated by the AWS IoT Greengrass OTA agent during the test. For OTA tests only.
<test-name>_basic_assertion_publisher_ggad_logs.tar.gz A compressed collection of logs generated by the AWS IoT publisher device during the test.
<test-name>_basic_assertion_subscriber_ggad_logs.tar.gz A compressed collection of logs generated by the AWS IoT subscriber device during the test.

Parsing errors

Occasionally, a typo in a JSON configuration can lead to parsing errors. Most of the time, the issue is a result of omitting a bracket, comma, or quotation mark from your JSON file. IDT performs JSON validation and prints debugging information. It prints the line where the error occurred, the line number, and the column number of the syntax error. This information should be enough to help you fix the error, but if you still cannot locate the error, you can perform validation manually in your IDE, a text editor such as Atom or Sublime, or through an online tool like JSONLint.

Required parameter missing error

Because new features are being added to IDT, changes to the configuration files might be introduced. Using an old configuration file might break your configuration. If this happens, the <test_case_id>.log file under /results/<execution-id>/logs explicitly lists all missing parameters. IDT also validates your JSON configuration file schemas to ensure that the latest supported version has been used.

Could not start test error

You might encounter errors that point to failures during test start. There are several possible causes, so do the following:

  • Make sure that the pool name you included in your execution command actually exists. The pool name is referenced directly from your device.json file.

  • Make sure that the devices in your pool have correct configuration parameters.

Not authorized to access resource error

You might see the <user or role> is not authorized to access this resource error message in the terminal output or in the test_manager.log file under /results/<execution-id>/logs. To resolve this issue, attach the AWSIoTDeviceTesterForGreengrassFullAccess managed policy to your test user. For more information, see Create and configure an AWS account.

Permission denied errors

IDT performs operations on various directories and files in a device under test. Some of these operations require root access. To automate these operations, IDT must be able to run commands with sudo without typing a password.

Follow these steps to allow sudo access without typing a password.

Note

user and username refer to the SSH user used by IDT to access the device under test.

  1. Use sudo usermod -aG sudo <ssh-username> to add your SSH user to the sudo group.

  2. Sign out and then sign in for changes to take effect.

  3. Open /etc/sudoers file and add the following line to the end of the file: <ssh-username> ALL=(ALL) NOPASSWD: ALL

    Note

    As a best practice, we recommend that you use sudo visudo when you edit /etc/sudoers.

SSH connection errors

When IDT cannot connect to a device under test, connection failures are logged in /results/<execution-id>/logs/<test-case-id>.log. SSH failure messages appear at the top of this log file because connecting to a device under test is one of the first operations that IDT performs.

Most Windows setups use the PuTTy terminal application to connect to Linux hosts. This application requires that standard PEM private key files are converted into a proprietary Windows format called PPK. When IDT is configured in your device.json file, use PEM files only. If you use a PPK file, IDT cannot create an SSH connection with the AWS IoT Greengrass device and cannot run tests.

Timeout errors

You can increase the timeout for each test by specifying a timeout multiplier, which is applied to the default value of each test's timeout. Any value configured for this flag must be greater than or equal to 1.0.

To use the timeout multiplier, use the flag --timeout-multiplier when running the tests. For example:

./devicetester_linux run-suite --suite-id GGQ_1.0.0 --pool-id DevicePool1 --timeout-multiplier 2.5

For more information, run run-suite --help.

Command not found errors while testing

You need an older version of the OpenSSL library (libssl1.0.0) to run tests on AWS IoT Greengrass devices. Most current Linux distributions use libssl version 1.0.2 or later (v1.1.0).

For example, on a Raspberry Pi, run the following commands to install the required version of libssl:

  1. wget http://ftp.us.debian.org/debian/pool/main/o/openssl/libssl1.0.0_1.0.2l-1~bpo8+1_armhf.deb
  2. sudo dpkg -i libssl1.0.0_1.0.2l-1~bpo8+1_armhf.deb

Security exception on macOS

When you run IDT on host machine that uses macOS 10.15, the notarization ticket for IDT is not correctly detected and IDT is blocked from being run. To run IDT, you will need to grant a security exception to the devicetester_mac_x86-64 executable.

To grant a security exception to the IDT executable

  1. Launch System Preferences from the Apple menu.

  2. Choose Security & Privacy, then on the General tab, click the lock icon to make changes to security settings.

  3. Look for the message "devicetester_mac_x86-64" was blocked from use because it is not from an identified developer. and choose Allow Anyway.

  4. Accept the security warning.

If you have questions about the IDT support policy, contact AWS Customer Support.