Troubleshooting IDT for AWS IoT Greengrass V2
IDT for AWS IoT Greengrass V2 writes errors to various locations based on the type of errors. IDT writes errors to the console, log files, and test reports.
Where to look for errors
High-level errors are displayed on the console while the test is running, and a summary of the
failed tests displays when all tests are complete.
awsiotdevicetester_report.xml
contains a summary of all the errors that
caused a test to fail. IDT stores the log files for each test run in a directory with a
UUID for the test execution, displayed on the console during the test run.
The IDT test logs directory is
.
This directory contains the following files displayed in the table. This is useful for debugging.<device-tester-extract-location>
/results/<execution-id>
/logs/
File | Description |
---|---|
test_manager.log |
The logs written to the console while the test was running. The summary of the results at the end of this file includes a list of which tests failed. The warning and error logs in this file can give you some information about the failures. |
|
Detailed logs for the specific test in a test group. For tests that deploy Greengrass
components, the test case log file is called
greengrass-test-run.log . |
|
Detailed logs for AWS IoT Greengrass Core software. IDT copies this file from the device under test when it runs tests that install AWS IoT Greengrass Core software on the device. For more information about the messages in this log file, see Troubleshooting AWS IoT Greengrass V2. |
|
Detailed logs for Greengrass components that are deployed during test runs. IDT copies component log files from the device under test when it runs tests that deploys specific components. The name of each component log file corresponds to the name of the deployed component. For more information about the messages in this log file, see Troubleshooting AWS IoT Greengrass V2. |
Resolving IDT for AWS IoT Greengrass V2 errors
Before you run IDT for AWS IoT Greengrass, get the correct configuration files in place. If you receive parsing and configuration errors, your first step is to locate and use a configuration template appropriate for your environment.
If you are still having issues, see the following debugging process.
Topics
- Alias resolution errors
- Conflict errors
- Could not start test error
- Docker qualification image exists errors
- Failed to read credential
- Guice errors with PreInstalled Greengrass
- Invalid signature exception
- Machine learning qualification errors
- Open Test Framework (OTF) failed deployments
- Parsing errors
- Permission denied errors
- Qualification report generation error
- Required parameter missing error
- Security exception on macOS
- SSH connection errors
- Stream manager qualification errors
- Timeout errors
- Version check errors
Alias resolution errors
When you run custom test suites, you might see the following error in the console and in
the test_manager.log
.
Couldn't resolve placeholders: couldn't do a json lookup: index out of range
This error can occur when the aliases configured in the IDT test orchestrator don't
resolve correctly or if the resolved values aren't present in the configuration files. To
resolve this error, make sure that your device.json
and
userdata.json
contain the correct information required for your test
suite. For information about the configuration required for AWS IoT Greengrass qualification, see Configure IDT settings to run the AWS IoT Greengrass qualification
suite.
Conflict errors
You might see the following error when you run the AWS IoT Greengrass qualification suite concurrently on more than one device.
ConflictException: Component [com.example.IDTHelloWorld : 1.0.0] for account [account-id
] already exists with state: [DEPLOYABLE] { RespMetadata: { StatusCode: 409, RequestID: “id
” }, Message_: “Component [com.example.IDTHelloWorld : 1.0.0] for account [account-id
] already exists with state: [DEPLOYABLE]” }
Concurrent test execution isn't yet supported for the AWS IoT Greengrass qualification suite. Run the qualification suite sequentially for each device.
Could not start test error
You might encounter errors that point to failures that occurred when the test was attempting to start. There are several possible causes, so do the following:
-
Make sure that the pool name in your execution command actually exists. IDT references the pool name directly from your
device.json
file. -
Make sure that the devices in your pool have correct configuration parameters.
Docker qualification image exists errors
The Docker application manager qualification tests use the
amazon/amazon-ec2-metadata-mock
container image in Amazon ECR to qualify the
device under test.
You might receive the following error if the image is already present in a Docker container on the device under test.
The Docker image amazon/amazon-ec2-metadata-mock:
version
already exists on the device.
If you previously downloaded this image and ran the
amazon/amazon-ec2-metadata-mock
container on your device, make sure you
remove this image from the device under test before you run the qualification tests.
Failed to read credential
When testing Windows devices, you might encounter the Failed to read
credential
error in the greengrass.log
file if the user that
you use to connect to the device under test is not set up in the credential manager on that
device.
To resolve this error, configure the user and password for the IDT user in the credential manager on the device under test.
For more information, see Configure user credentials for Windows devices.
Guice errors with PreInstalled Greengrass
While running IDT with PreInstalled Greengrass, if you encounter an
error of Guice
or ErrorInCustomProvider
, check if the file
userdata.json
has the InstalledDirRootOnDevice
set to
the Greengrass installation folder. IDT checks for the file
effectiveConfig.yaml
under
<InstallationDirRootOnDevice>/config/effectiveConfig.yaml
.
For more information, see Configure user credentials for Windows devices.
Invalid signature exception
When you run Lambda qualification tests, you might encounter the
invalidsignatureexception
error if your IDT host machine experiences network
access issues. Reset your router and run the tests again.
Machine learning qualification errors
When you run machine learning (ML) qualification tests, you might encounter qualification failures if your device doesn't meet the requirements to deploy the AWS-provided ML components. To troubleshoot ML qualification errors, do the following:
-
Look for error details in the component logs for the components that were deployed during the test run. Component logs are located in the
directory.<device-tester-extract-location>
/results/<execution-id>
/logs/<test-group-id>
-
Add the
-Dgg.persist=installed.software
argument to thetest.json
file for the failing test case. Thetest.json
file is located in the<device-tester-extract-location>
/tests/GGV2Q_version
directory.
Open Test Framework (OTF) failed deployments
If OTF tests fail to complete the deployment, a likely cause may be the
permissions set for the parent folder of TempResourcesDirOnDevice
and
InstallationDirRootOnDevice
. To set this folder's permissions
correctly, run the following command. Replace
with the name of the
parent folder.folder-name
sudo chmod
755
folder-name
Parsing errors
Typos in a JSON configuration can lead to parsing errors. Most of the time, the issue is a result of omitting a bracket, comma, or quotation mark from your JSON file. IDT performs JSON validation and prints debugging information. It prints the line where the error occurred, the line number, and the column number of the syntax error. This information should be enough to help you fix the error, but if you still can't locate the error, you can perform validation manually in your IDE, a text editor such as Atom or Sublime, or through an online tool like JSONLint.
Permission denied errors
IDT performs operations on various directories and files in a device under test. Some of these operations require root access. To automate these operations, IDT must be able to run commands with sudo without typing a password.
Follow these steps to allow sudo access without typing a password.
Note
user
and username
refer to the SSH user used by IDT to
access the device under test.
-
Use sudo usermod -aG sudo
<ssh-username>
to add your SSH user to the sudo group. -
Sign out and then sign in for changes to take effect.
-
Open
/etc/sudoers
file and add the following line to the end of the file:<ssh-username>
ALL=(ALL) NOPASSWD: ALLNote
As a best practice, we recommend that you use sudo visudo when you edit
/etc/sudoers
.
Qualification report generation error
IDT supports the four latest
versions
of the AWS IoT Greengrass V2 qualification suite (GGV2Q) to generate qualification reports that you can
submit to AWS Partner Network to include your devices in the AWS Partner Device Catalog. Earlier
versions of the qualification suite don’t generate qualification reports.major
.minor
If you have questions about the support policy, contact AWS Support
Required parameter missing error
When IDT adds new features, it might introduce changes to the configuration files. Using
an old configuration file might break your configuration. If this happens, the
file under
<test_case_id>
.log/results/
explicitly lists all missing parameters. IDT also validates your JSON configuration file
schemas to verify that you are using the latest supported version.<execution-id>
/logs
Security exception on macOS
When you run IDT on a macOS host computer, it blocks IDT from running. To run IDT, grant a security exception to the executables that is part of IDT runtime functionality. When you see the warning message display on your host computer, do the following for each of the applicable executables:
To grant a security exception to IDT executables
-
On the macOS computer, on the Apple menu, open System Preferences.
-
Choose Security & Privacy, then on the General tab, choose the lock icon to make changes to security settings.
-
In case of blocked
devicetester_mac_x86-64
, look for the message"devicetester_mac_x86-64" was blocked from use because it is not from an identified developer.
and choose Allow Anyway. -
Resume IDT testing, until you get through all executables involved.
SSH connection errors
When IDT can't connect to a device under test, it logs connection failures in
/results/
.
SSH messages appear at the top of this log file because connecting to a device under test is
one of the first operations that IDT performs.<execution-id>
/logs/<test-case-id>
.log
Most Windows configurations use the PuTTy terminal application to connect to Linux
hosts. This application requires that you convert standard PEM private key files into a
proprietary Windows format called PPK. If you configure SSH in your
device.json
file, use PEM files. If you use a PPK file, IDT can't
create an SSH connection with the AWS IoT Greengrass device and can't run tests.
Starting with IDT v4.4.0, if you haven't enabled SFTP on your device under test, then you might see the following error in the log file.
SSH connection failed with EOF
To resolve this error, enable SFTP on your device.
Stream manager qualification errors
When you run stream manager qualification tests, you might see the following error in
the com.aws.StreamManagerExport.log
file.
Failed to upload data to S3
This error can occur when stream manager uses the AWS credentials in the
~/root/.aws/credentials
file on your device instead of using the
environment credentials that IDT exports to the device under test. To prevent this issue,
delete the credentials
file on your device, and re-run the
qualification test.
Timeout errors
You can increase the timeout for each test by specifying a timeout multiplier applied to the default value of each test's timeout. Any value configured for this flag must be greater than or equal to 1.0.
To use the timeout multiplier, use the flag --timeout-multiplier
when
running the tests. For example:
./devicetester_linux run-suite --suite-id GGV2Q_1.0.0 --pool-id DevicePool1 --timeout-multiplier 2.5
For more information, run run-suite --help
.
Some timeout errors occur when IDT test cases can’t be completed because of configuration issues. You can’t resolve these errors by increasing the timeout multiplier. Use the logs from the test run to troubleshoot the underlying configuration issues.
-
If the MQTT or Lambda component logs contain
Access denied
errors, your Greengrass installation folder might not have the correct file permissions. Run the following command for each folder in the installation path that you defined in youruserdata.json
file.sudo chmod 755
folder-name
-
If the Greengrass logs indicate that the Greengrass CLI deployment isn't complete, do the following:
-
Verify that
bash
is installed on the device under test. -
If your
userdata.json
file includes theGreengrassCliVersion
configuration parameter, remove it. This parameter is deprecated in IDT v4.1.0 and later versions. For more information, see Configure userdata.json.
-
-
If the Lambda deployment test failed with an error message of "Validating Lambda publish: time out" and you receive an error in the test log file (
idt-gg2-lambda-function-idt-
) that says<resource-id>
.logError: Could not find or load main class com.amazonaws.greengrass.runtime.LambdaRuntime.
, do the following:-
Verify what folder was used for
InstallationDirRootOnDevice
in theuserdata.json
file. -
Make sure the correct user permissions are set up on your device. For more details, see Configure user permissions on your device.
-
Version check errors
IDT issues the following error when the AWS user credentials for the IDT user don't have the required IAM permissions.
Failed to check version compatibility
The AWS user that doesn't have the required IAM permissions.