Device Advisor workflow
This tutorial provides instructions on how to create a custom test suite and run tests against the device you want to test in the console. After the tests are complete, you can view the test results and detailed logs.
Tutorials
Prerequisites
To complete this tutorial, complete the steps outlined in Setting up.
Create a test suite definition
First, install an AWS SDK.
rootGroup
syntax
A root group is a JSON string that specifies which test cases are included in your test suite as well as any necessary configurations for those test cases. Use the root group to structure and order your test suite in any way you like. The hierarchy of a test suite is:
test suite → test group(s) → test case(s)
A test suite must have at least one test group, and each test group must have at least one test case. Device Advisor runs tests in the order in which you define the test groups and test cases.
Each root group follows this basic structure:
{ "configuration": { // for all tests in the test suite "": "" } "tests": [{ "name": "" "configuration": { // for all sub-groups in this test group "": "" }, "tests": [{ "name": "" "configuration": { // for all test cases in this test group "": "" }, "test": { "id": "" "version": "" } }] }] }
A block that contains a "name"
, "configuration"
, and
"tests"
is referred to as a "group definition". A block that
contains a "name"
, "configuration"
, and
"test"
is referred to as a "test case definition". Each
"test"
block that contains an "id"
and
"version"
is referred to as a "test case".
For information on how to fill in the "id"
and
"version"
fields for each test case ("test"
block), see Device Advisor test cases. That section also contains information on
the available "configuration"
settings.
The following block is an example of a root group configuration that specifies the "MQTT Connect Happy Case" and "MQTT Connect Exponential Backoff Retries" test cases, along with descriptions of the configuration fields.
{ "configuration": {}, // Suite-level configuration "tests": [ // Group definitions should be provided here { "name": "My_MQTT_Connect_Group", // Group definition name "configuration": {} // Group definition-level configuration, "tests": [ // Test case definitions should be provided here { "name": "My_MQTT_Connect_Happy_Case", // Test case definition name "configuration": { "EXECUTION_TIMEOUT": 300 // Test case definition-level configuration, in seconds }, "test": { "id": "MQTT_Connect", // test case id "version": "0.0.0" // test case version } }, { "name": "My_MQTT_Connect_Jitter_Backoff_Retries", // Test case definition name "configuration": { "EXECUTION_TIMEOUT": 600 // Test case definition-level configuration, in seconds }, "test": { "id": "MQTT_Connect_Jitter_Backoff_Retries", // test case id "version": "0.0.0" // test case version } }] }] }
You must supply the root group configuration when you create your test suite
definition. Save the suiteDefinitionId
that is returned in the
response object. This ID is used to retrieve your test suite definition
information and to run your test suite.
Here is a Java SDK example:
response = iotDeviceAdvisorClient.createSuiteDefinition( CreateSuiteDefinitionRequest.builder() .suiteDefinitionConfiguration(SuiteDefinitionConfiguration.builder() .suiteDefinitionName("
your-suite-definition-name
") .devices( DeviceUnderTest.builder() .thingArn("your-test-device-thing-arn
") .certificateArn("your-test-device-certificate-arn
") .build() ) .rootGroup("your-root-group-configuration
") .devicePermissionRoleArn("your-device-permission-role-arn
") .protocol("MqttV3_1_1 || MqttV5"
") .build() ) .build() )
Get a test suite definition
After you create your test suite definition, you receive the
suiteDefinitionId
in the response object of the
CreateSuiteDefinition
API.
You may see that there are new id
fields within each of the group and
test case definitions in the root group that is returned. This is expected; you can
use these IDs to run a subset of your test suite definition.
Java SDK example:
response = iotDeviceAdvisorClient.GetSuiteDefinition( GetSuiteDefinitionRequest.builder() .suiteDefinitionId("
your-suite-definition-id
") .build() )
Get a test endpoint
Use the GetEndpoint
API to get the test endpoint used by your device.
While choosing the endpoint, select the endpoint that best fits the situation. To simultaneously
run multiple test suites, use the Device-level endpoint by
providing a thing ARN
or a certificate ARN
. To run a single test suite,
choose the Account-level endpoint by providing no arguments.
SDK example:
response = iotDeviceAdvisorClient.getEndpoint(GetEndpointRequest.builder() .certificateArn("
your-test-device-certificate-arn
") .thingArn("your-test-device-thing-arn
") .build())
Start a test suite run
After you've successfully created a test suite definition and configured your test
device to connect to your Device Advisor test endpoint, run your test suite with the StartSuiteRun
API. Use
either certificateArn
or thingArn
to run the test suite.
If both are configured, the certificate will be used if it belongs to the
thing.
For .parallelRun()
, use true
if you use Device-level
endpoint to run multiple test suites in parallel using one AWS account.
SDK example:
response = iotDeviceAdvisorClient.startSuiteRun(StartSuiteRunRequest.builder() .suiteDefinitionId("
your-suite-definition-id
") .suiteRunConfiguration(SuiteRunConfiguration.builder() .primaryDevice(DeviceUnderTest.builder() .certificateArn("your-test-device-certificate-arn
") .thingArn("your-test-device-thing-arn
") .build()) .parallelRun(true | false) .build()) .build())
Save the suiteRunId
that is returned in the response. You will use
this to retrieve the results of this test suite run.
Get a test suite run
After you start a test suite run, you can check its progress and its
results with the GetSuiteRun
API.
SDK example:
// Using the SDK, call the GetSuiteRun API. response = iotDeviceAdvisorClient.GetSuiteRun( GetSuiteRunRequest.builder() .suiteDefinitionId("
your-suite-definition-id
") .suiteRunId("your-suite-run-id
") .build())
Stop a test suite run
To stop a test suite run that is still in progress, you can call the
StopSuiteRun
API. After you call the StopSuiteRun
API,
the service will start the cleanup process. While the service is running the cleanup
process, the test suite run status is updated to Stopping
. The cleanup
process will take several minutes, and once the process is complete, the test suite
run status is updated to Stopped
. After a test run has completely
stopped you will be able to start another test suite run. You can periodically check
the suite run status using the GetSuiteRun
API as shown in the previous
section.
SDK example:
// Using the SDK, call the StopSuiteRun API. response = iotDeviceAdvisorClient.StopSuiteRun( StopSuiteRun.builder() .suiteDefinitionId("
your-suite-definition-id
") .suiteRunId("your-suite-run-id
") .build())
Get a qualification report for a successful qualification test suite run
If you run a qualification test suite that completes, you can retrieve a
qualification report by using the GetSuiteRunReport
API. You can use
this qualification report to qualify your device with the AWS IoT Core qualification
program. To determine whether your test suite is a qualification test suite, check
whether the intendedForQualification
parameter is set to
true
. After you call the GetSuiteRunReport
API, the
download URL returned is available for you to download for 90 seconds. If more than
90 seconds elapse from the previous time you called the
GetSuiteRunReport
API, call the API again to retrieve a valid URL.
SDK example:
// Using the SDK, call the getSuiteRunReport API. response = iotDeviceAdvisorClient.getSuiteRunReport( GetSuiteRunReportRequest.builder() .suiteDefinitionId("
your-suite-definition-id
") .suiteRunId("your-suite-run-id
") .build() )