Amazon FreeRTOS
User Guide

The AWS Documentation website is getting a new look!
Try it now and let us know what you think. Switch to the new look >>

You can return to the original look by selecting English in the language selector above.

Understanding Results and Logs

This section describes how to view and interpret IDT result reports and logs.

Viewing Results

While running, IDT writes errors to the console, log files, and test reports. After IDT completes the qualification test suite, it writes a test run summary to the console and generates two test reports. These reports can be found in <devicetester-extract-location>/results/<execution-id>/. Both reports capture the results from the qualification test suite execution.

The awsiotdevicetester_report.xml is the qualification test report that you submit to AWS to list your device in the AWS Partner Device Catalog. The report contains the following elements:

  • The IDT for Amazon FreeRTOS version.

  • The Amazon FreeRTOS version that was tested.

  • The features of Amazon FreeRTOS that are supported by the device based on the tests passed.

  • The SKU and the device name specified in the device.json file.

  • The features of the device specified in the device.json file.

  • The aggregate summary of test case results.

  • A breakdown of test case results by libraries that were tested based on the device features (for example, FullWiFi, FullMQTT, and so on).

The AFQ_Report.xml is a report in standard JUnit XML format. You can integrate it into CI/CD platforms likeJenkins, Bamboo, and so on. The report contains the following elements:

  • An aggregate summary of test case results.

  • A breakdown of test case results by libraries that were tested based on the device features.

Interpreting IDT for Amazon FreeRTOS Results

The report section in awsiotdevicetester_report.xml or AFQ_Report.xml lists the tests that were run and the results of the tests.

The first XML tag <testsuites> contains the overall summary of the test execution. For example:

<testsuites name="AFQ results" time="5633" tests="184" failures="0" errors="0" disabled="0">

Attributes used in the <testsuites> tag

name

The name of the test suite.

time

The time, in seconds, it took to run the qualification suite.

tests

The number of test cases executed.

failures

The number of test cases that were run, but did not pass.

errors

The number of test cases that IDT for Amazon FreeRTOS couldn't execute.

disabled

This attribute is not used and can be ignored.

If there are no test case failures or errors, your device meets the technical requirements to run Amazon FreeRTOS and can interoperate with AWS IoT services. If you choose to list your device in the AWS Partner Device Catalog, you can use this report as qualification evidence.

In the event of test case failures or errors, you can identify the test case that failed by reviewing the <testsuites> XML tags. The <testsuite> XML tags inside the <testsuites> tag shows the test case result summary for a test group.

<testsuite name="FullMQTT" package="" tests="16" failures="0" time="76" disabled="0" errors="0" skipped="0">

The format is similar to the <testsuites> tag, but with an attribute called skipped that is not used and can be ignored. Inside each <testsuite> XML tag, there are <testcase> tags for each of the test cases that were executed for a test group. For example:

<testcase classname="mcu.Full_MQTT" name="AFQP_MQTT_Connect_HappyCase" attempts="1"></testcase>

Attributes used in the <testcase> tag

name

The name of the test case.

attempts

The number of times IDT for Amazon FreeRTOS executed the test case.

When a test fails or an error occurs, <failure> or <error> tags are added to the <testcase> tag with information for troubleshooting. For example:

<testcase classname="mcu.Full_MQTT" name="AFQP_MQTT_Connect_HappyCase"> <failure type="Failure">Reason for the test case failure</failure> <error>Reason for the test case execution error</error> </testcase>

For more information, see Troubleshooting.

Viewing Logs

You can find logs that IDT for Amazon FreeRTOS generates from test execution in <devicetester-extract-location>/results/<execution-id>/logs. Two sets of logs are generated:

test_manager.log

Contains logs generated from IDT for Amazon FreeRTOS (for example, logs related configuration and report generation).

<test_group_name>.log (for example, Full_MQTT.log)

The logs of the test group, including logs from the device under test.

On this page: