AWS Device Farm
Developer Guide (API Version 2015-06-23)

Working with Test Reports in AWS Device Farm

Use the Device Farm console to view your test reports. For more information, see Reports.

Prerequisites

Set up a test run and verify that it is complete.

  1. To create a run, see Create a Test Run in AWS Device Farm, and then return to this page.

  2. Verify the run is complete. During your test run, Device Farm displays a progress icon 
        Device Farm progress indicator.
    in the console. The progress icon changes to a test result icon when the run is complete. For more information, see Understanding Test Results.

Understanding Test Results

The Device Farm console displays icons and graphics that allow you to quickly assess the state of your completed test run .

Reporting Results for an Individual Test

For reports that describe an individual test, Device Farm displays an icon and a test result bar:

Description Icon Test Result
The test succeeded. 
        The test succeeded.
    
        The test succeeded.
The test failed. 
        The test failed.
    
        The test failed.
The test was skipped. 
        The test was skipped.
    
        The test was skipped.
The test was stopped. 
        The test was stopped.
    
        The test was stopped.
Device Farm returned a warning. 
        Device Farm returned a warning.
    
        Device Farm returned a warning.
Device Farm returned an error. 
        Device Farm returned an error.
    
        Device Farm returned an error.

Reporting Results for Multiple Tests

For reports that summarize more than one test, Device Farm displays a test results summary bar.


                        Device Farm test results summary bar.

For example, this test run results bar shows that the run had 1 stopped test, 12 failed tests, 4 skipped tests, and 93 successful tests.

The icon displayed for a result from multiple tests, such as a report by test suite, appear in this order:

Icon Description

        The test was stopped.
    There is at least one stopped test.

        Device Farm returned an error.
    There is at least one test with errors.

        The test failed.
    There is at least one failed test.

        Device Farm returned a warning.
    There is at least one test with warnings.

        The test succeeded.
    All tests were successful.

Viewing Reports

You can view the results of your test in the Device Farm console.

View the Test Run Summary Page

  1. Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

  2. In the list of projects, choose the project for the run.

    Tip

    If the list of projects is not displayed, then on the secondary navigation bar, choose the Device Farm console home button, and then choose the project.

  3. Choose a completed run to view its summary report page.

  4. The test run summary page displays an overview of your test results.

    • The Unique problems section lists unique warnings and failures. To view unique problems, follow the instructions in View Unique Problem Reports.

    • The Devices section displays the total number of tests, by outcome, for each device.

      
                                        Device Farm device summary results bar.

      In this example, the Google Pixel device running Android version 7.1.2 reports 11 successful tests that took 15:25 minutes to run.

      To view the results by device, follow the instructions in View Device Reports.

    • The Screenshots section displays a list of any screenshots Device Farm captured during the run, grouped by device.

View Unique Problem Reports

  1. In Unique problems, choose the problem that you want to view.

  2. Choose the device. The report displays information about the problem.

    The Video section displays a downloadable video recording of the test.

    The Logs section displays any information Device Farm logged during the test. To view this information, follow the instructions in View Log Information for a Problem, Device, Suite, or Test in a Report.

    The Performance section displays information about any performance data Device Farm generated during the test. To view this performance data, follow the instructions in View Performance Data for a Problem, Device, Suite, or Test in a Report.

    The Files section displays a list of tests for the suite and any associated files (such as log files) that can be downloaded. To download a file, choose the file's link in the list.

    The Screenshots section displays a list of any screenshots Device Farm captured during the test.

View Device Reports

  • In the Devices section, choose the device.

    Video displays a downloadable video recording of the test.

    Suites displays information about the suites for the device.

    Test results summarizes the number of tests by outcome for each of the test suites run on the device. For more information, see Reporting Results for Multiple Tests.

    To view the results by suite, follow the instructions in View Test Suite Reports.

    The Logs section displays any information Device Farm logged for the device during the run. To view this information, follow the instructions in View Log Information for a Problem, Device, Suite, or Test in a Report.

    The Performance section displays information about any performance data Device Farm generated for the device during the run. To view this performance data, follow the instructions in View Performance Data for a Problem, Device, Suite, or Test in a Report.

    The Files section displays a list of suites for the device and any associated files (such as log files) that can be downloaded. To download a file, choose the file's link in the list.

    The Screenshots section displays a list of any screenshots Device Farm captured during the run for the device, grouped by suite.

View Test Suite Reports

  1. In the Devices section, choose the device.

  2. In the Suites section, choose the suite.

    The suite's results page displays information about the tests for the suite. For more information, see Reporting Results for Multiple Tests.

    To view the results by test, follow the instructions in View Test Reports.

    The Logs section displays any information Device Farm logged during the run for the suite. To view this information, follow the instructions in View Log Information for a Problem, Device, Suite, or Test in a Report.

    The Performance section displays information about any performance data Device Farm generated during the run for the suite. To view this performance data, follow the instructions in View Performance Data for a Problem, Device, Suite, or Test in a Report.

    The Files section displays a list of tests for the suite and any associated files (such as log files) that can be downloaded. To download a file, choose the file's link in the list.

    The Screenshots section displays a list of any screenshots Device Farm captured during the run for the suite, grouped by test.

View Test Reports

  1. In the Devices section, choose the device.

  2. In the Suites section, choose the suite.

    The Tests tab displays information about the test, including a result icon. For more information, see Reporting Results for an Individual Test.

    The Logs section displays any information Device Farm logged during the test. To view this information, follow the instructions in View Log Information for a Problem, Device, Suite, or Test in a Report.

    The Performance tab displays information about any performance data Device Farm generated during the test. To view this performance data, follow the instructions in View Performance Data for a Problem, Device, Suite, or Test in a Report.

    The Files tab displays a list of any of the test's associated files (such as log files) that can be downloaded. To download a file, choose the file's link in the list.

    The Screenshots tab displays a list of any screenshots Device Farm captured during the test.

View Performance Data for a Problem, Device, Suite, or Test in a Report

  • The following information is displayed on the Performance tab:

    • The CPU graph displays the percentage of CPU the app used on a single core during the selected problem, device, suite, or test (along the vertical axis) over time (along the horizontal axis).

      The vertical axis is expressed in percentages, from 0% to the maximum recorded percentage.

      This percentage might exceed 100% if the app used more than one core. For example, if three cores are at 60% usage, this percentage is displayed as 180%.

    • The FPS graph displays the frame rate in frames per second (FPS) during the selected problem, device, suite, or test (along the vertical axis) over time (along the horizontal axis).

      The vertical axis is expressed in FPS, from 0 FPS to the maximum number of recorded FPS.

    • The Memory graph displays the number of MB the app used during the selected problem, device, suite, or test (along the vertical axis) over time (along the horizontal axis).

      The vertical axis is expressed in MB, from 0 MB to the maximum number of recorded MB.

    • The Threads graph displays the number of threads used during the selected problem, device, suite, or test (along the vertical axis) over time (along the horizontal axis).

      The vertical axis is expressed in number of threads, from 0 threads to the maximum number of recorded threads.

    In all cases, the horizontal axis is represented, in seconds, from the start and end of the run for the selected problem, device, suite, or test.

    To display information for a specific data point, pause in the desired graph at the desired second along the horizontal axis.

View Log Information for a Problem, Device, Suite, or Test in a Report

The following information is displayed in the Logs section:

  • Source represents the source of a log entry. Possible values include:

    • Harness represents a log entry Device Farm created. These log entries are typically created during start and stop events.

    • Device represents a log entry the device created. For Android, these log entries are logcat-compatible. For iOS, these log entries are syslog-compatible.

    • Test represents a log entry that either a test or its test framework created.

  • Time represents the elapsed time between the first log entry and this log entry. The time is expressed in MM:SS.SSS format, where M represents minutes and S represents seconds.

  • PID represents the process identifier (PID) that created the log entry. All log entries created by an app on a device have the same PID.

  • Level represents the logging level for the log entry. For example, Logger.debug("This is a message!") logs a Level of Debug. These are the possible values:

    • Alert

    • Critical

    • Debug

    • Emergency

    • Error

    • Errored

    • Failed

    • Info

    • Internal

    • Notice

    • Passed

    • Skipped

    • Stopped

    • Verbose

    • Warned

    • Warning

  • Tag represents arbitrary metadata for the log entry. For example, Android logcat can use this to describe which part of the system created the log entry (for example, ActivityManager).

  • Message represents the message or data for the log entry. For example, Logger.debug("Hello, World!") logs a Message of "Hello, World!".

To display only a portion of the information:

  • To show all log entries that match a value for a specific column, enter the value into the corresponding column header box. For example, to show all log entries with a Source value of Harness, enter Harness in the Source column header box.

  • To show all log entries that contain zero or more unknown characters for a specific column, use the wildcard character (*) to represent the unknown characters. For example, to show all log entries with a Source value that contain an es (such as Harness and Test), enter *es* in the Source column header box.

  • To show log entries that contain a choice between one or more known characters for a specific column, surround the set of choices in parentheses (( )), and use the pipe character (|) to separate each choice. For example, to show log entries with a Message value that contains either started or starting, enter *start(ed|ing)* in the Message column header box.

  • To remove all of the characters from a column header box, choose the X in that column header box. Removing all of the characters from a column header box is the same as typing * in that column header box.

To download all of the log information for the device, including all of the suites and tests that were run, choose Download logs.