Menu
AWS Device Farm
Developer Guide (API Version 2015-06-23)

Analyze a Report in AWS Device Farm

Use the Device Farm console to analyze a report. For more information, see Reports.

Prerequisites

  • Create a run in Device Farm, and verify the run is complete. Follow the instructions in Create a Test Run, and then return to this page.

Console Icons

Icon Description
Green check mark inside of a circle Success
Orange exclamation mark inside of a triangle Warning
Red exclamation mark inside of a circle Failure
Blue circle with a slash through it Skipped
Orange square Stopped

Open a Report with the Device Farm Console

  1. Make sure the run is complete.

  2. Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

  3. In the list of projects, choose the project for the run that corresponds to the report that you want to access.

    Tip

    If the list of projects is not displayed, then on the secondary navigation bar, choose the Device Farm console home button, and then choose the project.

  4. In the list, choose the run with the finished icon that corresponds to the report you want to access. The report's summary page is displayed.

To analyze the various parts of the report, follow the instructions in the following sections.

Analyze a Report's Summary Page with the Device Farm Console

  1. If the report's summary page is not already displayed, follow the instructions in Open a Report with the Device Farm Console.

  2. At the beginning of the summary page, the total number of tests, by outcome, is displayed.

    • An exclamation mark is displayed next to the number of tests with errors.

    • A square is displayed next to the number of stopped tests.

    • An exclamation mark inside of a circle is displayed next to the number of failed tests.

    • A check mark is displayed next to the number of successful tests.

    • A circle with a slash through it is displayed next to the number of skipped tests.

    • An exclamation mark inside of a triangle is displayed next to the number of tests with warnings.

  3. The summary page displays a list of test results as follows:

    • The Unique problems section lists unique warnings and failures. To analyze unique problems, follow the instructions in Analyze a Report's Unique Problems with the Device Farm Console.

    • The Devices section displays the total number of tests, by outcome, for each device.

    • Next to the device's name, one of the following icons is displayed:

      • If there is at least one stopped test for the device, an orange square is displayed.

      • If there is at least one test with errors, a red exclamation mark is displayed.

      • If there is at least one failed test, a red exclamation mark inside of a circle is displayed.

      • If there is at least one test with warnings, an orange exclamation mark inside of a triangle is displayed.

      • Otherwise, a green check mark inside of a circle is displayed.

    • To analyze the results by device, follow the instructions in Analyze a Report by Device with the Device Farm Console.

    • The Screenshots section displays a list of any screenshots Device Farm captured during the run, grouped by device.

Analyze a Report's Unique Problems with the Device Farm Console

  1. If the report's summary page is not already displayed, follow the instructions in Open a Report with the Device Farm Console.

  2. Following the total number of tests by outcome for the run, for Unique problems, choose the problem that you want to analyze. The list of devices for the problem is displayed.

  3. Choose the device whose results you want to analyze. The report displays information about the problem.

  4. The Video section displays a downloadable video recording of the test.

  5. The Logs section displays any information Device Farm logged during the test. To analyze this information, follow the instructions in Analyze Log Information for a Problem, Device, Suite, or Test in a Report with the Device Farm Console.

  6. The Performance section displays information about any performance data Device Farm generated during the test. To analyze this performance data, follow the instructions in Analyze Performance Data for a Problem, Device, Suite, or Test in a Report with the Device Farm Console.

  7. The Files section displays a list of tests for the suite and any associated files (such as log files) that can be downloaded. To download a file, choose the file's link in the list.

  8. The Screenshots section displays a list of any screenshots Device Farm captured during the test.

Analyze a Report by Device with the Device Farm Console

  1. If the report's summary page is not already displayed, follow the instructions in Open a Report with the Device Farm Console.

  2. In the Devices section, choose the device whose results you want to analyze.

  3. The Video section displays a downloadable video recording of the test.

  4. The Suites section displays information about the suites for the device. For each suite, the following test results are displayed:

    • For Test Results, the total number of tests for the suite is displayed by outcome.

    • Next to the suite's name, one of the following icons is displayed:

      • An orange square is displayed if there is at least one stopped test for the suite.

      • A red exclamation mark is displayed if there is at least one test with errors for the suite.

      • A red exclamation mark inside of a circle is displayed if there is at least one failed test for the suite.

      • An orange exclamation mark inside of a triangle is displayed if there is at least one test for the suite with warnings.

      • Otherwise, a green check mark inside of a circle is displayed.

    To analyze the results by suite, follow the instructions in Analyze a Report by Suite with the Device Farm Console.

    The Logs section displays any information Device Farm logged for the device during the run. To analyze this information, follow the instructions in Analyze Log Information for a Problem, Device, Suite, or Test in a Report with the Device Farm Console.

  5. The Performance section displays information about any performance data Device Farm generated for the device during the run. To analyze this performance data, follow the instructions in Analyze Performance Data for a Problem, Device, Suite, or Test in a Report with the Device Farm Console.

  6. The Files section displays a list of suites for the device and any associated files (such as log files) that can be downloaded. To download a file, choose the file's link in the list.

  7. The Screenshots section displays a list of any screenshots Device Farm captured during the run for the device, grouped by suite.

Analyze a Report by Suite with the Device Farm Console

  1. If the report's summary page is not already displayed, follow the instructions in Open a Report with the Device Farm Console.

  2. In the Devices section, choose the device that corresponds to the suite whose results you want to analyze. The device's results page is displayed.

  3. In the Suites section, choose the suite that you want to analyze for results. The suite's results page is displayed.

  4. The suite's results page displays information about the tests for the suite. For each test, the following test results are displayed:

    • For Tests, the outcome for the test is displayed as follows:

      • If the test succeeded, the number 1 is displayed next to a green check mark inside of a circle.

      • If the test has warnings, the number 1 is displayed next to an orange exclamation mark inside of a triangle.

      • If the test was skipped, the number 1 is displayed next to a blue circle with a slash through it.

      • If the test failed, the number 1 is displayed next to a red exclamation mark inside of a circle.

      • If the test has errors, the number 1 is displayed next to a red exclamation mark.

      • If the test was stopped, the number 1 is displayed next to an orange square.

    • These icons are also displayed next to the test's name.

    To analyze the results by test, follow the instructions in Analyze a Report by Test with the Device Farm Console.

  5. The Logs section displays any information Device Farm logged during the run for the suite. To analyze this information, follow the instructions in Analyze Log Information for a Problem, Device, Suite, or Test in a Report with the Device Farm Console.

  6. The Performance section displays information about any performance data Device Farm generated during the run for the suite. To analyze this performance data, follow the instructions in Analyze Performance Data for a Problem, Device, Suite, or Test in a Report with the Device Farm Console.

  7. The Files section displays a list of tests for the suite and any associated files (such as log files) that can be downloaded. To download a file, choose the file's link in the list.

  8. The Screenshots section displays a list of any screenshots Device Farm captured during the run for the suite, grouped by test.

Analyze a Report by Test with the Device Farm Console

  1. If the report's summary page is not already displayed, open the report by following the instructions in Open a Report with the Device Farm Console.

  2. In the Devices section, choose the device that corresponds to the test you want to analyze for results.

  3. In the Suites section, choose the suite that corresponds to the test you want to analyze for results.

  4. The Tests tab displays information about the test.

    • If the test was stopped, an orange square is displayed.

    • If the test contains errors, a red exclamation mark is displayed.

    • If the test failed, a red exclamation mark inside of a circle is displayed.

    • If the test had a warning, an orange exclamation mark inside of a triangle is displayed.

    • Otherwise, a green check mark inside of a circle is displayed .

    The Logs section displays any information Device Farm logged during the test. To analyze this information, follow the instructions in Analyze Log Information for a Problem, Device, Suite, or Test in a Report with the Device Farm Console.

  5. The Performance tab displays information about any performance data Device Farm generated during the test. To analyze this performance data, follow the instructions in Analyze Performance Data for a Problem, Device, Suite, or Test in a Report with the Device Farm Console.

  6. The Files tab displays a list of any of the test's associated files (such as log files) that can be downloaded. To download a file, choose the file's link in the list.

  7. The Screenshots tab displays a list of any screenshots Device Farm captured during the test.

Analyze Performance Data for a Problem, Device, Suite, or Test in a Report with the Device Farm Console

  1. If the Performance tab is not already displayed, follow one of these sets of instructions and choose the Performance tab:

  2. The following information is displayed:

    • The CPU graph displays the percentage of CPU the app used on a single core during the selected problem, device, suite, or test (along the vertical axis) over time (along the horizontal axis).

      The vertical axis is expressed in percentages from 0% to the maximum recorded percentage.

      This percentage may exceed 100% if the app used more than one core. For example, if three cores are at 60% usage, this percentage will be displayed as 180%.

    • The FPS graph displays the frame rate in frames per second (FPS) during the selected problem, device, suite, or test (along the vertical axis) over time (along the horizontal axis).

      The vertical axis is expressed in FPS from 0 FPS to the maximum number of recorded FPS.

    • The Memory graph displays the number of MB the app used during the selected problem, device, suite, or test (along the vertical axis) over time (along the horizontal axis).

      The vertical axis is expressed in MB from 0 MB to the maximum number of recorded MB.

    • The Threads graph displays the number of threads used during the selected problem, device, suite, or test (along the vertical axis) over time (along the horizontal axis).

      The vertical axis is expressed in number of threads from 0 threads to the maximum number of recorded threads.

    In all cases, the horizontal axis is represented, in seconds, from the start and end of the run for the selected problem, device, suite, or test.

  3. To display information for a specific data point, pause in the desired graph at the desired second along the horizontal axis.

Analyze Log Information for a Problem, Device, Suite, or Test in a Report with the Device Farm Console

  1. If the Logs section is not already displayed, follow one of these sets of instructions and choose the Logs tab:

  2. The following information is displayed:

    • Source represents the source of a log entry. Possible values include:

      • Harness represents a log entry Device Farm created. These log entries are typically created during start and stop events.

      • Device represents a log entry the device created. For Android, these log entries are logcat-compatible. For iOS, these log entries are syslog compatible.

      • Test represents a log entry that either a test or its test framework created.

    • Time represents the elapsed time between the first log entry and this log entry. The time is expressed in MM:SS.SSS format, where M represents minutes and S represents seconds.

    • PID represents the process identifier (PID) that created the log entry. All log entries created by an app on a device will have the same PID.

    • Level represents the logging level for the log entry. For example, Logger.debug("This is a message!") would log a Level of Debug. Possible values include the following:

      • Alert

      • Critical

      • Debug

      • Emergency

      • Error

      • Errored

      • Failed

      • Info

      • Internal

      • Notice

      • Passed

      • Skipped

      • Stopped

      • Verbose

      • Warned

      • Warning

    • Tag represents arbitrary metadata for the log entry. For example, Android logcat can use this to describe which part of the system created the log entry (for example, ActivityManager).

    • Message represents the message or data for the log entry. For example, Logger.debug("Hello, World!") would log a Message of "Hello, World!".

  3. To display only a portion of the information, do one or more of the following:

    • To show all log entries that match a value for a specific column, type the value into the corresponding column header box. For example, to show all log entries with a Source value of Harness, type Harness in the Source column header box. Similarly, to show all log entries with a PID value of 969 and a Tag value of ActivityManager, type 969 in the PID column header box, and type ActivityManager in the Tag column header box.

    • To show all log entries that contain zero or more unknown characters for a specific column, use the wildcard character (*) to represent the unknown characters. For example, to show all log entries with a Source value that contain an es (such as Harness and Test), type *es* in the Source column header box. Similarly, to show all log entries that start with a Source value of H (such as Harness) and have a Level value that contains an e (such as Passed), type H* in the PID column header box, and type *e* in the Level column header box.

    • To show log entries that contain a choice between one or more known characters for a specific column, surround the set of choices in parentheses (( )), and use the pipe character (|) to separate each choice. For example, to show log entries with a Message value that contains either started or starting, type *start(ed|ing)* in the Message column header box. Similarly, to show all log entries with a Log value of Info or Debug, type *(Info|Debug)* in the Log column header box.

    • To remove all of the characters from a column header box, choose the X in that column header box. Removing all of the characters from a column header box is the same as typing * in that column header box.

  4. To download all of the log information for the device, including all of the suites and tests that were run, choose Download logs.

    Note

    Even if you display only a portion of the information, if you choose Download logs, all log information for the device will be downloaded.