Working with custom test environments in AWS Device Farm

In addition to providing a default standard test environment, AWS Device Farm also allows you to configure a custom environment for your automated tests. For more information, see Test environments.

You can set up and configure your custom automated test environment with a YAML-formatted test specification (test spec) file. Device Farm provides a default test spec for each supported test type. You can use the default test spec as-is or edit it, if you prefer. If you edit the test spec file, you can save it for future runs.

For more information, see Uploading a Custom Test Spec Using the AWS CLI and Create a test run in Device Farm.

Topics

Test spec syntax

This is the YAML test spec file structure:



version: 0.1

phases:
  install:
    commands:
      - command
      - command
  pre_test:
    commands:
      - command
      - command
  test:
    commands:
      - command
      - command
  post_test:
    commands:
      - command
      - command

artifacts:
  - location
  - location

The test spec contains the following:

version

Reflects the Device Farm supported test spec version. The current version number is 0.1.

phases

This section contains groups of commands executed during a test run.

The allowed test phase names are:

install

Optional.

Default dependencies for testing frameworks supported by Device Farm are already installed. This phase contains additional commands, if any, that Device Farm runs during installation.

pre_test

Optional.

The commands, if any, executed before your automated test run.

test

Optional.

The commands executed during your automated test run. If any command in the test phase fails, the test is marked as failed.

post_test

Optional.

The commands, if any, executed after your automated test run.

artifacts

Optional.

Device Farm gathers artifacts such as custom reports, log files, and images from a location specified here. Wildcard characters are not supported as part of an artifact location, so you must specify a valid path for each location.

These test artifacts are available for each device in your test run. For information about retrieving your test artifacts, see Using artifacts in a custom test environment.

Important

A test spec must be formatted as a valid YAML file. If the indenting or spacing in your test spec are invalid, your test run can fail. Tabs are not allowed in YAML files. You can use a YAML validator to test whether your test spec is a valid YAML file. For more information, see the YAML website.

Test spec example

This is an example of a Device Farm YAML test spec that configures an Appium Java TestNG test run:


version: 0.1

# Phases are a collection of commands that get executed on Device Farm.
phases:
  # The install phase includes commands that install dependencies that your tests use.
  # Default dependencies for testing frameworks supported on Device Farm are already installed.
  install:
    commands:
      # By default, Appium server version used is 1.7.2.
      # You can switch to an alternate supported version from 1.6.5, 1.7.1, 1.7.2, 1.8.0 or 1.8.1 by using a command like "avm 1.7.1"
      # OR
      # To install a newer version of Appium use the following commands:
      # - export APPIUM_VERSION=1.8.1
      # - avm $APPIUM_VERSION
      # - ln -s /usr/local/avm/versions/$APPIUM_VERSION/node_modules/.bin/appium  /usr/local/avm/versions/$APPIUM_VERSION/node_modules/appium/bin/appium.js

  # The pre-test phase includes commands that setup your test environment.
  pre_test:
    commands:
      # Setup environment variables for java
      - export CLASSPATH=$CLASSPATH:$DEVICEFARM_TESTNG_JAR
      - export CLASSPATH=$CLASSPATH:$DEVICEFARM_TEST_PACKAGE_PATH/*
      - export CLASSPATH=$CLASSPATH:$DEVICEFARM_TEST_PACKAGE_PATH/dependency-jars/*
      # We recommend starting appium server process in the background using the command below.
      # Appium server log will go to $DEVICEFARM_LOG_DIR directory.
      # The environment variables below will be auto-populated during run time.
      - echo "Start appium server"
      - >-
        appium --log-timestamp --device-name $DEVICEFARM_DEVICE_NAME
        --platform-name $DEVICEFARM_DEVICE_PLATFORM_NAME --app $DEVICEFARM_APP_PATH
        --udid $DEVICEFARM_DEVICE_UDID --chromedriver-executable $DEVICEFARM_CHROMEDRIVER_EXECUTABLE  >> $DEVICEFARM_LOG_DIR/appiumlog.txt 2>&1 &

      - >-
        start_appium_timeout=0;
        while [ true ];
        do
            if [ $start_appium_timeout -gt 30 ];
            then
                echo "appium server never started in 30 seconds. Exiting";
                exit 1;
            fi;
            grep -i "Appium REST http interface listener started on 0.0.0.0:4723" $DEVICEFARM_LOG_DIR/appiumlog.txt >> /dev/null 2>&1;
            if [ $? -eq 0 ];
            then
                echo "Appium REST http interface listener started on 0.0.0.0:4723";
                break;
            else
                echo "Waiting for appium server to start. Sleeping for 1 second";
                sleep 1;
                start_appium_timeout=$((start_appium_timeout+1));
            fi;
        done;

  # The test phase includes commands that start your test suite execution.
  test:
    commands:
      # Your test package is downloaded in $DEVICEFARM_TEST_PACKAGE_PATH so we first change directory to that path.
      - echo "Navigate to test package directory"
      - cd $DEVICEFARM_TEST_PACKAGE_PATH

      # By default, the following command is used by Device Farm to run your Appium TestNG test.
      # The goal is to run to your tests jar file with all the dependencies jars in the CLASSPATH.
      # Alternatively, You may specify your customized command.
      # Note: For most use cases, the default command works fine.
      # Please refer "http://testng.org/doc/documentation-main.html#running-testng" for more options on running TestNG tests from the command line.
      - echo "Start Appium TestNG test"
      - java -Dappium.screenshots.dir=$DEVICEFARM_SCREENSHOT_PATH org.testng.TestNG -testjar *-tests.jar -d $DEVICEFARM_LOG_DIR/test-output -verbose 10

      # To run your tests with testng.xml file, use the following sequence:
      # Note: you don't have to unzip your test jar unless you have xml file with custom name
      # 1. Your testng.xml is inside the test jar after packaging step. Unzip it using commands:
      # - echo "Unzipping TestNG tests jar"
      # - unzip *-tests.jar
      # 2. Run your TestSuite by specifying the unzipped testng.xml using commands:
      # - echo "Start Appium TestNG test"
      # - java org.testng.TestNG -d $DEVICEFARM_LOG_DIR/test-output testng.xml

  # The post test phase includes are commands that are run after your tests are executed.
  post_test:
    commands:

# The artifacts phase lets you specify the location where your tests logs, device logs will be stored.
# And also let you specify the location of your test logs and artifacts which you want to be collected by Device Farm.
# These logs and artifacts will be available through ListArtifacts API in Device Farm.
artifacts:
  # By default, Device Farm will collect your artifacts from following directories
  - $DEVICEFARM_LOG_DIR

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

Built-in: fuzz (Android and iOS)

Environment variables