AWS Device Farm
Developer Guide (API Version 2015-06-23)

Create a Test Run in AWS Device Farm

You can use the Device Farm console, AWS CLI, or Device Farm API to create a test run. You can also use a supported plugin, such as the Jenkins or Gradle plugins for Device Farm. For more information about plugins, see Tools and Plugins. For information about runs, see Runs.

Prerequisites

You must have a project in Device Farm. Follow the instructions in Create a Project, and then return to this page.

Create a Test Run (Console)

  1. Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

  2. If you see the AWS Device Farm console home page, enter a name for your project, and then choose Create project. Refresh the page to see your new project.

  3. If you already have a project, you can upload your tests to it.

    Otherwise, choose Create a new project and enter a name.

  4. Open your project, and then choose Create a new run.

  5. On the Choose your application page, choose native application ( 
        Android icon.
    
        iOS icon.
    ) or web application ( 
        HTML 5 web icon.
    ).

    
        Choose an application in the Device Farm console.
  6. Upload your application file. You can also drag and drop your file or choose a recent upload.

    If you are uploading an iOS app, be sure to choose iOS device, as opposed to a simulator.

  7. (Optional) In Run name, enter a name.

    If you don't specify a run name, Device Farm uses the app file name.

  8. Choose Next step.

  9. On the Configure page, choose one of the available test suites.

    Note

    If you do not have any tests available, choose Built-in: Fuzz to run a standard, built-in test suite. If you choose Built-in: Fuzz, and the Event count, Event throttle, and Randomizer seed boxes appear, you can change or keep the values.

    For information about the available test suites, see Working with Test Types in AWS Device Farm.

  10. If you did not choose Built-in: Fuzz, choose Upload, and then browse to and choose the file that contains your tests.

  11. For your execution environment, choose Run test in our standard environment or Customize your test environment. For more information, see Test Environments.

  12. If you are using the standard test environment, skip to step 13. If you are using a custom test environment with the default test spec YAML file, skip to step 13.

    1. If you want to edit the default test spec in a custom test environment, choose Edit to update the default YAML specification.

    2. If you made changes to the test spec, choose Save as New to update the test spec.

  13. If you want to configure the video recording or performance data capture options, choose Advanced Configuration.

    1. Select Enable Video Recording to enable video recording during test execution.

    2. Select Enable App Performance Data Capture to enable the capture of performance data from the device.

    Note

    If you have private devices, Configuration for Private Devices is also displayed.

  14. Choose Next step.

  15. On the Select devices page, do one of the following:

    • To choose a built-in device pool to run the tests against, for Device pool, choose Top Devices.

    • To create your own device pool to run the tests against, follow the instructions in Create a Device Pool, and then return to this page.

    • If you created your own device pool earlier, for Device pool, choose your device pool.

    For more information, see Devices.

  16. Choose Next step.

  17. On the Specify device state page:

    • To provide other data for Device Farm to use during the run, next to Add extra data, choose Upload, and then browse to and choose the .zip file that contains the data.

    • To install an additional app for Device Farm to use during the run, next to Install other apps, choose Upload, and then browse to and choose the .apk or .ipa file that contains the app. Repeat this for other apps you want to install. You can change the installation order by dragging and dropping the apps after you upload them.

    • To specify whether Wi-Fi, Bluetooth, GPS, or NFC is enabled during the run, next to Set radio states, select the appropriate boxes.

    • To preset the device latitude and longitude for the run, next to Device location, type the coordinates.

    • To preset the device locale for the run, in Device Locale, choose the locale.

  18. Choose Review and start run.

  19. On this page, you can specify the execution timeout for your test run.

    If you are using unlimited testing slots, confirm that Run on unmetered slots is selected.

  20. Enter a value or use the slider bar to change the execution timeout. For more information, see Set the Execution Timeout for Test Runs in AWS Device Farm.

  21. Choose Confirm and start run.

Device Farm starts the run as soon as devices are available, typically within a few minutes. Until the run starts, Device Farm displays a calendar icon 
        Device Farm scheduled a job.
    . After the run starts, results appear as tests are completed. During this time, Device Farm displays a progress icon 
        Device Farm progress indicator.
    .

If you need to stop the test run, see Stop a Run in AWS Device Farm.

Create a Test Run (AWS CLI)

You can use the AWS CLI to create a test run.

Step 1: Choose a Project

You must associate your test run with a Device Farm project.

  1. To list your Device Farm projects, run list-projects.

    If you do not have a project, see Create a Project.

    Example:

    aws devicefarm list-projects

    The response includes a list of your Device Farm projects.

    { "projects": [ { "name": "MyProject", "arn": "arn:aws:devicefarm:us-west-2:123456789101:project:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE", "created": 1503612890.057 } ] }
  2. Choose a project to associate with your test run, and make a note of its ARN.

Step 2: Choose Your Devices

You must specify a set of devices for your test run. You can use a device filter to select the devices at the time of a test run, or use an existing device pool.

To Select Devices Using a Device Filter

You can select devices dynamically using the device-selection-configuration parameter when you schedule a test run. It includes your device filters, as well as the maximum number of devices in the test run.

JSON syntax:

{ "filters": [ { "attribute": MyAttribute, "operator": MyOperator, "values": ["MyValue"] } ... ], "maxDevices": MyMaxDevices }
  1. Choose your filter criteria. A filter is made up of an attribute, an operator, and one or more values. For more information, see Select Devices Using a Device Filter.

  2. Choose the maximum number of devices to be included in a test run.

  3. This example selects a single device by its Amazon Resource Name (ARN). It also sets the maximum number of devices to 16.

    { "filters": [ { "attribute": "ARN", "operator": "EQUALS", "value": ["arn:aws:devicefarm:us-west-2::device:1a01b8c7-c972-4c0a-b1d5-12345EXAMPLE"] } ], "maxDevices": 16 }

    Save your filters and maxDevices values in a JSON file to use as the device-selection-configuration request parameter for schedule-run.

  • To Select Devices Using a Device Pool

    You can also select devices for your test run using a device pool. Skip to the next step if you are using a device filter.

    1. Create a device pool. For more information, see Create a Device Pool (AWS CLI).

    2. To view your device pools, run list-device-pools specifying your project ARN.

      Example:

      aws devicefarm list-device-pools --arn arn:MyProjectARN

      The response includes the built-in Device Farm device pools, such as Top Devices, and any device pools previously created for this project:

      { "devicePools": [ { "rules": [ { "attribute": "ARN", "operator": "IN", "value": "[\"arn:aws:devicefarm:us-west-2::device:example1\",\"arn:aws:devicefarm:us-west-2::device:example2\",\"arn:aws:devicefarm:us-west-2::device:example3\"]" } ], "type": "CURATED", "name": "Top Devices", "arn": "arn:aws:devicefarm:us-west-2::devicepool:example", "description": "Top devices" }, { "rules": [ { "attribute": "PLATFORM", "operator": "EQUALS", "value": "\"ANDROID\"" } ], "type": "PRIVATE", "name": "MyAndroidDevices", "arn": "arn:aws:devicefarm:us-west-2:605403973111:devicepool:example2" } ] }
    3. Choose a device pool, and make a note of its ARN.

Step 3: Upload Your Application File

To create your upload request and get an Amazon S3 presigned upload URL, you need:

  • Your project ARN.

  • The name of your app file.

  • The type of the upload.

For more information, see create-upload.

  1. To upload a file, run create-upload with the –-project-arn, --name, and --type parameters.

    This example creates an upload for an Android app:

    aws devicefarm create-upload -–project-arn arn:MyProjectArn -–name MyAndroid.apk -–type ANDROID_APP

    The response includes your app upload ARN and a presigned URL.

    { "upload": { "status": "INITIALIZED", "name": "MyAndroid.apk", "created": 1535732625.964, "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL", "type": "ANDROID_APP", "arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE" } }
  2. Make a note of the app upload ARN and the presigned URL.

  3. Upload your app file using the Amazon S3 presigned URL. This example uses curl to upload an Android .apk file:

    curl -T MyAndroid.apk "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL"

    For more information, see Amazon S3 presigned upload URL.

  4. To check the status of your app upload, run get-upload and specify the ARN of the app upload.

    aws devicefarm get-upload –-arn arn:MyAppUploadARN

    Wait until the status in the response is SUCCEEDED before you upload your test scripts package.

    { "upload": { "status": "SUCCEEDED", "name": "MyAndroid.apk", "created": 1535732625.964, "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL", "type": "ANDROID_APP", "arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE", "metadata": "{"valid": true}" } }

Step 4: Upload Your Test Scripts Package

Next, you upload your test scripts package.

  1. To create your upload request and get an Amazon S3 presigned upload URL, run create-upload with the –-project-arn, --name, and --type parameters.

    This example creates an Appium Java TestNG test package upload:

    aws devicefarm create-upload –-project-arn arn:MyProjectARN -–name MyTests.zip –-type APPIUM_JAVA_TESTNG_TEST_PACKAGE

    The response includes your test package upload ARN and a presigned URL.

    { "upload": { "status": "INITIALIZED", "name": "MyTests.zip", "created": 1535738627.195, "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL", "type": "APPIUM_JAVA_TESTNG_TEST_PACKAGE", "arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE" } }
  2. Make a note of the ARN of the test package upload and the presigned URL.

  3. Upload your test scripts package file using the Amazon S3 presigned URL. This example uses curl to upload a zipped Appium TestNG scripts file:

    curl -T MyTests.zip "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL"
  4. To check the status of your test scripts package upload, run get-upload and specify the ARN of the test package upload from step 1.

    aws devicefarm get-upload –-arn arn:MyTestsUploadARN

    Wait until the status in the response is SUCCEEDED before you continue to the next, optional step.

    { "upload": { "status": "SUCCEEDED", "name": "MyTests.zip", "created": 1535738627.195, "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL", "type": "APPIUM_JAVA_TESTNG_TEST_PACKAGE", "arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE", "metadata": "{"valid": true}" } }

Step 5: Upload Your Custom Test Spec (Optional)

If you are running your tests in a standard test environment, skip this step.

Device Farm maintains a default test spec file for each supported test type. Next, you download your default test spec and use it to create a custom test spec upload for running your tests in a custom test environment. For more information, see Test Environments.

  1. To find the upload ARN for your default test spec, run list-uploads specifying your project ARN.

    aws devicefarm list-uploads --arn arn:MyProjectARN

    The response contains an entry for each default test spec:

    { "uploads": [ { { "status": "SUCCEEDED", "name": "Default TestSpec for Android Appium Java TestNG", "created": 1529498177.474, "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL", "type": "APPIUM_JAVA_TESTNG_TEST_SPEC", "arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE" } } ] }
  2. Choose your default test spec from the list. Make a note of its upload ARN.

  3. To download your default test spec, run get-upload specifying the upload ARN.

    Example:

    aws devicefarm get-upload –-arn arn:MyDefaultTestSpecARN

    The response contains a presigned URL where you can download your default test spec.

  4. This example uses curl to download the default test spec and save it as MyTestSpec.yml:

    curl "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL" > MyTestSpec.yml
  5. You can edit the default test spec to meet your testing requirements, and then use your modified test spec in future test runs. Skip this step to use the default test spec as-is in a custom test environment.

  6. To create an upload of your custom test spec, run create-upload specifying your test spec name, test spec type, and project ARN.

    This example creates an upload for an Appium Java TestNG custom test spec:

    aws devicefarm create-upload --name MyTestSpec.yml --type APPIUM_JAVA_TESTNG_TEST_SPEC --project-arn arn:MyProjectARN

    The response includes the test spec upload ARN and presigned URL:

    { "upload": { "status": "INITIALIZED", "category": "PRIVATE", "name": "MyTestSpec.yml", "created": 1535751101.221, "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL", "type": "APPIUM_JAVA_TESTNG_TEST_SPEC", "arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE" } }
  7. Make a note of the ARN for the test spec upload and the presigned URL.

  8. Upload your test spec file using the Amazon S3 presigned URL. This example uses curl to upload an Appium JavaTestNG test spec:

    curl -T MyTestSpec.yml "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL"
  9. To check the status of your test spec upload, run get-upload and specify the upload ARN.

    aws devicefarm get-upload –-arn arn:MyTestSpecUploadARN

    Wait until the status in the response is SUCCEEDED before you schedule your test run.

    { "upload": { "status": "SUCCEEDED", "name": "MyTestSpec.yml", "created": 1535732625.964, "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL", "type": "APPIUM_JAVA_TESTNG_TEST_SPEC", "arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE", "metadata": "{"valid": true}" } }

    To update your custom test spec, run update-upload specifying the upload ARN for the test spec. For more information, see update-upload.

Step 6: Schedule a Test Run

To schedule a test run with the AWS CLI, run schedule-run specifying:

  • The project ARN from step 1.

  • The device selection filter configuration or the device pool ARN from step 2.

  • The app upload ARN from step 3.

  • The test package upload ARN from step 4.

If you are running tests in a custom test environment, you also need your test spec ARN from step 5.

To schedule a run in a standard test environment

  • Example using a device selection filter:

    aws devicefarm schedule-run --project-arn arn:MyProjectARN --app-arn arn:MyAppUploadARN --device-selection-configuration file://MyDeviceConfig.json --name MyTestRun --test type=APPIUM_JAVA_TESTNG,testPackageArn=arn:MyTestPackageARN

    Example using a device pool:

    aws devicefarm schedule-run --project-arn arn:MyProjectARN --app-arn arn:MyAppUploadARN --device-pool-arn arn:MyDevicePoolARN --name MyTestRun --test type=APPIUM_JAVA_TESTNG,testPackageArn=arn:MyTestPackageARN

    Note

    Either --device-pool-arn or --device-selection-configuration is required for schedule-run.

    The response contains a run ARN that you can use to check the status of your test run.

    { "run": { "status": "SCHEDULING", "appUpload": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345appEXAMPLE", "name": "MyTestRun", "radios": { "gps": true, "wifi": true, "nfc": true, "bluetooth": true }, "created": 1535756712.946, "totalJobs": 179, "completedJobs": 0, "platform": "ANDROID_APP", "result": "PENDING", "devicePoolArn": "arn:aws:devicefarm:us-west-2:123456789101:devicepool:5e01a8c7-c861-4c0a-b1d5-12345devicepoolEXAMPLE", "jobTimeoutMinutes": 150, "billingMethod": "METERED", "type": "APPIUM_JAVA_TESTNG", "testSpecArn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345specEXAMPLE", "arn": "arn:aws:devicefarm:us-west-2:123456789101:run:5e01a8c7-c861-4c0a-b1d5-12345runEXAMPLE", "counters": { "skipped": 0, "warned": 0, "failed": 0, "stopped": 0, "passed": 0, "errored": 0, "total": 0 } } }

    For more information, see schedule-run.

To schedule a run in a custom test environment

  • The steps are the almost the same as those for the standard test environment with an additional testSpecArn attribute included in the --test parameter.

    Example using a device selection filter:

    aws devicefarm schedule-run --project-arn arn:MyProjectARN --app-arn arn:MyAppUploadARN --device-selection-configuration file://MyDeviceConfig.json --name MyTestRun --test testSpecArn=arn:MyTestSpecUploadARN,type=APPIUM_JAVA_TESTNG,testPackageArn=arn:MyTestPackageARN

    Example using a device pool:

    aws devicefarm schedule-run --project-arn arn:MyProjectARN --app-arn arn:MyAppUploadARN --device-pool-arn arn:MyDevicePoolARN --name MyTestRun --test testSpecArn=arn:MyTestSpecUploadARN,type=APPIUM_JAVA_TESTNG,testPackageArn=arn:MyTestPackageARN

    Note

    Either --device-pool-arn or --device-selection-configuration is required for schedule-run.

To check the status of your test run

  • Use the get-run command and specify the run ARN:

    aws devicefarm get-run –arn arn:aws:devicefarm:us-west-2:123456789101:run:5e01a8c7-c861-4c0a-b1d5-12345runEXAMPLE

For more information, see get-run. For information about using Device Farm with the AWS CLI, see AWS CLI Reference.

Create a Test Run (API)

The steps are the same as those described in the AWS CLI section. See Create a Test Run (AWS CLI).

You need this information to call the ScheduleRun API:

Note

If you're running tests in a custom test environment, you also need your test spec upload ARN. For more information, see Step 5: Upload Your Custom Test Spec (Optional) and CreateUpload.

For information about using the Device Farm API, see API Reference.

Next Steps

In the Device Farm console, the progress icon 
        Device Farm progress indicator.
    changes to a result icon such as success 
        The test succeeded.
    when the run is complete. A report for the run appears as soon as tests are complete. For more information, see Reports.

To use the report, follow the instructions in Working with Test Reports.