Creating a test run in Device Farm - AWS Device Farm
PrerequisitesCreate a test run (console)Create a test run (AWS CLI)Create a test run (API)Next steps

Creating a test run in Device Farm

You can use the Device Farm console, AWS CLI, or Device Farm API to create a test run. You can also use a supported plugin, such as the Jenkins or Gradle plugins for Device Farm. For more information about plugins, see Tools and plugins. For information about runs, see Runs.

Prerequisites

You must have a project in Device Farm. Follow the instructions in Creating a project in AWS Device Farm, and then return to this page.

Create a test run (console)

  1. Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

  2. In the navigation pane, choose Mobile Device Testing, and then choose Projects.

  3. If you already have a project, you can upload your tests to it. Otherwise, choose New project, enter a Project Name, and then choose Create.

  4. Open your project, and then choose Create a new run.

  5. On the Choose application page, choose Mobile App or Web App.

    Choose an application in the Device Farm console.

  6. Upload your application file. You can also drag and drop your file or choose a recent upload. If you're uploading an iOS app, be sure to choose iOS device, as opposed to a simulator.

  7. (Optional) In Run name, enter a name. By default, Device Farm uses the app file name.

  8. Choose Next.

  9. On the Configure page, choose one of the available test suites.

    Note

    If you don't have any tests available, choose Built-in: Fuzz to run a standard, built-in test suite. If you choose Built-in: Fuzz, and the Event count, Event throttle, and Randomizer seed boxes appear, you can change or keep the values.

    For information about the available test suites, see Test frameworks and built-in tests in AWS Device Farm.

  10. If you didn't choose Built-in: Fuzz, select Choose File, and then browse to and choose the file that contains your tests.

  11. For your testing environment, choose Run your test in our standard environment or Run your test in a custom environment. For more information, see Test environments in AWS Device Farm.

  12. If you're using the standard test environment, skip to step 13. If you're using a custom test environment with the default test spec YAML file, skip to step 13.

    1. If you want to edit the default test spec in a custom test environment, choose Edit to update the default YAML specification.

    2. If you changed the test spec, choose Save as New to update it.

  13. If you want to configure the video recording or performance data capture options, choose Advanced Configuration.

    1. Select Enable video recording to record video during testing.

    2. Select Enable app performance data capture to capture performance data from the device.

    Note

    If you have private devices, Configuration specific to Private Devices is also displayed.

  14. Choose Next.

  15. On the Select devices page, do one of the following:

    • To choose a built-in device pool to run the tests against, for Device pool, choose Top Devices.

    • To create your own device pool to run the tests against, follow the instructions in Creating a device pool, and then return to this page.

    • If you created your own device pool earlier, for Device pool, choose your device pool.

    For more information, see Device support in AWS Device Farm.

  16. Choose Next.

  17. On the Specify device state page:

    • To provide other data for Device Farm to use during the run, next to Add extra data, choose Choose File, and then browse to and choose the .zip file that contains the data.

    • To install an additional app for Device Farm to use during the run, next to Install other apps, choose Choose File, and then browse to and choose the .apk or .ipa file that contains the app. Repeat this for other apps you want to install. You can change the installation order by dragging and dropping the apps after you upload them.

    • To specify whether Wi-Fi, Bluetooth, GPS, or NFC is enabled during the run, next to Set radio states, select the appropriate boxes.

    • To preset the device latitude and longitude for the run, next to Device location, enter the coordinates.

    • To preset the device locale for the run, in Device locale, choose the locale.

    Note

    Setting the device radio state and locale are options only available for Android native tests at this time.

  18. Choose Next.

  19. On the Review and start run page, you can specify the execution timeout for your test run. If you're using unlimited testing slots, confirm that Run on unmetered slots is selected.

  20. Enter a value or use the slider bar to change the execution timeout. For more information, see Setting the execution timeout for test runs in AWS Device Farm.

  21. Choose Confirm and start run.

Device Farm starts the run as soon as devices are available, typically within a few minutes. During your test run, the Device Farm console displays a pending icon Device Farm scheduled a job. in the run table. Each device in the run will also start with the pending icon, then switch to the running icon Device Farm progress indicator. when the test begins. As each test finishes, a test result icon is displayed next to the device name. When all tests have been completed, the pending icon next to the run changes to a test result icon.

If you want to stop the test run, see Stopping a run in AWS Device Farm.

Create a test run (AWS CLI)

You can use the AWS CLI to create a test run.

Step 1: Choose a project

You must associate your test run with a Device Farm project.

  1. To list your Device Farm projects, run list-projects. If you do not have a project, see Creating a project in AWS Device Farm.

    Example:

    aws devicefarm list-projects

    The response includes a list of your Device Farm projects.

    {
    "projects": [
        {
            "name": "MyProject",
            "arn": "arn:aws:devicefarm:us-west-2:123456789101:project:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE",
            "created": 1503612890.057
        }
    ]
}

  2. Choose a project to associate with your test run, and make a note of its Amazon Resource Name (ARN).

Step 2: Choose a device pool

You must choose a device pool to associate with your test run.

  1. To view your device pools, run list-device-pools, specifying your project ARN.

    Example:

    aws devicefarm list-device-pools --arn arn:MyProjectARN

    The response includes the built-in Device Farm device pools, such as Top Devices, and any device pools previously created for this project:

    {
    "devicePools": [
        {
            "rules": [
                {
                    "attribute": "ARN",
                    "operator": "IN",
                    "value": "[\"arn:aws:devicefarm:us-west-2::device:example1\",\"arn:aws:devicefarm:us-west-2::device:example2\",\"arn:aws:devicefarm:us-west-2::device:example3\"]"
                }
            ],
            "type": "CURATED",
            "name": "Top Devices",
            "arn": "arn:aws:devicefarm:us-west-2::devicepool:example",
            "description": "Top devices"
        },
        {
            "rules": [
                {
                    "attribute": "PLATFORM",
                    "operator": "EQUALS",
                    "value": "\"ANDROID\""
                }
            ],
            "type": "PRIVATE",
            "name": "MyAndroidDevices",
            "arn": "arn:aws:devicefarm:us-west-2:605403973111:devicepool:example2"
        }
    ]
}

  2. Choose a device pool, and make a note of its ARN.

    You can also create a device pool, and then return to this step. For more information, see Create a device pool (AWS CLI).

Step 3: Upload your application file

To create your upload request and get an Amazon Simple Storage Service (Amazon S3) presigned upload URL, you need:

  • Your project ARN.

  • The name of your app file.

  • The type of the upload.

For more information, see create-upload.

  1. To upload a file, run create-upload with the –-project-arn, --name, and --type parameters.

    This example creates an upload for an Android app:

    aws devicefarm create-upload -–project-arn arn:MyProjectArn -–name MyAndroid.apk -–type ANDROID_APP

    The response includes your app upload ARN and a presigned URL.

    {
    "upload": {
        "status": "INITIALIZED",
        "name": "MyAndroid.apk",
        "created": 1535732625.964,
        "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL",
        "type": "ANDROID_APP",
        "arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE"
    }
}

  2. Make a note of the app upload ARN and the presigned URL.

  3. Upload your app file using the Amazon S3 presigned URL. This example uses curl to upload an Android .apk file:

    curl -T MyAndroid.apk "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL"

    For more information, see Uploading objects using presigned URLs in the Amazon Simple Storage Service User Guide.

  4. To check the status of your app upload, run get-upload and specify the ARN of the app upload.

    aws devicefarm get-upload –-arn arn:MyAppUploadARN

    Wait until the status in the response is SUCCEEDED before you upload your test scripts package.

    {
    "upload": {
        "status": "SUCCEEDED",
        "name": "MyAndroid.apk",
        "created": 1535732625.964,
        "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL", 
        "type": "ANDROID_APP",
        "arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE",
        "metadata": "{"valid": true}"
    }
}

Step 4: Upload your test scripts package

Next, you upload your test scripts package.

  1. To create your upload request and get an Amazon S3 presigned upload URL, run create-upload with the –-project-arn, --name, and --type parameters.

    This example creates an Appium Java TestNG test package upload:

    aws devicefarm create-upload –-project-arn arn:MyProjectARN -–name MyTests.zip –-type APPIUM_JAVA_TESTNG_TEST_PACKAGE

    The response includes your test package upload ARN and a presigned URL.

    {
    "upload": {
        "status": "INITIALIZED",
        "name": "MyTests.zip",
        "created": 1535738627.195,
        "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL",
        "type": "APPIUM_JAVA_TESTNG_TEST_PACKAGE",
        "arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE"
    }    
}

  2. Make a note of the ARN of the test package upload and the presigned URL.

  3. Upload your test scripts package file using the Amazon S3 presigned URL. This example uses curl to upload a zipped Appium TestNG scripts file:

    curl -T MyTests.zip "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL"

  4. To check the status of your test scripts package upload, run get-upload and specify the ARN of the test package upload from step 1.

    aws devicefarm get-upload –-arn arn:MyTestsUploadARN

    Wait until the status in the response is SUCCEEDED before you continue to the next, optional step.

    {
    "upload": {
        "status": "SUCCEEDED",
        "name": "MyTests.zip",
        "created": 1535738627.195,
        "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL", 
        "type": "APPIUM_JAVA_TESTNG_TEST_PACKAGE",
        "arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE",
        "metadata": "{"valid": true}"
    }
}

Step 5: (Optional) Upload your custom test spec

If you're running your tests in a standard test environment, skip this step.

Device Farm maintains a default test spec file for each supported test type. Next, you download your default test spec and use it to create a custom test spec upload for running your tests in a custom test environment. For more information, see Test environments in AWS Device Farm.

  1. To find the upload ARN for your default test spec, run list-uploads and specify your project ARN.

    aws devicefarm list-uploads --arn arn:MyProjectARN

    The response contains an entry for each default test spec:

    {
    "uploads": [
        {

            {
                "status": "SUCCEEDED",
                "name": "Default TestSpec for Android Appium Java TestNG",
                "created": 1529498177.474,
                "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL", 
                "type": "APPIUM_JAVA_TESTNG_TEST_SPEC",
                "arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE"
            }
        }
    ]
}

  2. Choose your default test spec from the list. Make a note of its upload ARN.

  3. To download your default test spec, run get-upload and specify the upload ARN.

    Example:

    aws devicefarm get-upload –-arn arn:MyDefaultTestSpecARN

    The response contains a presigned URL where you can download your default test spec.

  4. This example uses curl to download the default test spec and save it as MyTestSpec.yml:

    curl "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL" > MyTestSpec.yml

  5. You can edit the default test spec to meet your testing requirements, and then use your modified test spec in future test runs. Skip this step to use the default test spec as-is in a custom test environment.

  6. To create an upload of your custom test spec, run create-upload, specifying your test spec name, test spec type, and project ARN.

    This example creates an upload for an Appium Java TestNG custom test spec:

    aws devicefarm create-upload --name MyTestSpec.yml --type APPIUM_JAVA_TESTNG_TEST_SPEC --project-arn arn:MyProjectARN

    The response includes the test spec upload ARN and presigned URL:

    {
    "upload": {
        "status": "INITIALIZED",
        "category": "PRIVATE",
        "name": "MyTestSpec.yml",
        "created": 1535751101.221,
        "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL", 
        "type": "APPIUM_JAVA_TESTNG_TEST_SPEC",
        "arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE"
    }
}

  7. Make a note of the ARN for the test spec upload and the presigned URL.

  8. Upload your test spec file using the Amazon S3 presigned URL. This example uses curl to upload an Appium JavaTestNG test spec:

    curl -T MyTestSpec.yml "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL"

  9. To check the status of your test spec upload, run get-upload and specify the upload ARN.

    aws devicefarm get-upload –-arn arn:MyTestSpecUploadARN

    Wait until the status in the response is SUCCEEDED before you schedule your test run.

    {
    "upload": {
        "status": "SUCCEEDED",
        "name": "MyTestSpec.yml",
        "created": 1535732625.964,
        "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL", 
        "type": "APPIUM_JAVA_TESTNG_TEST_SPEC",
        "arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE",
        "metadata": "{"valid": true}"
    }
}

    To update your custom test spec, run update-upload, specifying the upload ARN for the test spec. For more information, see update-upload.

Step 6: Schedule a test run

To schedule a test run with the AWS CLI, run schedule-run, specifying:

  • The project ARN from step 1.

  • The device pool ARN from step 2.

  • The app upload ARN from step 3.

  • The test package upload ARN from step 4.

If you are running tests in a custom test environment, you also need your test spec ARN from step 5.

To schedule a run in a standard test environment

  • Run schedule-run, specifying your project ARN, device pool ARN, application upload ARN, and test package information.

    Example:

    aws devicefarm schedule-run --project-arn arn:MyProjectARN --app-arn arn:MyAppUploadARN --device-pool-arn arn:MyDevicePoolARN --name MyTestRun --test type=APPIUM_JAVA_TESTNG,testPackageArn=arn:MyTestPackageARN

    The response contains a run ARN that you can use to check the status of your test run.

    {
    "run": {
        "status": "SCHEDULING",
        "appUpload": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345appEXAMPLE",
        "name": "MyTestRun",
        "radios": {
            "gps": true,
            "wifi": true,
            "nfc": true,
            "bluetooth": true
        },
        "created": 1535756712.946,
        "totalJobs": 179,
        "completedJobs": 0,
        "platform": "ANDROID_APP",
        "result": "PENDING",
        "devicePoolArn": "arn:aws:devicefarm:us-west-2:123456789101:devicepool:5e01a8c7-c861-4c0a-b1d5-12345devicepoolEXAMPLE",
        "jobTimeoutMinutes": 150,
        "billingMethod": "METERED",
        "type": "APPIUM_JAVA_TESTNG",
        "testSpecArn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345specEXAMPLE",
        "arn": "arn:aws:devicefarm:us-west-2:123456789101:run:5e01a8c7-c861-4c0a-b1d5-12345runEXAMPLE",
        "counters": {
            "skipped": 0,
            "warned": 0,
            "failed": 0,
            "stopped": 0,
            "passed": 0,
            "errored": 0,
            "total": 0
        }
    }
}

    For more information, see schedule-run.

To schedule a run in a custom test environment

  • The steps are the almost the same as those for the standard test environment, with an additional testSpecArn attribute in the --test parameter.

    Example:

    aws devicefarm schedule-run --project-arn arn:MyProjectARN --app-arn arn:MyAppUploadARN --device-pool-arn arn:MyDevicePoolARN --name MyTestRun --test testSpecArn=arn:MyTestSpecUploadARN,type=APPIUM_JAVA_TESTNG,testPackageArn=arn:MyTestPackageARN
To check the status of your test run

  • Use the get-run command and specify the run ARN:

    aws devicefarm get-run --arn arn:aws:devicefarm:us-west-2:111122223333:run:5e01a8c7-c861-4c0a-b1d5-12345runEXAMPLE

For more information, see get-run. For information about using Device Farm with the AWS CLI, see AWS CLI reference.

Create a test run (API)

The steps are the same as those described in the AWS CLI section. See Create a test run (AWS CLI).

You need this information to call the ScheduleRun API:

Note

If you're running tests in a custom test environment, you also need your test spec upload ARN. For more information, see Step 5: (Optional) Upload your custom test spec and CreateUpload.

For information about using the Device Farm API, see Automating Device Farm.

Next steps

In the Device Farm console, the clock icon Device Farm scheduled a job. changes to a result icon such as success The test succeeded. when the run is complete. A report for the run appears as soon as tests are complete. For more information, see Reports in AWS Device Farm.

To use the report, follow the instructions in Viewing test reports in Device Farm.