Setting a PIN Speeding up Appium-based tests through desired capabilities Using Webhooks and other APIs after your tests run Adding extra files to your test package

Extending custom test environments in Device Farm

The Device Farm Custom Mode enables you to run more than just your test suite. In this section, you learn how to extend your test suite and optimize your tests.

Setting a PIN

Some applications require that you set a PIN on the device. Device Farm does not support setting a PIN on devices natively. However, this is possible with the following caveats:

The device must be running Android 8 or above.
The PIN must be removed after the test is complete.

To set the PIN in your tests, use the pre_test and post_test phases to set and remove the PIN, as shown following:



phases:
    pre_test:
      - # ... among your pre_test commands
      - DEVICE_PIN_CODE="1234"
      - adb shell locksettings set-pin "$DEVICE_PIN_CODE"
    post_test:
      - # ... Among your post_test commands
      - adb shell locksettings clear --old "$DEVICE_PIN_CODE"

When your test suite begins, the PIN 1234 is set. After your test suite exits, the PIN is removed.

Warning

If you don't remove the PIN from the device after the test is complete, the device and your account will be quarantined.

Speeding up Appium-based tests through desired capabilities

When using Appium, you might find that the standard mode test suite is very slow. This is because Device Farm applies the default settings and doesn't make any assumptions about how you want to use the Appium environment. While these defaults are built around industry best practices, they might not apply to your situation. To fine-tune the parameters of the Appium server, you can adjust the default Appium capabilities in your test spec. For example, the following sets the usePrebuildWDA capability to true in an iOS test suite to speed up initial start time:



phases:
  pre_test:
    - # ... Start up Appium
    - >-
    appium --log-timestamp
    --default-capabilities "{\"usePrebuiltWDA\": true, \"derivedDataPath\":\"$DEVICEFARM_WDA_DERIVED_DATA_PATH\",
    \"deviceName\": \"$DEVICEFARM_DEVICE_NAME\", \"platformName\":\"$DEVICEFARM_DEVICE_PLATFORM_NAME\", \"app\":\"$DEVICEFARM_APP_PATH\",
    \"automationName\":\"XCUITest\", \"udid\":\"$DEVICEFARM_DEVICE_UDID_FOR_APPIUM\", \"platformVersion\":\"$DEVICEFARM_DEVICE_OS_VERSION\"}"
    >> $DEVICEFARM_LOG_DIR/appiumlog.txt 2>&1 &

Appium capabilities must be a shell-escaped, quoted JSON structure.

The following Appium capabilities are common sources of performance improvements:

noReset and fullReset

These two capabilities, which are mutually exclusive, describe the behavior of Appium after each session is complete. When noReset is set to true, the Appium server doesn't remove data from your application when an Appium session ends, effectively doing no cleanup whatsoever. fullReset uninstalls and clears all application data from the device after the session has closed. For more information, see Reset Strategies in the Appium documentation.

ignoreUnimportantViews (Android only)

Instructs Appium to compress the Android UI hierarchy only to relevant views for the test, speeding up certain element lookups. However, this can break some XPath-based test suites because the hierarchy of the UI layout has been changed.

skipUnlock (Android only)

Informs Appium that there is no PIN code currently set, which speeds up tests after a screen off event or other lock event.

webDriverAgentUrl (iOS only)

Instructs Appium to assume that an essential iOS dependency, webDriverAgent, is already running and available to accept HTTP requests at the specified URL. If webDriverAgent isn't already up and running, it can take Appium some time at the beginning of a test suite to start the webDriverAgent. If you start webDriverAgent yourself and set webDriverAgentUrl to http://localhost:8100 when starting Appium, you can boot up your test suite faster. Note that this capability should never be used alongside the useNewWDA capability.

You can use the following code to start webDriverAgent from your test spec file on the device's local port 8100, then forward it to the test host's local port 8100 (this allows you to set webDriverAgentUrl's value to http://localhost:8100). This code should be run during the install phase after any code for setting up the Appium and webDriverAgent environment variables has been defined:


      # Start WebDriverAgent and iProxy
      - >-
        xcodebuild test-without-building -project /usr/local/avm/versions/$APPIUM_VERSION/node_modules/appium/node_modules/appium-webdriveragent/WebDriverAgent.xcodeproj
        -scheme WebDriverAgentRunner -derivedDataPath $DEVICEFARM_WDA_DERIVED_DATA_PATH
        -destination id=$DEVICEFARM_DEVICE_UDID_FOR_APPIUM IPHONEOS_DEPLOYMENT_TARGET=$DEVICEFARM_DEVICE_OS_VERSION
        GCC_TREAT_WARNINGS_AS_ERRORS=0 COMPILER_INDEX_STORE_ENABLE=NO >> $DEVICEFARM_LOG_DIR/webdriveragent_log.txt 2>&1 &
        
        iproxy 8100 8100 >> $DEVICEFARM_LOG_DIR/iproxy_log.txt 2>&1 &

Then, you can add the following code to your test spec file to ensure that webDriverAgent started successfully. This code should be run at the end of the pre-test phase after ensuring that Appium started successfully:


      # Wait for WebDriverAgent to start
      - >-
        start_wda_timeout=0;
        while [ true ];
        do
          if [ $start_wda_timeout -gt 60 ];
          then
              echo "WebDriverAgent server never started in 60 seconds.";
              exit 1;
          fi;
          grep -i "ServerURLHere" $DEVICEFARM_LOG_DIR/webdriveragent_log.txt >> /dev/null 2>&1;
          if [ $? -eq 0 ];
          then
              echo "WebDriverAgent REST http interface listener started";
              break;
          else
              echo "Waiting for WebDriverAgent server to start. Sleeping for 1 seconds";
              sleep 1;
              start_wda_timeout=$((start_wda_timeout+1));
          fi;
        done;

For more information on the capabilities that Appium supports, see Appium Desired Capabilities in the Appium documentation.

Using Webhooks and other APIs after your tests run

You can have Device Farm call a webhook after every test suite finishes using curl. The process to do this varies with the destination and formatting. For your specific webhook, see the documentation for that webhook. The following example posts a message each time a test suite has finished to a Slack webhook:


phases:
  post_test:
    - curl -X POST -H 'Content-type: application/json' --data '{"text":"Tests on '$DEVICEFARM_DEVICE_NAME' have finished!"}' https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX

For more information on using webhooks with Slack, see Sending your first Slack message using Webhook in the Slack API reference.

You are not limited to using curl to call webhooks. Test packages can include extra scripts and tools, as long as they are compatible with the Device Farm execution environment. For example, your test package may include auxiliary scripts that make requests to other APIs. Make sure that any required packages are installed alongside your test suite's requirements. To add a script that runs after your test suite is complete, include the script in your test package and add the following to your test spec:


phases:
  post_test:
    - python post_test.py

Note

Maintaining any API keys or other authentication tokens used in your test package is your responsibility. We recommend that you keep any form of security credential out of source control, use credentials with the fewest possible privileges, and use revokable, short-lived tokens whenever possible. To verify security requirements, see the documentation for the third-party APIs that you use.

If you plan on using AWS services as a part of your test execution suite, you should use IAM temporary credentials, generated outside of your test suite and included in your test package. These credentials should have the fewest granted permissions and shortest lifespan possible. For more information on creating temporary credentials, see Requesting temporary security credentials in the IAM User Guide.

Adding extra files to your test package

You may want to use additional files as a part of your tests either as extra configuration files or additional test data. You can add these additional files to your test package before uploading it to AWS Device Farm, then access them from the custom environment mode. Fundamentally, all test package upload formats (ZIP, IPA, APK, JAR, etc.) are package archive formats that support standard ZIP operations.

You can add files to your test archive before uploading it to AWS Device Farm by using the following command:


$ zip zip-with-dependencies.zip extra_file

For a directory of extra files:


$ zip -r zip-with-dependencies.zip extra_files/

These commands work as expected for all test package upload formats except for IPA files. For IPA files, especially when used with XCUITests, we recommend that you put any extra files in a slightly different location due to how AWS Device Farm resigns iOS test packages. When building your iOS test, the test application directory will be located inside of another directory named Payload.

For example, this is how one such iOS test directory may look:


$ tree
.
└── Payload
    └── ADFiOSReferenceAppUITests-Runner.app
        ├── ADFiOSReferenceAppUITests-Runner
        ├── Frameworks
        │   ├── XCTAutomationSupport.framework
        │   │   ├── Info.plist
        │   │   ├── XCTAutomationSupport
        │   │   ├── _CodeSignature
        │   │   │   └── CodeResources
        │   │   └── version.plist
        │   └── XCTest.framework
        │       ├── Info.plist
        │       ├── XCTest
        │       ├── _CodeSignature
        │       │   └── CodeResources
        │       ├── en.lproj
        │       │   └── InfoPlist.strings
        │       └── version.plist
        ├── Info.plist
        ├── PkgInfo
        ├── PlugIns
        │   ├── ADFiOSReferenceAppUITests.xctest
        │   │   ├── ADFiOSReferenceAppUITests
        │   │   ├── Info.plist
        │   │   └── _CodeSignature
        │   │       └── CodeResources
        │   └── ADFiOSReferenceAppUITests.xctest.dSYM
        │       └── Contents
        │           ├── Info.plist
        │           └── Resources
        │               └── DWARF
        │                   └── ADFiOSReferenceAppUITests
        ├── _CodeSignature
        │   └── CodeResources
        └── embedded.mobileprovision

For these XCUITest packages, add any extra files to the directory ending in .app inside of the Payload directory. For example, the following commands show how you can add a file to this test package:


$ mv extra_file Payload/*.app/
$ zip -r my_xcui_tests.ipa Payload/

When you add a file to your test package, you can expect slightly different interaction behavior in AWS Device Farm based on its upload format. If the upload used the ZIP file extension, AWS Device Farm will automatically unzip the upload before your test and leave the unzipped files at the location with the $DEVICEFARM_TEST_PACKAGE_PATH environment variable. (This means that if you added a file called extra_file to the root of the archive as in the first example, it would be located at $DEVICEFARM_TEST_PACKAGE_PATH/extra_file during the test).

To use a more practical example, if you’re an Appium TestNG user who wants to include a testng.xml file with your test, you can include it in your archive using the following command:


$ zip zip-with-dependencies.zip testng.xml

Then, you can change your test command in the custom environment mode to the following:


java -D appium.screenshots.dir=$DEVICEFARM_SCREENSHOT_PATH org.testng.TestNG -testjar *-tests.jar -d $DEVICEFARM_LOG_DIR/test-output $DEVICEFARM_TEST_PACKAGE_PATH/testng.xml

If your test package upload extension isn't ZIP (e.g., APK, IPA, or JAR file), the uploaded package file itself is found at $DEVICEFARM_TEST_PACKAGE_PATH. Because these are still archive format files, you can unzip the file in order to access the additional files from within. For example, the following command will unzip the contents of the test package (for APK, IPA, or JAR files) to the /tmp directory:


unzip $DEVICEFARM_TEST_PACKAGE_PATH -d /tmp

In the case of an APK or JAR file, you would find your extra files unzipped to the /tmp directory (e.g., /tmp/extra_file). In the case of an IPA file, as explained before, extra files would be in a slightly different location inside the folder ending in .app, which is inside of the Payload directory. For example, based on the IPA example above, the file would be found at the location /tmp/Payload/ADFiOSReferenceAppUITests-Runner.app/extra_file (referenceable as /tmp/Payload/*.app/extra_file).

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

Migrating tests

Working with remote access