Using AZ Test Scanner
Open 3D Engine (O3DE), the successor to Lumberyard, is now available in Developer Preview. Download O3DE |
The AZ test scanner is a tool for running unit tests that are built into Lumberyard libraries and executables. This tool simplifies testing by automatically finding libraries and executables to test. It also provides flexibility for programmers to focus on testing the parts of Lumberyard that they care about.
The AZ test scanner has two components:
-
An AZ test runner executable that loads libraries to test and capture the test results
-
An AZ test Python module that performs the scanning and reporting functions
Creating Unit and Integration Test Builds
Unit and integration tests are not included in Lumberyard builds by default as they increase the overall size of a game project. Test code can also have unexpected effects on performance. To build components with tests included, you can use a special test variant that works with each configuration.
To create test builds, use the Waf build system in the
same way that you create regular builds. The only difference is that you add
test
to the platform. You can create a test build on Windows using one
of the following examples:
// Build with tests using debug configuration. Outputs to the \Bin64vc141.Debug.Test folder. lmbr_waf.bat build_win_x64_vs2017_debug_test -p all // Build with tests using profile configuration. Outputs to the \Bin64vc141.Test folder. lmbr_waf.bat build_win_x64_vs2017_profile_test -p all // Build with tests using debug configuration. Outputs to the \Bin64vc142.Debug.Test folder. lmbr_waf.bat build_win_x64_vs2019_debug_test -p all // Build with tests using profile configuration. Outputs to the \Bin64vc142.Test folder. lmbr_waf.bat build_win_x64_vs2019_profile_test -p all
Only Windows debug and profile builds are supported for testing. Other platforms are not supported. Release builds are not supported either.
For more information on writing tests, see Writing Tests for AzTest.
Running Unit and Integration Test Builds
A completed test build includes the file AzTestRunner.exe
in the
\Bin64vc141.Test
folder. Although you can use this to run
tests, we recommend that you use the test scanner that uses
AzTestRunner.exe
in an automated manner.
You have two ways to use the scanner:
-
Include the AZ test module in your Python path:
python -m aztest
. -
Use the
lmbr_test.cmd
script located in the Lumberyard\dev
folder. This automatically includes the AZ test module in your Python path and sends all script parameters to the module.
The following example uses the lmbr_test.cmd
scripts. The scanner
has several options but only requires one parameter to operate: the build directory to
scan. You can use the following command to scan your entire test build:
// Scan entire test build and run all found tests lmbr_test.cmd scan --dir Bin64vc141.Debug.Test
The default scan tests libraries only. It does not attempt to test any executables it finds. This is because executables that are not set up to run tests interrupt the scanner until you close the application.
The scanner produces three types of files. All files are created in the current working directory from which the scanner is called:
-
The
aztest.log
file that contains a log of all test output -
Several
.xml
files that contain the test results of each library and executable that has tests, time stamped by default -
An
.html
file that contains a summary of the test results from the entire scan, time stamped by default
The full list of options is shown as follows:
The scanner runs only unit tests by default. This is because unit tests are designed
to be fast and do not rely on engine resources. To run integration tests instead, use
the --integ
flag when calling the scanner:
// Scan test build and run integration tests on CrySystem.dll lmbr_test.cmd scan --dir Bin64vc141.Debug.Test --only CrySystem.dll --integ
For best results, run integration tests on a single library or use an allow list . Scanning the full build might take hours to complete.
Option | Required? | Description |
---|---|---|
--dir, -d |
Yes | The directory to scan for tests. |
--runner-path |
No | Path to the AZ test runner executable (the default is to look in the
directory specified by --dir ). |
--add-path |
No | Adds path to system path before running tests; used for resolving library or executable dependencies. |
--output-path |
No | Sets the path for output folder prefix (the default is
\dev\TestResults ). |
--integ, -i |
No | If set, runs integration tests instead of unit tests. |
--no-timestamp |
No | If set, removes the time stamp from output files. |
--wait-for-debugger |
No | If set, tells the AZ test runner executable to wait for a debugger to be attached before running tests. |
--bootstrap-config |
No | Path to a JSON configuration file for bootstrapping applications required by libraries. |
--limit, -n |
No | Sets a limit for the maximum number of modules to scan. |
--only, -o |
No | Sets a filter to run tests on only the specified library or executable name. |
--whitelist-file |
No | Path to a new line-delimited file used as an inclusion list. The new line-delimited file allows for regular expressions when matching. |
--blacklist-file |
No | Path to a new line-delimited file used as an exclusion list. The exclusion list takes precedence over the inclusion list. The new line-delimited file allows for regular expressions when matching. |
--exe |
No | If set, causes the scanner to call executables for testing. (The default is to test only libraries.) |
The scanner also accepts additional parameters that are passed to the testing framework. For Lumberyard, GoogleTest, and GoogleMock for C++ are used for unit testing. You can enter parameters in the scanner command line as shown in the following example:
// Scan CrySystem.dll and shuffle the test order before running lmbr_test.cmd scan --dir Bin64vc141.Test --only CrySystem.dll --gtest_shuffle
The scanner can also be called as a chained command using Waf. This means that you can
build tests and run them using a single command line. The Waf command
run_tests
calls the scanner on the most recent build folder. For
example:
// Build a debug test build and then run tests in it lmbr_waf.bat build_win_x64_vs2017_debug_test -p all run_tests
The
run_tests
command automatically points to the \Bin64vc141.Debug.Test
folder
to scan. It also uses the
all
option
for inclusion. The build step does not require the use of
run_tests
; it always matches the last build. You can also send all of
the scanner parameters through using --test-params
:
// Run tests on the last build with additional parameters (use quotes to capture as string) lmbr_waf.bat run_tests --test-params="--no-timestamp"
You can also use the --target
flag to build and test just one
module:
lmbr_waf.bat build_win_x64_vs2017_debug_test -p all --target CrySystem run_tests
Including and excluding to filter tests
The test scanner includes the ability to use include and exclude files to filter out libraries and executables that you do not want to test. By default, all found tests are run. In all cases, modules that are excluded are never tested even if they are part of the inclusion list.
Both include and exclude lists use a new line-delimited text file for defining what modules to scan. Each line is treated as a regular expression for matching, allowing for easy filtering by modules with similar names or in the same directory. Here is an example file:
# List files directly (remember to escape backslashes in regex) CrySystem.dll rc\\ResourceCompilerPC.dll # Match similar modules using regex (include all gem libraries) Gem\..*\.dll # Match all in a subdirectory using regex EditorPlugins\\.*
To run the scanner using text files, use the following example:
lmbr_test.cmd scan --dir Bin64vc141.Test --whitelist-file my_include_list.txt --blacklist-file my_exclude_list.txt
You can specify as many inclusion and exclusion lists as you need. The patterns in each file are combined into one set. For example:
lmbr_test.cmd scan --dir Bin64vc141.Test --whitelist-file include1.txt --whitelist-file include2.txt
To set a default include list, create a text file called
lmbr_test_whitelist.txt
. Place the text file in the root
directory of the build (where lmbr_test.cmd
is). The default
exclude list is similarly called lmbr_test_blacklist.txt
. A
default exclude list is provided with the build to capture known libraries that do not
need to be scanned.