173 lines
5.9 KiB
Markdown
173 lines
5.9 KiB
Markdown
# Running ART Tests with Atest / Trade Federation
|
|
|
|
ART Testing has early support for execution in the [Trade
|
|
Federation](https://source.android.com/devices/tech/test_infra/tradefed)
|
|
("TradeFed") test harness, in particular via the
|
|
[Atest](https://source.android.com/compatibility/tests/development/atest)
|
|
command line tool.
|
|
|
|
Atest conveniently takes care of building tests and their dependencies (using
|
|
Soong, the Android build system) and executing them using Trade Federation.
|
|
|
|
See also [README.md](README.md) for a general introduction to ART run-tests and
|
|
gtests.
|
|
|
|
## ART run-tests
|
|
|
|
### Running ART run-tests on device
|
|
|
|
ART run-tests are defined in sub-directories of `test/` starting with a number
|
|
(e.g. `test/001-HelloWorld`). Each ART run-test is identified in the build
|
|
system by a Soong module name following the `art-run-test-`*`<test-directory>`*
|
|
format (e.g. `art-run-test-001-HelloWorld`).
|
|
|
|
You can run a specific ART run-test on device by passing its Soong module name
|
|
to Atest:
|
|
```bash
|
|
atest art-run-test-001-HelloWorld
|
|
```
|
|
|
|
To run all ART run-tests in a single command, the currently recommended way is
|
|
to use [test mapping](#test-mapping) (see below).
|
|
|
|
You can nonetheless run all supported ART run-tests with a single Atest command,
|
|
using its support for wildcards:
|
|
```bash
|
|
atest art-run-test-\*
|
|
```
|
|
|
|
Note: Many ART run-tests are failing with the TradeFed harness as of March 2021,
|
|
so the above Atest command will likely report many tests failures. The ART team
|
|
is actively working on this issue.
|
|
|
|
## ART gtests
|
|
|
|
### Running ART gtests on device
|
|
|
|
There are three ways to run ART gtests on device:
|
|
1. by building "standalone" ART gtests and running them against the active ART
|
|
APEX on the device;
|
|
2. by installing the Testing ART APEX (i.e. manually "updating" the ART APEX on
|
|
device); or
|
|
3. by setting up a `chroot` environment on the device, and "activating" the
|
|
Testing ART APEX in that environment.
|
|
|
|
The first approach can be used to test the ART APEX presently residing on a
|
|
device (either the original one, located in the "system" partition, or an
|
|
updated package, present in the "data" partition).
|
|
|
|
The second and third approaches make use of the Testing ART APEX
|
|
(`com.android.art.testing.apex`), and were previously the only options to run
|
|
ART gtests on device, because of build- and link-related limitations (the ART
|
|
gtests had to be part of the ART APEX package itself to be able to build and run
|
|
properly).
|
|
|
|
### Running standalone ART gtests on device
|
|
|
|
Standalone ART gtests are defined as Soong modules `art_standalone_*_tests`. You
|
|
can run them individually with Atest, e.g:
|
|
|
|
```bash
|
|
atest art_standalone_cmdline_tests
|
|
```
|
|
|
|
You can also run all of them with a single Atest command, using its support for
|
|
wildcards:
|
|
|
|
```bash
|
|
atest art_standalone_\*_tests
|
|
```
|
|
|
|
The previous commands build the corresponding ART gtests and their dependencies,
|
|
dynamically link them against local ART APEX libraries (in the source tree), and
|
|
run them on device against the active ART APEX.
|
|
|
|
### Running ART gtests on device by installing the Testing ART APEX
|
|
|
|
You can run ART gtests on device with the ART APEX installation strategy by
|
|
using the following `atest` command:
|
|
|
|
```bash
|
|
atest ArtGtestsTargetInstallApex
|
|
```
|
|
|
|
This command:
|
|
1. builds the Testing ART APEX from the Android source tree (including the ART
|
|
gtests);
|
|
2. installs the Testing ART APEX using `adb install`;
|
|
3. reboots the device;
|
|
4. runs the tests; and
|
|
5. uninstalls the module.
|
|
|
|
You can run the tests of a single ART gtest C++ class using the
|
|
`ArtGtestsTargetInstallApex:`*`<art-gtest-c++-class>`* syntax, e.g.:
|
|
```bash
|
|
atest ArtGtestsTargetInstallApex:JniInternalTest
|
|
```
|
|
|
|
This syntax also supports the use of wildcards, e.g.:
|
|
```bash
|
|
atest ArtGtestsTargetInstallApex:*Test*
|
|
```
|
|
|
|
You can also use Trade Federation options to run a subset of ART gtests, e.g.:
|
|
```bash
|
|
atest ArtGtestsTargetInstallApex -- \
|
|
--module ArtGtestsTargetInstallApex --test '*JniInternalTest*'
|
|
```
|
|
|
|
You can also pass option `--gtest_filter` to the gtest binary to achieve a
|
|
similar effect:
|
|
```bash
|
|
atest ArtGtestsTargetInstallApex -- \
|
|
--test-arg com.android.tradefed.testtype.GTest:native-test-flag:"--gtest_filter=*JniInternalTest*"
|
|
```
|
|
|
|
### Running ART gtests on device using a `chroot` environment
|
|
|
|
You can run ART gtests on device with the chroot-based strategy by using the
|
|
following command:
|
|
|
|
```bash
|
|
atest ArtGtestsTargetChroot
|
|
```
|
|
|
|
This sequence:
|
|
1. builds the Testing ART APEX from the Android source tree (including the ART
|
|
gtests) and all the necessary dependencies for the `chroot` environment;
|
|
2. sets up a `chroot` environment on the device;
|
|
3. "activates" the Testing ART APEX (and other APEXes that it depends on) in the
|
|
`chroot` environment;
|
|
4. runs the tests within the `chroot` environment; and
|
|
5. cleans up the environment (deactivates the APEXes and removes the `chroot`
|
|
environment).
|
|
|
|
## Test Mapping
|
|
|
|
ART Testing supports the execution of tests via [Test
|
|
Mapping](https://source.android.com/compatibility/tests/development/test-mapping).
|
|
The tests declared in ART's [TEST_MAPPING](../TEST_MAPPING) file are executed
|
|
during pre-submit testing (when an ART changelist in Gerrit is verified by
|
|
Treehugger) and/or post-submit testing (when a given change is merged in the
|
|
Android code base), depending on the "test group" where a test is declared.
|
|
|
|
### Running tests via Test Mapping with Atest
|
|
|
|
It is possible to run tests via test mapping locally using Atest.
|
|
|
|
To run all the tests declared in ART's `TEST_MAPPING` file, use the following
|
|
command from the Android source tree top-level directory:
|
|
```bash
|
|
atest --test-mapping art:all
|
|
```
|
|
In the previous command, `art` is the (relative) path to the directory
|
|
containing the `TEST_MAPPING` file listing the tests to run, while `all` means
|
|
that tests declared in all [test
|
|
groups](https://source.android.com/compatibility/tests/development/test-mapping#defining_test_groups)
|
|
shall be run.
|
|
|
|
To only run tests executed during pre-submit testing, use:
|
|
```bash
|
|
atest --test-mapping art:presubmit
|
|
```
|