120 lines
4.3 KiB
Markdown
120 lines
4.3 KiB
Markdown
# Codelab: Finding UI elements using chrome.automation API
|
||
|
||
A common task in autotests is to make hardware changes and verify that UI gets
|
||
updated or interact with UI elements and verify that hardware is updated. We can
|
||
use the [chrome.automation] API to help with both of these tasks.
|
||
|
||
[TOC]
|
||
|
||
## Getting familiar with chrome.automation API
|
||
|
||
Detailed information about chrome.automation API can be found at
|
||
https://developer.chrome.com/extensions/automation.
|
||
|
||
In short, the API is a wrapper around Chrome's hierarchy of accessibility nodes
|
||
that describe Chrome UI elements. The most important attributes of accessibility
|
||
nodes are **role** and **name**. See the section on [Accessibility Attributes]
|
||
of the accessiblity overview.
|
||
|
||
## Setup
|
||
|
||
Follow the steps in [Loading autotest extension on
|
||
device](loading-autotest-extension-on-device.md). Loading the AutotestPrivate
|
||
extension will give you access to chrome.automation API as well.
|
||
|
||
## To find a specific UI element
|
||
|
||
Load a js console connected to the autotest extension's background page. See the
|
||
previous section for steps on how to connect to the extension's background page.
|
||
|
||
**NOTE**: The following steps are meant to be run interactively in the console
|
||
and will not work if used in a real test. Section [Using chrome.automation in
|
||
autotests](#Using-chrome_automation-in-autotests) shows how to use the API
|
||
in a real test.
|
||
|
||
Let's start by grabbing a reference to the root node of the accessibility tree.
|
||
|
||
``` js
|
||
var root;
|
||
chrome.automation.getDesktop(r => root = r);
|
||
```
|
||
|
||
### Finding a button in the hierarchy
|
||
|
||
Let's demonstrate how to simulate a click on the launcher button in the system
|
||
shelf.
|
||
|
||
We'll start by listing all buttons visible in the tree.
|
||
|
||
``` js
|
||
root.findAll({attributes: {role: "button"}}).map(node => node.name);
|
||
```
|
||
|
||
After typing that into the console you should get a response such as this:
|
||
|
||
``` js
|
||
> (7) ["Back", "Launcher", "Chromium", "Stylus tools", "Status tray, time 4:21
|
||
PM, Battery is 22% full.", "Connected to Ethernet", "Battery is 22% full. Time
|
||
left until battery is empty, 1 hour and 39 minutes"]
|
||
```
|
||
|
||
**NOTE**: Names will change depending on the locale of the device. We currently
|
||
don't have a locale independent way of identifying UI nodes.
|
||
|
||
Just by looking at button names we can easily guess that the button named
|
||
"Launcher" is the one we're looking for.
|
||
|
||
Finally, to simulate a click on our button:
|
||
|
||
``` js
|
||
var launcher = root.find({attributes: {role: "button", name: "Launcher"}});
|
||
launcher.doDefault();
|
||
```
|
||
|
||
The `doDefault` method performs an action based on the node's *role*, which for
|
||
buttons is a button click.
|
||
|
||
The `find` method supports multiple attributes filters. It returns UI elements
|
||
that satisfy all conditions.
|
||
|
||
## Important roles
|
||
|
||
The API supports interactions with many types of UI elements.
|
||
|
||
The following table contains chrome.automation roles for common UI elements:
|
||
|
||
| views class | chorme.automation role |
|
||
|-----------------:|------------------------|
|
||
| views::Button | button |
|
||
| views::Label | staticText |
|
||
| views::ImageView | image |
|
||
| views::TextField | textField |
|
||
|
||
## Finding name and role of a view subclass
|
||
|
||
View subclasses override the `GetAccessibleNodeData` method to provide role and
|
||
name information.
|
||
|
||
For example, look at [views::Button::GetAccessibleNodeData].
|
||
|
||
## Using chrome.automation in autotests
|
||
|
||
chrome.automation extension can be accessed through the autotest extension.
|
||
|
||
``` python
|
||
with Chrome.chrome(autotest_ext=True) as cr:
|
||
ext = cr.autotest_ext
|
||
ext.ExecuteJavaScript("""
|
||
chrome.automation.getDesktop(root => {
|
||
var launcher = root.find({attributes: {role: 'button', name: 'Launcher'}});
|
||
launcher.doDefault();
|
||
});
|
||
""")
|
||
```
|
||
|
||
[chrome.automation]: https://developer.chrome.com/extensions/automation
|
||
[Accessibility Attributes]: https://chromium.googlesource.com/chromium/src/+/HEAD/docs/accessibility/overview.md#the-accessibility-tree-and-accessibility-attributes
|
||
[views::Button::GetAccessibleNodeData]: https://cs.chromium.org/search/?q=views::Button::GetAccessibleNodeData
|
||
[Getting to a command prompt]: https://www.chromium.org/chromium-os/poking-around-your-chrome-os-device#TOC-Getting-to-a-command-prompt
|
||
[Run Chromium with flags]: https://www.chromium.org/developers/how-tos/run-chromium-with-flags
|