diff options
Diffstat (limited to 'en/compatibility/vts/multi-device-testing.html')
| -rw-r--r-- | en/compatibility/vts/multi-device-testing.html | 212 |
1 files changed, 212 insertions, 0 deletions
diff --git a/en/compatibility/vts/multi-device-testing.html b/en/compatibility/vts/multi-device-testing.html new file mode 100644 index 00000000..039baf77 --- /dev/null +++ b/en/compatibility/vts/multi-device-testing.html @@ -0,0 +1,212 @@ +<html devsite> + <head> + <title>Multi-Device Testing</title> + <meta name="project_path" value="/_project.yaml" /> + <meta name="book_path" value="/_book.yaml" /> + </head> + <body> + <!-- + Copyright 2018 The Android Open Source Project + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + --> + + +<p>VTS supports tests that require interaction between multiple Android +devices.</p> + +<h2 id="architecture">Architecture</h2> +<p>VTS uses the TradeFed framework to get and pass device serials to test +modules.</p> + +<p><img src="images/vts_device_serials.png"></p> +<figcaption><strong>Figure 1.</strong> VTS passing device serials.</figcaption> + +<p>Device requirements, such as number of devices and device types, are +specified in test plan configuration. For example, you can specify a test plan +that requires two Android devices with Sailfish build targets.</p> + +<h3 id="device-allocation">Device allocation</h3> +<p>The test infrastructure (usually the test scheduler) allocates available +devices that satisfy the requirements specified in test plan configuration to +the VTS framework. Allocated devices are reserved for the test plan even if the +test module is not using them. VTS agent binaries are then pushed to and run on +all allocated devices (unless specifically instructed not to run). This ensures +that TCP connections for shell commands and HAL RPCs are available for all +devices in a test script.</p> + +<h3 id="test-preparers">Test preparers</h3> +<p>The framework runs test preparers for all devices for which it received +serial numbers. Target preparers can be single or multi-device:</p> +<ul> +<li>Single-device target preparers (example at +<a href="https://android.googlesource.com/platform/test/vts/+/master/harnesses/tradefed/src/com/android/tradefed/targetprep/VtsDeviceInfoCollector.java" class="external">VtsDeviceInfoCollector</a>): +<ul> +<li>Can be specified only in test plan configuration with the required +device list (future versions will allow module level configuration).</li> +<li>Receive only one device serial.</li> +<li>Run preparing and cleanup tasks against a specific device.</li> +</ul> +</li> +<li>Multi-device target preparers (example at +<a href="https://android.googlesource.com/platform/test/vts/+/master/harnesses/tradefed/src/com/android/tradefed/targetprep/VtsPythonVirtualenvPreparer.java" class="external">VtsPythonVirtualenvPreparer</a>): +<ul> +<li>Can be specified in test plan configuration or test module +configuration</li> +<li>Receive all device serials</li> +<li>Run preparing and cleanup tasks for each device or all devices.</li> +</ul> +</li> +</ul> + +<h3 id="test-modules">Test modules</h3> +<p>Test modules get a list of devices after the test preparers finish setting up +the host/devices. One host-side Python test module runs for each multi-device +test module. Allocated Android devices are accessible from Python test modules +as a list of +<a href="https://android.googlesource.com/platform/test/vts/+/master/utils/python/controllers/android_device.py#322" class="external">AndroidDevice</a> +objects:</p> +<pre class="devsite-click-to-copy"> +devices = self.android_devices +device1 = devices[0] +device1_serial = device1.serial +</pre> + +<p>All allocated devices are reserved for the test plan, even though a test +module in the plan is only using one device.</p> + +<h2 id="device-communication">Device communication during testing</h1> +<p>Effective multi-Android tests involve communication between allocated +devices. When developing such tests, you must determine how to establish +communication between the allocated devices. The following sections provide +three communication examples (however, test developers are free to design other +models).</p> + +<h3 id="type1">Type 1: Host-side HAL tests</h3> +<p>Host-side HAL tests can use VTS HAL drivers that are pushed to devices by +default:</p> + +<p><img src="images/vts_hostside_hal.png"></p> +<figcaption><strong>Figure 2.</strong> Host-side HAL test.</figcaption> + +<p>In this scenario:</p> +<ul> +<li>Test logic executes on the host.</li> +<li>Host-side test script issues RPC calls to the drivers on each device.</li> +<li>Host side coordinates device interactions.</li> +</ul> + +<h3 id="type2">Type 2: Host-side agent-based tests</h3> +<p>Instead of using VTS agents on device, a host-side test can also push its own +agent (app or binary) to each device:</p> + +<p><img src="images/vts_hostside_agent.png"></p> +<figcaption><strong>Figure 3.</strong> Host-side, agent-based test.</figcaption> + +<p>In this scenario:</p> +<ul> +<li>Test logic executes on the host.</li> +<li>Agent app (or binary) installs on each device.</li> +<li>Host-side test script issues commands to apps on each device.</li> +<li>Host side coordinates device interactions.</li> +</ul> + +<p>For example, the +<a href="https://android.googlesource.com/platform/test/vts-testcase/nbu/+/master" class="external">Next +Billion User tests</a> in current VTS repo are host-side, app-based, +multi-device tests.</p> + +<h3 id="type3">Type 3: Target-side HIDL tests</h2> +<p>Target-side, multi-device HIDL tests put all test logic on device-side test +binaries, which requires the tests to synchronize devices during test +execution:</p> + +<p><img src="images/vts_target_hidl.png"></p> +<figcaption><strong>Figure 4. </strong>Target-based HIDL test.</figcaption> + +<p>In this scenario:</p> +<ul> +<li>Test logic executes on devices.</li> +<li>Host-side framework provides initial device identification.</li> +<li>Target-side test binary requires synchronization:<ul> +<li>Same test binary for all devices.</li> +<li>Different test binaries for each role.</li> +</ul> +</li> +</ul> + +<h2 id="example-multi-device">Example: Multi-device test plan</h2> +<p>This example specifies the config for two devices:</p> +<ul> +<li>Device 1 includes a build provider and +<code>VtsDeviceInfoCollector</code> target preparer.</li> +<li>Device 2 includes an additional <code>FilePusher</code> preparer that pushes +a group of host-driven related files to the device.</li> +</ul> + +<pre class="devsite-click-to-copy"> +<configuration description="VTS Codelab Plan"> + ... +<device name="device1"> +<build_provider class="com.android.compatibility.common.tradefed.build.CompatibilityBuildProvider" /> +<target_preparer class="com.android.tradefed.targetprep.VtsDeviceInfoCollector" /> +</device> +<device name="device2" > +<build_provider class="com.android.compatibility.common.tradefed.build.CompatibilityBuildProvider" /> +<target_preparer class="com.android.tradefed.targetprep.VtsDeviceInfoCollector" /> +<target_preparer class="com.android.compatibility.common.tradefed.targetprep.VtsFilePusher"> +<option name="push-group" value="HostDrivenTest.push" /> +</target_preparer> +</device> +<option name="compatibility:include-filter" value="VtsCodelabHelloWorldMultiDeviceTest" /> +</configuration> +</pre> + +<h2 id="example-hostside">Example: Host-side Python test script</h2> +<p>For details and examples on test preparers, see +<a href="#Test-preparers">Test Preparers</a>. For a complete host-side +multi-device example, refer to the +<a href="https://android.googlesource.com/platform/test/vts/+/master/testcases/codelab/hello_world_multi" class="external">hello_world_multi +codelab</a>. +</p> + +<pre class="devsite-click-to-copy"> +def setUpClass(self): +logging.info('number of device: %s', self.android_devices) +asserts.assertEqual(len(self.android_devices), 2, 'number of device is wrong.') +self.dut1 = self.android_devices[0] +self.dut2 = self.android_devices[1] +self.shell1 = self.dut1.shell +self.shell2 = self.dut2.shell + +def testSerialNotEqual(self): +'''Checks serial number from two device not being equal.''' +command = 'getprop | grep ro.serial' +res1 = self.shell1.Execute(command) +res2 = self.shell2.Execute(command) + +def getSerialFromShellOutput(output): +'''Get serial from getprop query''' +return output[const.STDOUT][0].strip().split(' ')[-1][1:-1] +serial1 = getSerialFromShellOutput(res1) +serial2 = getSerialFromShellOutput(res2) + +logging.info('Serial number of device 1 shell output: %s', serial1) +logging.info('Serial number of device 2 shell output: %s', serial2) +asserts.assertNotEqual(serial1, serial2, 'serials from two devices should not be the same') +asserts.assertEqual(serial1, self.dut1.serial, 'serial got from device system property is different from allocated serial') +asserts.assertEqual(serial2, self.dut2.serial, 'serial got from device system property is different from allocated serial') +</pre> + +</body> +</html>
\ No newline at end of file |