1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
|
<html devsite>
<head>
<title>Running CTS tests</title>
<meta name="project_path" value="/_project.yaml" />
<meta name="book_path" value="/_book.yaml" />
</head>
<body>
<!--
Copyright 2017 The Android Open Source Project
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<h2 id=cts_tradefed>Getting started with CTS tradefed</h2>
<p>See the <a
href="/devices/tech/test_infra/tradefed/index.html">Trade Federation
Overview</a> for an explanation of the Trade Federation (tradefed or TF for
short) continuous test framework.</p>
<p>To run a test plan:</p>
<ol>
<li>Connect at least one device.
<li>Press the <strong>home</strong> button to set the device to the home screen at the start of CTS.
<li>While a device is running tests, it must not be used for any other tasks
and must be kept in a stationary position (to avoid triggering sensor activity)
with the cameras pointing at an object that could be focused.
<li>Do not press any keys on the device while the CTS is running. Pressing keys
or touching the screen of a test device will interfere with the running tests
and may lead to test failures.
<li>Launch the CTS console by running the <em>cts-tradefed</em> script from
the folder where the CTS package has been unzipped, e.g.
<code>$ ./android-cts/tools/cts-tradefed</code>
<li>Start the default test plan (contains all test packages) by appending: <code>run
cts --plan CTS</code> . This kicks off all CTS tests required for compatibility.
<ul>
<li>For CTS v1 (Android 6.0 and earlier), enter <code>list plans</code> to view a list of
test plans in the repository or <code>list package</code>s to view a list of
test packages in the repository.
<li>For CTS v2 (Android 7.0 and later), enter <code>list modules</code> to
see a list of test modules.
</ul>
</li>
<li>Alternately, run the CTS plan of your choosing from the command line
using: <code>cts-tradefed run cts --plan
<plan_name>
</code>
<p class="note"><strong>Note:</strong> When running Android 6.0 (Marshmallow)
CTS only, we recommend you use the <code>--skip-preconditions</code> option to
skip the experimental pre-conditions feature that may cause issues for when
executing CTS tests.</p>
<li>View test progress and results reported on the console.
<li>If your device is Android 5.0 or later and declares support for an ARM and a
x86 ABI, you should run both the ARM and x86 CTS packages.
</ol>
<h2 id=using-cts-v1>Using the CTS v1 console</h2>
<p>For Android 6.0 or earlier, you'll use CTS v1.</p>
<h3 id=selecting_cts_plans>Selecting plans</h3>
<p>The following test plans are available:</p>
<ul>
<li><em>CTS</em>—all tests required for compatibility. </li>
<li><em>Signature</em>—the signature verification of all public APIs </li>
<li><em>Android</em>—tests for the Android APIs </li>
<li><em>Java</em>—tests for the Java core library </li>
<li><em>VM</em>—tests for ART or Dalvik </li>
<li><em>Performance</em>—performance tests for your implementation </li>
</ul>
<p>These can be executed with the <code>run cts</code> command.</p>
<h3 id=cts_reference>CTS v1 console command reference</h3>
<p class="table-caption" id="console-commands">
<strong>Table 1.</strong> This table summarizes the CTS v1 console commands for
various uses.</p>
<table>
<tbody>
<tr>
<th>Host</th>
<th>Description</th>
</tr>
<tr>
<td><code>help</code></td>
<td>Display a summary of the most commonly used commands</td>
</tr>
<tr>
<td><code>help all</code></td>
<td>Display the complete list of available commands</td>
</tr>
<tr>
<td><code>exit</code></td>
<td>Gracefully exit the CTS console. Console will close when all currently running tests are finished</td>
</tr>
<tr>
<th>Run</th>
<th>Description</th>
</tr>
<tr>
<td><code>run cts</code></td>
<td>Run the specified tests and displays progress information. One of
<code>--plan</code>, <code>--package</code>, <code>--class</code> or
<code>--continue-session</code> needs to be specified
<p>The CTS console can accept other commands while tests are in progress </p>
<p>If no devices are connected, the CTS desktop machine (or host) will wait for a device to be connected before starting tests </p>
<p>If more than one device is connected, the CTS host will choose a device automatically</p></td>
</tr>
<tr>
<td><code>--plan <test_plan_name></code></td>
<td>Run the specified test plan</td>
</tr>
<tr>
<td><code>--package/-p <test_package_name> [--package/-p <test_package2>...]</code></td>
<td>Run the specified test packages</td>
</tr>
<tr>
<td><code>--class/-c <class_name> [--method/-m <test_method_name></code></td>
<td>Run the specified test class and/or method</td>
</tr>
<tr>
<td><code>--continue-session</code></td>
<td>Run all not executed tests from previous CTS session; the sessions testResult.xml will be updated with the new results</td>
</tr>
<tr>
<td><code>--shards <number_of_shards></code></td>
<td>Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel</td>
</tr>
<tr>
<td><code>--serial/-s <deviceID></code></td>
<td>Run CTS on the specific device</td>
</tr>
<tr>
<td><code>-t <class_name>#<test_method_name></code></td>
<td>Run a specific test method</td>
</tr>
<tr>
<td><code>--force-abi 32|64</code></td>
<td>On 64-bit devices, run the test against only the 32-bit or 64-bit ABI</td>
</tr>
<tr>
<th>List</th>
<th>Description</th>
</tr>
<tr>
<td><code>list packages</code></td>
<td>List all available test packages in the repository</td>
</tr>
<tr>
<td><code>list plans</code></td>
<td>List all available test plans in the repository</td>
</tr>
<tr>
<td><code>list invocations</code></td>
<td>List 'run' commands currently being executed on devices</td>
</tr>
<tr>
<td><code>list commands</code></td>
<td>List all 'run' commands currently in the queue waiting to be assigned to devices</td>
</tr>
<tr>
<td><code>list results</code></td>
<td>List CTS results currently stored in repository</td>
</tr>
<tr>
<td><code>list devices</code></td>
<td>List currently connected devices and their state
<p> </p>
<p>'Available' devices are functioning, idle devices, available for running tests</p>
<p> </p>
<p>'Unavailable' devices are devices visible via adb, but are not responding to adb commands and won't be allocated for tests</p>
<p> </p>
<p>'Allocated' devices are devices currently running tests</td>
</tr>
<tr>
<th>Add</th>
<th>Description</th>
</tr>
<tr>
<td><code>add derivedplan --plan <plan_name><br>
--result/-r<br>
[pass | fail | timeout | notExecuted]<br>
[--session/-s <session_id>]</code></td>
<td>Create a plan derived from given result session; use this option to rerun reports and validate test issues</td>
</tr>
</tbody>
</table>
<h2 id=using-cts-v2>Using the CTS v2 console</h2>
<p>For Android 7.0 or later, you'll use CTS v2.</p>
<h3 id=selecting_ctsv2_plans>Selecting plans</h3>
<p>Available test plans include the following:</p>
<ul>
<li><em>cts</em>—Runs CTS from an pre-existing CTS installation.</li>
<li><em>cts-camera</em>— Runs CTS-camera from a pre-existing CTS installation.</li>
<li><em>cts-java</em>— Runs Core Java Tests from a pre-existing CTS installation.</li>
<li><em>cts-pdk</em>— Runs Tests useful on validating a PDK fusion build.</li>
<li><em>everything</em>— Common config for Compatibility suites.</li>
</ul>
<p>Other available configurations include the following:</p>
<ul>
<li><em>basic-reporters</em>— Configuration with basic CTS reporters.</li>
<li><em>collect-tests-only</em>—Runs CTS from a pre-existing CTS installation.</li>
<li><em>common-compatibility-config</em>— Common config for Compatibility suites.</li>
<li><em>cts-filtered-sample</em>— Common config for Compatibility suites.</li>
<li><em>cts-known-failures</em>— Configuration with CTS known failures.</li>
<li><em>cts-preconditions</em>— CTS precondition configs.</li>
<li><em>host</em>— Runs a single host-based test on an existing device.</li>
<li><em>instrument</em>— Runs a single Android instrumentation test on an existing device.</li>
<li><em>native-benchmark</em>— Runs a native stress test on an existing device.</li>
<li><em>native-stress</em>— Runs a native stress test on an existing device.</li>
<li><em>recharge</em>— A fake test that waits for nearly-discharged devices and holds them for charging.</li>
<li><em>testdef</em>— Runs tests contained in test_def.xml files on an existing device.</li>
<li><em>util/wifi</em>— Utility config to configure Wi-Fi on device.</li>
<li><em>util/wipe</em>— Wipes user data on device.</li>
</ul>
<p>All of these plans and configs can be executed with the <code>run cts</code> command.</p>
<h3 id=ctsv2_reference>CTS v2 console command reference</h3>
<p class="table-caption" id="console-commands">
<strong>Table 1.</strong> This table summarizes the CTS V2 console commands for
various uses.</p>
<table>
<tbody>
<tr>
<th>Host</th>
<th>Description</th>
</tr>
<tr>
<td><code>help</code></td>
<td>Display a summary of the most commonly used commands</td>
</tr>
<tr>
<td><code>help all</code></td>
<td>Display the complete list of available commands</td>
</tr>
<tr>
<td><code>version</code></td>
<td>Show the version.</td>
</tr>
<tr>
<td><code>exit</code></td>
<td>Gracefully exit the CTS console. Console will close when all currently running tests are finished.</td>
</tr>
<tr>
<th>Run</th>
<th>Description</th>
</tr>
<tr>
<td><code>run cts</code></td>
<td><p>Run the default CTS plan (that is, the full CTS invocation).</p>
<p>The CTS console can accept other commands while tests are in progress.</p>
<p>If no devices are connected, the CTS desktop machine (or host) will wait
for a device to be connected before starting tests.</p>
<p>If more than one device is connected, the CTS host will choose a device automatically.</p></td>
</tr>
<tr>
<td><code>run retry</code></td>
<td><p><strong>For Android 9</strong>. Retry all tests that failed or were not executed from the
previous sessions. For example, <code>run retry --retry <session id> -s<device serial></code>,
or <code>run retry --retry <session id> --shard-count</code> with TF sharding.</p>
<p><code>run cts --retry</code> is not allowed for Android 9.</td>
</tr>
<tr>
<td><code>--plan <test_plan_name></code></td>
<td>Run the specified test plan.</td>
</tr>
<tr>
<td><code>--module/-m <test_module_name> [--module/-m <test_module2>...]</code></td>
<td>Run the specified test module or modules. For example, <code>run cts --module CtsGestureTestCases</code>
executes the gesture test module (this can be shortened to <code>run cts -m Gesture</code>).</br>
<code>run cts -m Gesture --test android.gesture.cts.GestureTest#testGetStrokes</code> runs the specific
package, class, or test.</td>
</tr>
<tr>
<td><code>--subplan <subplan_name></code></td>
<td>Run the specified subplan.</td>
</tr>
<tr>
<td><code>-- module/-m <test_module_name> -- test <test_name> </code></td>
<td>Run the specified module and test. For example,
<code>run cts -m Gesture --test android.gesture.cts.GestureTest#testGetStrokes</code> runs the specific
package, class, or test.</td>
</tr>
<tr>
<td><code>--retry</code></td>
<td>Retry all tests that failed or were not executed from the previous sessions.
Use <code>list results</code> to get the session id.</td>
</tr>
<tr>
<td><code>--retry-type not_executed</code></td>
<td>Retry only tests that were not executed from the previous sessions.
Use <code>list results</code> to get the session id.</td>
</tr>
<tr>
<td><code>--shards <number_of_shards></code></td>
<td><strong>For Android 8.1 and earlier versions</strong>. Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel.</td>
</tr>
<tr>
<td><code>--shard-count <number_of_shards></code></td>
<td><strong>For Android 9</strong>. Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel.</td>
</tr>
<tr>
<td><code>--serial/-s <deviceID></code></td>
<td>Run CTS on the specific device.</td>
</tr>
<tr>
<td><code>--include-filter <module_name> [--include-filter <module2>...]</code></td>
<td>Run only with the specified modules.</td>
</tr>
<tr>
<td><code>--exclude-filter <module_name> [--exclude-filter <module2>...]</code></td>
<td>Exclude the specified modules from the run.</td>
</tr>
<tr>
<td><code>--log-level-display/-l <log_level></code></td>
<td>Run with the minimum specified log level displayed to STDOUT. Valid values: [VERBOSE, DEBUG, INFO, WARN, ERROR, ASSERT].</td>
</tr>
<tr>
<td><code>--abi <abi_name></code></td>
<td>Force the test to run on the given ABI, 32 or 64. By default CTS runs a test once for each ABI the device supports.</td>
</tr>
<tr>
<td><code>--logcat</code>, <code>--bugreport</code>, and <code>--screenshoot-on-failure</code></td>
<td>Give more visibility into failures and can help with diagnostics.</td>
</tr>
<tr>
<td><code>--device-token</code></td>
<td>Specifies a given device has the given token eg. <code>--device-token 1a2b3c4d:sim-card.</code>.</td>
</tr>
<tr>
<td><code>--skip-device-info</code></td>
<td>Skips collection of information about the device. Note: do not use this option when running CTS for approval.</td>
</tr>
<tr>
<td><code>--skip-preconditions</code></td>
<td>Bypasses verification and setup of the device's configuration, such as pushing media files or checking for Wi-Fi connection.</td>
</tr>
<tr>
<th>List</th>
<th>Description</th>
</tr>
<tr>
<td><code>list modules</code></td>
<td>List all available test modules in the repository.</td.>
</tr>
<tr>
<td><code>list plans</code> or <code>list configs</code></td>
<td>List all available test plans (configs) in the repository.</td>
</tr>
<tr>
<td><code>list subplans</code></td>
<td>List all available subplans in the repository.</td>
</tr>
<tr>
<td><code>list invocations</code></td>
<td>List 'run' commands currently being executed on devices.</td>
</tr>
<tr>
<td><code>list commands</code></td>
<td>List all 'run' commands currently in the queue waiting to be assigned to devices.</td>
</tr>
<tr>
<td><code>list results</code></td>
<td>List CTS results currently stored in repository.</td>
</tr>
<tr>
<td><code>list devices</code></td>
<td>List currently connected devices and their state.
<p> </p>
<p>'Available' devices are functioning, idle devices, available for running tests.</p>
<p> </p>
<p>'Unavailable' devices are devices visible via adb, but are not responding to adb commands and won't be allocated for tests.</p>
<p> </p>
<p>'Allocated' devices are devices currently running tests.</td>
</tr>
<tr>
<th>Dump</th>
<th>Description</th>
</tr>
<tr>
<td><code>dump logs</code></td>
<td>Dump the tradefed logs for all running invocations.</td>
</tr>
<tr>
<th>Add</th>
<th>Description</th>
</tr>
<tr>
<td><code>add subplan --name/-n <subplan_name><br>
--result-type<br>
[pass | fail | timeout | notExecuted]<br>
[--session/-s <session_id>]</code></td>
<td>Create a subplan derived from previous session; this option generates
a subplan that can be used to run a subset of tests.<br><br> The only
required option is <code>--session</code>. Others are optional but, when
included, must be followed by a value. The
<code>--result-type</code> option is repeatable; for example
<code>add subplan --session 0 --result-type passed --result-type
failed</code> is valid.</td>
</tr>
</tbody>
</table>
</body>
</html>
|
