diff options
Diffstat (limited to 'en/compatibility/cts/run.html')
| -rw-r--r-- | en/compatibility/cts/run.html | 315 |
1 files changed, 174 insertions, 141 deletions
diff --git a/en/compatibility/cts/run.html b/en/compatibility/cts/run.html index 0bd66d90..0caf4c7e 100644 --- a/en/compatibility/cts/run.html +++ b/en/compatibility/cts/run.html @@ -45,156 +45,27 @@ short) continuous test framework.</p> <li>Start the default test plan (contains all test packages) by appending: <code>run cts --plan CTS</code> . This kicks off all CTS tests required for compatibility. <ul> + <li>For CTS v2 (Android 7.0 and later), enter <code>list modules</code> to + see a list of test modules. <li>For CTS v1 (Android 6.0 and earlier), enter <code>list plans</code> to view a list of test plans in the repository or <code>list package</code>s to view a list of test packages in the repository. - <li>For CTS v2 (Android 7.0 and later), enter <code>list modules</code> to - see a list of test modules. </ul> </li> <li>Alternately, run the CTS plan of your choosing from the command line using: <code>cts-tradefed run cts --plan <plan_name> </code> -<p class="note"><strong>Note:</strong> When running Android 6.0 (Marshmallow) -CTS only, we recommend you use the <code>--skip-preconditions</code> option to -skip the experimental pre-conditions feature that may cause issues for when -executing CTS tests.</p> +<p class="note"><strong>Note:</strong> You may save run time in Android 7.0 (Nougat) and higher, by + using the <code>run cts-dev</code> command instead of <code>run cts</code>. This command skips + preconditions, device-information collection, and all system status checkers. It also runs the + tests on only a single ABI. For device validation, avoid this optimization and +include all preconditions and checks.</p> <li>View test progress and results reported on the console. <li>If your device is Android 5.0 or later and declares support for an ARM and a x86 ABI, you should run both the ARM and x86 CTS packages. </ol> -<h2 id=using-cts-v1>Using the CTS v1 console</h2> - -<p>For Android 6.0 or earlier, you'll use CTS v1.</p> -<h3 id=selecting_cts_plans>Selecting plans</h3> -<p>The following test plans are available:</p> -<ul> - <li><em>CTS</em>—all tests required for compatibility. </li> - <li><em>Signature</em>—the signature verification of all public APIs </li> - <li><em>Android</em>—tests for the Android APIs </li> - <li><em>Java</em>—tests for the Java core library </li> - <li><em>VM</em>—tests for ART or Dalvik </li> - <li><em>Performance</em>—performance tests for your implementation </li> -</ul> -<p>These can be executed with the <code>run cts</code> command.</p> -<h3 id=cts_reference>CTS v1 console command reference</h3> - -<p class="table-caption" id="console-commands"> - <strong>Table 1.</strong> This table summarizes the CTS v1 console commands for -various uses.</p> -<table> - <tbody> - <tr> - <th>Host</th> - <th>Description</th> - </tr> - <tr> - <td><code>help</code></td> - <td>Display a summary of the most commonly used commands</td> - </tr> - <tr> - <td><code>help all</code></td> - <td>Display the complete list of available commands</td> - </tr> - <tr> - <td><code>exit</code></td> - <td>Gracefully exit the CTS console. Console will close when all currently running tests are finished</td> - </tr> - <tr> - <th>Run</th> - <th>Description</th> - </tr> - <tr> - <td><code>run cts</code></td> - <td>Run the specified tests and displays progress information. One of -<code>--plan</code>, <code>--package</code>, <code>--class</code> or -<code>--continue-session</code> needs to be specified - <p>The CTS console can accept other commands while tests are in progress </p> - <p>If no devices are connected, the CTS desktop machine (or host) will wait for a device to be connected before starting tests </p> - <p>If more than one device is connected, the CTS host will choose a device automatically</p></td> - </tr> - <tr> - <td><code>--plan <test_plan_name></code></td> - <td>Run the specified test plan</td> - </tr> - <tr> - <td><code>--package/-p <test_package_name> [--package/-p <test_package2>...]</code></td> - <td>Run the specified test packages</td> - </tr> - <tr> - <td><code>--class/-c <class_name> [--method/-m <test_method_name></code></td> - <td>Run the specified test class and/or method</td> - </tr> - <tr> - <td><code>--continue-session</code></td> - <td>Run all not executed tests from previous CTS session; the sessions testResult.xml will be updated with the new results</td> - </tr> - <tr> - <td><code>--shards <number_of_shards></code></td> - <td>Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel</td> - </tr> - <tr> - <td><code>--serial/-s <deviceID></code></td> - <td>Run CTS on the specific device</td> - </tr> - <tr> - <td><code>-t <class_name>#<test_method_name></code></td> - <td>Run a specific test method</td> - </tr> - <tr> - <td><code>--force-abi 32|64</code></td> - <td>On 64-bit devices, run the test against only the 32-bit or 64-bit ABI</td> - </tr> - <tr> - <th>List</th> - <th>Description</th> - </tr> - <tr> - <td><code>list packages</code></td> - <td>List all available test packages in the repository</td> - </tr> - <tr> - <td><code>list plans</code></td> - <td>List all available test plans in the repository</td> - </tr> - <tr> - <td><code>list invocations</code></td> - <td>List 'run' commands currently being executed on devices</td> - </tr> - <tr> - <td><code>list commands</code></td> - <td>List all 'run' commands currently in the queue waiting to be assigned to devices</td> - </tr> - <tr> - <td><code>list results</code></td> - <td>List CTS results currently stored in repository</td> - </tr> - <tr> - <td><code>list devices</code></td> - <td>List currently connected devices and their state - <p> </p> - <p>'Available' devices are functioning, idle devices, available for running tests</p> - <p> </p> - <p>'Unavailable' devices are devices visible via adb, but are not responding to adb commands and won't be allocated for tests</p> - <p> </p> - <p>'Allocated' devices are devices currently running tests</td> - </tr> - <tr> - <th>Add</th> - <th>Description</th> - </tr> - <tr> - <td><code>add derivedplan --plan <plan_name><br> - --result/-r<br> - [pass | fail | timeout | notExecuted]<br> - [--session/-s <session_id>]</code></td> - <td>Create a plan derived from given result session; use this option to rerun reports and validate test issues</td> - </tr> - </tbody> -</table> - <h2 id=using-cts-v2>Using the CTS v2 console</h2> <p>For Android 7.0 or later, you'll use CTS v2.</p> @@ -261,11 +132,32 @@ various uses.</p> </tr> <tr> <td><code>run cts</code></td> - <td><p>Run the default CTS plan (that is, the full CTS invocation).</p> - <p>The CTS console can accept other commands while tests are in progress.</p> + <td><p>Run the default CTS plan (that is, the full CTS invocation). Use this + comprehensive option (including preconditions) for device validation. + See <a + href="https://android.googlesource.com/platform/test/suite_harness/+/master/tools/cts-tradefed/res/config/cts.xml">/test/suite_harness/tools/cts-tradefed/res/config/cts.xml</a> + for inclusions.</p> + <p>The CTS console can accept other commands while tests are in progress.</p> + <p>If no devices are connected, the CTS desktop machine (or host) will wait + for a device to be connected before starting tests. If more than one + device is connected, the CTS host will choose a device + automatically.</p> + </tr> + <td><code>run cts-dev</code></td> + <td><p>Run the default CTS plan (that is, the full CTS invocation) but + skip preconditions to save run time for iterative development of a new + test. This bypasses verification and setup of the device's + configuration, such as pushing media files or checking for Wi-Fi + connection, as is done when the <code>--skip-preconditions</code> option is used. This + command also skips device-information collection, and all system status checkers. It also + runs the tests on only a single ABI. For device validation, avoid this optimization and + include all preconditions and checks. + See <a href="https://android.googlesource.com/platform/test/suite_harness/+/master/tools/cts-tradefed/res/config/cts-dev.xml">/test/suite_harness/tools/cts-tradefed/res/config/cts-dev.xml</a> for exclusions.</p> + <p>The CTS console can accept other commands while tests are in progress.</p> <p>If no devices are connected, the CTS desktop machine (or host) will wait - for a device to be connected before starting tests.</p> - <p>If more than one device is connected, the CTS host will choose a device automatically.</p></td> + for a device to be connected before starting tests. If more than one + device is connected, the CTS host will choose a device + automatically.</p></td> </tr> <tr> <td><code>run retry</code></td> @@ -347,7 +239,10 @@ various uses.</p> </tr> <tr> <td><code>--skip-preconditions</code></td> - <td>Bypasses verification and setup of the device's configuration, such as pushing media files or checking for Wi-Fi connection.</td> + <td>Skip preconditions to save run time for iterative development of a + new test. This bypasses verification and setup of the device's + configuration, such as pushing media files or checking for Wi-Fi + connection.</td> </tr> <tr> <th>List</th> @@ -416,5 +311,143 @@ various uses.</p> </tbody> </table> +<h2 id=using-cts-v1>Using the CTS v1 console</h2> + +<p>For Android 6.0 or earlier, you'll use CTS v1.</p> +<h3 id=selecting_cts_plans>Selecting plans</h3> +<p>The following test plans are available:</p> +<ul> + <li><em>CTS</em>—all tests required for compatibility. </li> + <li><em>Signature</em>—the signature verification of all public APIs </li> + <li><em>Android</em>—tests for the Android APIs </li> + <li><em>Java</em>—tests for the Java core library </li> + <li><em>VM</em>—tests for ART or Dalvik </li> + <li><em>Performance</em>—performance tests for your implementation </li> +</ul> +<p>These can be executed with the <code>run cts</code> command.</p> + +<h3 id=cts_reference>CTS v1 console command reference</h3> + +<p class="table-caption" id="console-commands"> + <strong>Table 1.</strong> This table summarizes the CTS v1 console commands for +various uses.</p> +<table> + <tbody> + <tr> + <th>Host</th> + <th>Description</th> + </tr> + <tr> + <td><code>help</code></td> + <td>Display a summary of the most commonly used commands</td> + </tr> + <tr> + <td><code>help all</code></td> + <td>Display the complete list of available commands</td> + </tr> + <tr> + <td><code>exit</code></td> + <td>Gracefully exit the CTS console. Console will close when all currently running tests are finished</td> + </tr> + <tr> + <th>Run</th> + <th>Description</th> + </tr> + <tr> + <td><code>run cts</code></td> + <td>Run the specified tests and displays progress information. One of +<code>--plan</code>, <code>--package</code>, <code>--class</code> or +<code>--continue-session</code> needs to be specified + <p>The CTS console can accept other commands while tests are in progress </p> + <p>If no devices are connected, the CTS desktop machine (or host) will wait for a device to be connected before starting tests </p> + <p>If more than one device is connected, the CTS host will choose a device automatically</p></td> + </tr> + <tr> + <td><code>--plan <test_plan_name></code></td> + <td>Run the specified test plan</td> + </tr> + <tr> + <td><code>--package/-p <test_package_name> [--package/-p <test_package2>...]</code></td> + <td>Run the specified test packages</td> + </tr> + <tr> + <td><code>--class/-c <class_name> [--method/-m <test_method_name></code></td> + <td>Run the specified test class and/or method</td> + </tr> + <tr> + <td><code>--continue-session</code></td> + <td>Run all not executed tests from previous CTS session; the sessions testResult.xml will be updated with the new results</td> + </tr> + <tr> + <td><code>--shards <number_of_shards></code></td> + <td>Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel</td> + </tr> + <tr> + <td><code>--serial/-s <deviceID></code></td> + <td>Run CTS on the specific device</td> + </tr> + <tr> + <td><code>-t <class_name>#<test_method_name></code></td> + <td>Run a specific test method</td> + </tr> + <tr> + <td><code>--force-abi 32|64</code></td> + <td>On 64-bit devices, run the test against only the 32-bit or 64-bit ABI</td> + </tr> + <tr> + <td><code>--skip-preconditions</code></td> + <td>Skip preconditions to save run time for iterative development of a + new test. This bypasses verification and setup of the device's + configuration, such as pushing media files or checking for Wi-Fi + connection.</td> + </tr> + <tr> + <th>List</th> + <th>Description</th> + </tr> + <tr> + <td><code>list packages</code></td> + <td>List all available test packages in the repository</td> + </tr> + <tr> + <td><code>list plans</code></td> + <td>List all available test plans in the repository</td> + </tr> + <tr> + <td><code>list invocations</code></td> + <td>List 'run' commands currently being executed on devices</td> + </tr> + <tr> + <td><code>list commands</code></td> + <td>List all 'run' commands currently in the queue waiting to be assigned to devices</td> + </tr> + <tr> + <td><code>list results</code></td> + <td>List CTS results currently stored in repository</td> + </tr> + <tr> + <td><code>list devices</code></td> + <td>List currently connected devices and their state + <p> </p> + <p>'Available' devices are functioning, idle devices, available for running tests</p> + <p> </p> + <p>'Unavailable' devices are devices visible via adb, but are not responding to adb commands and won't be allocated for tests</p> + <p> </p> + <p>'Allocated' devices are devices currently running tests</td> + </tr> + <tr> + <th>Add</th> + <th>Description</th> + </tr> + <tr> + <td><code>add derivedplan --plan <plan_name><br> + --result/-r<br> + [pass | fail | timeout | notExecuted]<br> + [--session/-s <session_id>]</code></td> + <td>Create a plan derived from given result session; use this option to rerun reports and validate test issues</td> + </tr> + </tbody> +</table> + </body> </html> |