1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
|
<html devsite>
<head>
<title>Camera ITS-in-a-Box</title>
<meta name="project_path" value="/_project.yaml" />
<meta name="book_path" value="/_book.yaml" />
</head>
<body>
<!--
Copyright 2017 The Android Open Source Project
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<p>Android Camera Image Test Suite (ITS) is part of Android Compatibility Test
Suite (CTS) Verifier and includes tests that verify image content. As of CTS
7.0_r8, CTS Verifier supports ITS test automation via Camera ITS-in-a-box;
support for manual tests continues to ensure coverage for all Android device
form factors.</p>
<p>ITS-in-a-box brings the following benefits:</p>
<ul>
<li><strong>Automation</strong>. No human intervention is needed during the
test.</li>
<li><strong>Reduced testing time</strong>. Parallel testing of front/back camera
cuts
testing cycle time by 50%.</li>
<li><strong>Easier troubleshooting</strong>. Consistency of test environment
leads to fewer setup errors and increases reproducibility.</li>
<li><strong>Efficiency</strong>. Ability to retry for individual Camera/Scene
improves test execution efficiency.</li>
</ul>
<h2 id=get-started>Getting started</h2>
<p>ITS-in-a-box consists of a plastic box that is laser cut from computer-aided
design (CAD) drawings, a chart tablet, and a device under test (DUT). You can
use the wide field of view (WFoV) ITS-in-a-box, which is capable
of testing both WFoV (FoV > 90 degrees) and RFoV (FoV < 90 degrees) cameras, or
the regular field of view (RFoV) ITS-in-a-box.</p>
<p>To get started with the Camera ITS-in-a-box:</p>
<ol>
<li>Purchase or build a <a
href="/compatibility/cts/camera-wfov-box-assembly">wide field of view
(WFoV)</a> or <a href="/compatibility/cts/camera-its-box-assembly">regular field
of view (RFoV)</a> ITS-in-a-box.</li>
<li><a href="#configure-tablet">Configure a tablet</a> with Camera ITS
software.</li>
<li><a href="#run-tests">Run tests</a>.</li>
<li><a href="#get-results">Get results</a> from the DUT.</li>
</ol>
<h2 id=configure-tablet>Configuring the tablet</h2>
<p>This section provides step-by-step instructions for setting up a
tablet for use with the CameraITS software. These instructions use a Pixel C as
an example tablet. For information on tablet
requirements and recommendations, see <a href="#tablet-requirements">Tablet
requirements</a>.</p>
<p class="note"><strong>Note:</strong> The CameraITS python scripts
automatically set the following options on the tablet for you:
<br><em>Settings > Display > Sleep > After 30 minutes of inactivity</em>
<br><em>Adaptive brightness > OFF</em>
</p>
<ol>
<li>Charge the tablet and power it on. If prompted to set up an account, skip it
(CameraITS does not require any account paired with the tablet).</li>
<li>Update the tablet to Android 7.0 or higher. Android 6.x and lower versions
do not support CameraITS.</li>
<li>Enable <a href="https://developer.android.com/studio/debug/dev-options#enable"
class="external">developer mode</a>.</li>
<li>Return to <em>Settings</em> and select <strong>Developer options</strong>.
<table>
<tr>
<th>Enable options</th>
<td>
<ul>
<li>On</li>
<li>Stay awake</li>
<li>USB debugging (This allows the host to run the tablet in debug mode. When you
connect the tablet to the host for the first time, the tablet prompts you to
"Allow USB debugging?" If the tablet does not display the debug prompt,
disconnect then reconnect tablet.)</li>
</ul>
</td>
</tr>
<tr>
<th>Disable options</th>
<td>
<ul>
<li>Automatic system updates</li>
<li>Verify apps over USB</li>
</ul>
</td>
</tr>
</table>
</li>
<li>Determine DUT and chart IDs by running <code>$ adb devices</code> to list
available devices. To determine <code>device_id</code> and
<code>chart_id</code>, plug and unplug devices and observe the devices that
connect and disconnect.</li>
<li>Perform three test runs to suppress hints and user prompts that can obscure
charts on the tablet screen.
<ol style=list-style-lower-alpha>
<li>Position the tablet face-up on a table (do not attach the tablet to the back
panel of the box)</li>
<li>Run the following command:
<pre class="devsite-terminal devsite-click-to-copy">
python tools/run_all_tests.py device=$device_id camera=0 chart=$chart_id scenes=2,3
</pre>
Scenes 2 and 3 require the tablet to display an image, so the tablet prompts you
to "Allow Drive to access photos, media, and files on your device?". Clear this
prompt (and prevent future prompts) by pressing <strong>Allow</strong>.</li>
<li>Run the command again. The tablet prompts you to "Keep a copy of this file?"
and suggests Google Drive. Clear this prompt (and prevent future prompts) by
pressing the Drive icon then <strong>Cancel</strong> for upload to drive.</li>
<li>Finally, run <code>tools/run_all_tests.py</code> and confirm that scenes
change automatically as script cycles through different scenes. While most
tests will FAIL (as the camera is not pointed at the chart), you can verify the
tablet correctly cycles through the scenes without displaying any prompts or
other pop-ups on the screen.</li></ol></li></ol>
<h2 id=run-tests>Running tests</h2>
<p>Before running the ITS-in-a-box, ensure your test setup includes the following
hardware and software:</p>
<ul>
<li>One (1) ITS-in-a-box</li>
<li>One (1) Pixel C for displaying Scenes, S/N: 5811000011</li>
<li>Two (2) DUTs that use the same build fingerprint and have the CTS Verifier
7.0_8+ application installed. Example DUTs:
<ul>
<li>One (1) Pixel NOF26W for the back camera(0) testing, S/N: FA6BM0305016. To
install the CTS Verifier app, unzip android-cts-verifier.zip then run
<pre class="devsite-terminal devsite-click-to-copy">
adb -s FA6BM0305016 install -r android-cts-verifier/CtsVerifier.apk
</pre></li>
<li>One (1) Pixel NOF26W for the front camera(1) testing, S/N: FA6BM0305439. To
install the CTS Verifier app, unzip android-cts-verifier.zip then run
<pre class="devsite-terminal devsite-click-to-copy">
adb -s FA6BM0305439 install -r android-cts-verifier/CtsVerifier.apk
</pre>
</li>
</ul>
</li></ul>
<h3 id=scenes-0-4>Running scenes 0-4</h3>
<p>To run scenes 0-4 on the front and back camera in parallel:</p>
<pre class="devsite-click-to-copy">
<code class="devsite-terminal">cd android-cts-verifier/CameraITS</code>
<code class="devsite-terminal">. build/envsetup.sh</code>
<code class="devsite-terminal">python tools/run_parallel_tests.py device0=FA6BM0305016 device1=FA6BM0305439 chart=5811000011</code>
</pre>
<p>Examples:</p>
<table>
<tr>
<td ><img src=/compatibility/cts/images/camera_its_cam0.png align="center"></td>
<td align="center"><img src=/compatibility/cts/images/camera_its_cam0.png></td>
</tr>
<tr>
<td align="center"><p class=caption><strong>Figure 1</strong>. Camera 0 S/N:
FA6BM0305016</p>
</td>
<td align="center"><p class=caption><strong>Figure 2</strong>. Camera 1 S/N:
FA6BM0305439</p>
</td>
</tr>
</table>
<h3 id=retry-scenes>Retrying scenes</h3>
<p>You can retry scenes for both front and back cameras or a single camera:
<ul>
<li>To retry scenes on front and back cameras in parallel:
<pre class="devsite-terminal devsite-click-to-copy">
python tools/run_parallel_tests.py device0=FA6BM0305016 device1=FA6BM0305439 chart=5811000011 scenes=3,4
</pre>
</li>
<li>To retry scenes on a single camera:
<pre class="devsite-terminal devsite-click-to-copy">
python tools/run_all_tests.py device=FA6BM0305016 chart=5811000011 camera=1 scenes=3,4
</pre>
</li>
</ul>
<h3 id=scenes-0-4>Running scene 5</h3>
<p>Scene 5 requires special setup with specific lighting (for details, refer to
CameraITS.pdf in CTS Verifier, which you can download at <a href="/compatibility/cts/downloads">Compatibility Test Suite Downloads</a>).
You can run Scene 5 separately
(outside of the box) to test two devices in parallel.</p>
<ul>
<li>To run Scene 5 on front and back cameras on two devices in parallel:
<pre class="devsite-terminal devsite-click-to-copy">
python tools/run_parallel_tests.py device0=FA6BM0305016 device1=FA6BM0305439 chart=5811000011 scenes=5
</pre>
<br><img src=/compatibility/cts/images/camera_its_scene5.png width="50%"><br>
<strong>Figure 3</strong>. Camera scene 5.</li>
<li>To run Scene 5 for front and back cameras on a single device:
<pre class="devsite-click-to-copy">
<code class="devsite-terminal">python tools/run_all_tests.py device=FA6BM0305016 camera=0 scenes=5</code>
<code class="devsite-terminal">python tools/run_all_tests.py device=FA6BM0305016 camera=1 scenes=5</code>
</pre></li>
</ul>
<h2 id=get-results>Getting results</h2>
<p>You can view results during testing and save the completed results as a report.
</p>
<ul>
<li><strong>View progress of running tests</strong>. The command
<code>run_parallel_tests</code> prints results only after Camera-Scene tests
have finished, so to view results during test execution you must use Android
Device Monitor or <code>adb logcat</code> to verify progress and/or view
screenshots.<br>
<br>Example adb command:
<pre class="devsite-terminal devsite-click-to-copy">
adb -s FA6BM0305016 logcat -v time
</pre>
Example screenshots command:
<pre class="devsite-click-to-copy">
<code class="devsite-terminal">adb -s FA6BM0305016 shell screencap -p /sdcard/screencap.png</code>
<code class="devsite-terminal">adb -s FA6BM0305016 pull /sdcard/screencap.png</code>
<code class="devsite-terminal">display ./screencap.png</code>
</pre></li>
<li><strong>View results</strong>. To save Camera ITS results as a report:
<ol>
<li>Press <strong>Pass</strong> and save the report:
<br><img src=/compatibility/cts/images/camera_its_results.png width="50%"><br>
<strong>Figure 4</strong>. Camera ITS report.</li>
<li>Pull reports from the device:
<pre class="devsite-terminal devsite-click-to-copy">
adb -s FA6BM0305016 pull /sdcard/verifierReports
</pre>
</li>
<li>Unzip the report file and view the test_result.xml.
<br><img src=/compatibility/cts/images/camera_its_reports.png><br>
<strong>Figure 5</strong>. Camera ITS reports.<br></li>
</ol>
</li>
</ul>
<h2 id=tablet-requirements>Tablet requirements</h2>
<p>Tablets must have a display size of around 10 inches with a screen resolution
greater than 2000 x 1500 pixels. The <code>DISPLAY_LEVEL</code> value must be
set in
<code>CameraITS/tools/wake_up_screen.py</code> according to the tablet model.
The table below lists the values for recommended tablets.</p>
<p>Recommended tablets for ITS testing are shown below:</p>
<table>
<tr>
<th>Device</th>
<th>Display size<br>(inches)</th>
<th>Display size<br>(pixels)</th>
<th>Tablet dimensions<br>(inches)</th>
<th>DISPLAY_LEVEL</th>
<th>OS</th>
</tr>
<tr>
<td>Asus Zen Pad 3</td>
<td>9.7</td>
<td>2048 x 1536</td>
<td>9.47 x 6.44 x 0.28</td>
<td>192</td>
<td>Android 6.0</td>
</tr>
<tr>
<td>Huawei Mediapad m5</td>
<td>10.8</td>
<td>2560 x 1600</td>
<td>10.18 x 6.76 x 0.29</td>
<td>192</td>
<td>Android 8.0</td>
</tr>
<tr>
<td>Pixel C</td>
<td>10.2</td>
<td>2560 x 1800</td>
<td>9.53 x 7.05 x 0.28</td>
<td>96</td>
<td>Android 8.0</td>
</tr>
<tr>
<td>Samsung S3</td>
<td>9.7</td>
<td>2048 x 1536</td>
<td>10.76 x 6.65 x 0.24</td>
<td>192</td>
<td>Android 7.0</td>
</tr>
<tr>
<td>Sony Xperia Z4</td>
<td>10.1</td>
<td>2560 x 1600</td>
<td>10 x 6.57 x 0.24</td>
<td>192</td>
<td>Android 7.0</td>
</tr>
</table>
</body>
</html>
|
