Android Camera Image Test Suite (ITS) is part of Android Compatibility Test Suite (CTS) Verifier and includes tests that verify image content. As of CTS 7.0_r8, CTS Verifier supports ITS test automation via Camera ITS-in-a-box; support for manual tests continues to ensure coverage for all Android device form factors.

ITS-in-a-box brings the following benefits:

• Automation. No human intervention is needed during the test.
• Reduced testing time. Parallel testing of front/back camera cuts testing cycle time by 50%.
• Easier troubleshooting. Consistency of test environment leads to fewer setup errors and increases reproducibility.
• Efficiency. Ability to retry for individual Camera/Scene improves test execution efficiency.

Getting started

ITS-in-a-box consists of a plastic box that is laser cut from computer-aided design (CAD) drawings, a chart tablet, and a device under test (DUT). You can use the wide field of view (WFoV) ITS-in-a-box, which is capable of testing both WFoV (FoV > 90 degrees) and RFoV (FoV < 90 degrees) cameras, or the regular field of view (RFoV) ITS-in-a-box.

To get started with the Camera ITS-in-a-box:

1. Purchase or build a wide field of view (WFoV) or regular field of view (RFoV) ITS-in-a-box.
2. Configure a tablet with Camera ITS software.
3. Run tests.
4. Get results from the DUT.

Configuring the tablet

This section provides step-by-step instructions for setting up a tablet for use with the CameraITS software. These instructions use a Pixel C as an example tablet. For information on tablet requirements and recommendations, see Tablet requirements.

Note: The CameraITS python scripts automatically set the following options on the tablet for you:
Settings > Display > Sleep > After 30 minutes of inactivity
Adaptive brightness > OFF

1. Charge the tablet and power it on. If prompted to set up an account, skip it (CameraITS does not require any account paired with the tablet).
2. Update the tablet to Android 7.0 or higher. Android 6.x and lower versions do not support CameraITS.
3. Enable developer mode.
4. Return to Settings and select Developer options.

   Enable options	On Stay awake USB debugging (This allows the host to run the tablet in debug mode. When you connect the tablet to the host for the first time, the tablet prompts you to "Allow USB debugging?" If the tablet does not display the debug prompt, disconnect then reconnect tablet.)
   Disable options	Automatic system updates Verify apps over USB

5. Determine DUT and chart IDs by running $ adb devices to list available devices. To determine device_id and chart_id, plug and unplug devices and observe the devices that connect and disconnect.
6. Perform three test runs to suppress hints and user prompts that can obscure charts on the tablet screen.
   1. Position the tablet face-up on a table (do not attach the tablet to the back panel of the box)
   2. Run the following command:

      python tools/run_all_tests.py device=$device_id camera=0 chart=$chart_id scenes=2,3
      

      Scenes 2 and 3 require the tablet to display an image, so the tablet prompts you to "Allow Drive to access photos, media, and files on your device?". Clear this prompt (and prevent future prompts) by pressing Allow.
   3. Run the command again. The tablet prompts you to "Keep a copy of this file?" and suggests Google Drive. Clear this prompt (and prevent future prompts) by pressing the Drive icon then Cancel for upload to drive.
   4. Finally, run tools/run_all_tests.py and confirm that scenes change automatically as script cycles through different scenes. While most tests will FAIL (as the camera is not pointed at the chart), you can verify the tablet correctly cycles through the scenes without displaying any prompts or other pop-ups on the screen.

Running tests

Before running the ITS-in-a-box, ensure your test setup includes the following hardware and software:

• One (1) ITS-in-a-box
• One (1) Pixel C for displaying Scenes, S/N: 5811000011
• Two (2) DUTs that use the same build fingerprint and have the CTS Verifier 7.0_8+ application installed. Example DUTs:
  • One (1) Pixel NOF26W for the back camera(0) testing, S/N: FA6BM0305016. To install the CTS Verifier app, unzip android-cts-verifier.zip then run

    adb -s FA6BM0305016 install -r android-cts-verifier/CtsVerifier.apk
    

  • One (1) Pixel NOF26W for the front camera(1) testing, S/N: FA6BM0305439. To install the CTS Verifier app, unzip android-cts-verifier.zip then run

    adb -s FA6BM0305439 install -r android-cts-verifier/CtsVerifier.apk
    

Running scenes 0-4

To run scenes 0-4 on the front and back camera in parallel:

cd android-cts-verifier/CameraITS
. build/envsetup.sh
python tools/run_parallel_tests.py device0=FA6BM0305016 device1=FA6BM0305439 chart=5811000011

Examples:

Figure 1. Camera 0 S/N: FA6BM0305016	Figure 2. Camera 1 S/N: FA6BM0305439

Retrying scenes

You can retry scenes for both front and back cameras or a single camera:

• To retry scenes on front and back cameras in parallel:

  python tools/run_parallel_tests.py device0=FA6BM0305016 device1=FA6BM0305439 chart=5811000011 scenes=3,4
  

• To retry scenes on a single camera:

  python tools/run_all_tests.py device=FA6BM0305016 chart=5811000011 camera=1 scenes=3,4
  

Running scene 5

Scene 5 requires special setup with specific lighting (for details, refer to CameraITS.pdf in CTS Verifier, which you can download at Compatibility Test Suite Downloads). You can run Scene 5 separately (outside of the box) to test two devices in parallel.

• To run Scene 5 on front and back cameras on two devices in parallel:

  python tools/run_parallel_tests.py device0=FA6BM0305016 device1=FA6BM0305439 chart=5811000011 scenes=5
  

  
  
  Figure 3. Camera scene 5.
• To run Scene 5 for front and back cameras on a single device:

  python tools/run_all_tests.py device=FA6BM0305016 camera=0 scenes=5
  python tools/run_all_tests.py device=FA6BM0305016 camera=1 scenes=5
  

Getting results

You can view results during testing and save the completed results as a report.

• View progress of running tests. The command run_parallel_tests prints results only after Camera-Scene tests have finished, so to view results during test execution you must use Android Device Monitor or adb logcat to verify progress and/or view screenshots.
  
  Example adb command:

  adb -s FA6BM0305016 logcat -v time
  

  Example screenshots command:

  adb -s FA6BM0305016 shell screencap -p /sdcard/screencap.png
  adb -s FA6BM0305016 pull /sdcard/screencap.png
  display ./screencap.png
  

• View results. To save Camera ITS results as a report:
  1. Press Pass and save the report:
     
     Figure 4. Camera ITS report.
  2. Pull reports from the device:

     adb -s FA6BM0305016 pull /sdcard/verifierReports
     

  3. Unzip the report file and view the test_result.xml.
     
     Figure 5. Camera ITS reports.
     

Tablet requirements

Tablets must have a display size of around 10 inches with a screen resolution greater than 2000 x 1500 pixels. The DISPLAY_LEVEL value must be set in CameraITS/tools/wake_up_screen.py according to the tablet model. The table below lists the values for recommended tablets.

Recommended tablets for ITS testing are shown below: