1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
|
<html devsite>
<head>
<title>HAL subsystem</title>
<meta name="project_path" value="/_project.yaml" />
<meta name="book_path" value="/_book.yaml" />
</head>
<body>
<!--
Copyright 2017 The Android Open Source Project
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<h2 id="requests">Requests</h2>
<p> The app framework issues requests for captured results to the camera subsystem.
One request corresponds to one set of results. A request encapsulates all
configuration information about the capturing and processing of those results.
This includes things such as resolution and pixel format; manual sensor, lens,
and flash control; 3A operating modes; RAW to YUV processing control; and
statistics generation. This allows for much more control over the results'
output and processing. Multiple requests can be in flight at once, and
submitting requests is non-blocking. And the requests are always processed in
the order they are received.</p>
<img src="images/camera_model.png" alt="Camera request model" id="figure1" />
<p class="img-caption">
<strong>Figure 1.</strong> Camera model
</p>
<h2 id="hal-subsystem">The HAL and camera subsystem</h2>
<p> The camera subsystem includes the implementations for components in the camera
pipeline such as the 3A algorithm and processing controls. The camera HAL
provides interfaces for you to implement your versions of these components. To
maintain cross-platform compatibility between multiple device manufacturers and
Image Signal Processor (ISP, or camera sensor) vendors, the camera pipeline
model is virtual and does not directly correspond to any real ISP. However, it
is similar enough to real processing pipelines so that you can map it to your
hardware efficiently. In addition, it is abstract enough to allow for multiple
different algorithms and orders of operation without compromising either
quality, efficiency, or cross-device compatibility.</p>
<p>The camera pipeline also supports triggers that the app framework can initiate
to turn on things such as auto-focus. It also sends notifications back to the
app framework, notifying apps of events such as an auto-focus lock or errors.</p>
<img src="images/camera_hal.png" alt="Camera hardware abstraction layer" id="figure2" />
<p class="img-caption">
<strong>Figure 2.</strong> Camera pipeline</p>
<p>Please note, some image processing blocks shown in the diagram above are not
well-defined in the initial release. The camera pipeline makes the following
assumptions:</p>
<ul>
<li>RAW Bayer output undergoes no processing inside the ISP.</li>
<li>Statistics are generated based off the raw sensor data.</li>
<li>The various processing blocks that convert raw sensor data to YUV are in an
arbitrary order.</li>
<li>While multiple scale and crop units are shown, all scaler units share the
output region controls (digital zoom). However, each unit may have a different
output resolution and pixel format.</li>
</ul>
<p><strong>Summary of API use</strong><br/>
This is a brief summary of the steps for using the Android camera API. See the
Startup and expected operation sequence section for a detailed breakdown of
these steps, including API calls.</p>
<ol>
<li>Listen for and enumerate camera devices.</li>
<li>Open device and connect listeners.</li>
<li>Configure outputs for target use case (such as still capture, recording,
etc.).</li>
<li>Create request(s) for target use case.</li>
<li>Capture/repeat requests and bursts.</li>
<li>Receive result metadata and image data.</li>
<li>When switching use cases, return to step 3.</li>
</ol>
<p><strong>HAL operation summary</strong></p>
<ul>
<li>Asynchronous requests for captures come from the framework.</li>
<li>HAL device must process requests in order. And for each request, produce
output result metadata, and one or more output image buffers.</li>
<li>First-in, first-out for requests and results, and for streams referenced by
subsequent requests. </li>
<li>Timestamps must be identical for all outputs from a given request, so that the
framework can match them together if needed. </li>
<li>All capture configuration and state (except for the 3A routines) is
encapsulated in the requests and results.</li>
</ul>
<img src="images/camera-hal-overview.png" alt="Camera HAL overview" id="figure3" />
<p class="img-caption">
<strong>Figure 3.</strong> Camera HAL overview
</p>
<h2 id="startup">Startup and expected operation sequence</h2>
<p>This section contains a detailed explanation of the steps expected when using
the camera API. Please see <a href="https://android.googlesource.com/platform/hardware/interfaces/+/master/camera/">platform/hardware/interfaces/camera/</a> for HIDL interface
definitions.</p>
<h3 id="open-camera-device">Enumerating, opening camera devices and
creating an active session</h3>
<ol>
<li>After initialization, the framework starts listening for any present
camera providers that implement the
<code><a href="https://android.googlesource.com/platform/hardware/interfaces/+/master/camera/provider/2.4/ICameraProvider.hal">ICameraProvider</a></code> interface. If such provider or
providers are present, the framework will try to establish a connection.</li>
<li>The framework enumerates the camera devices via
<code>ICameraProvider::getCameraIdList()</code>.</li>
<li>The framework instantiates a new <code>ICameraDevice</code> by calling the respective
<code>ICameraProvider::getCameraDeviceInterface_VX_X()</code>.</li>
<li>The framework calls <code>ICameraDevice::open()</code> to create a new
active capture session ICameraDeviceSession.</li>
</ol>
<h3 id="use-active-session">Using an active camera session</h3>
<ol>
<li>The framework calls <code>ICameraDeviceSession::configureStreams()</code>
with a list of input/output streams to the HAL device.</li>
<li>The framework requests default settings for some use cases with
calls to <code>ICameraDeviceSession::constructDefaultRequestSettings()</code>.
This may occur at any time after the <code>ICameraDeviceSession</code> is
created by <code>ICameraDevice::open</code>.
</li>
<li>The framework constructs and sends the first capture request to the HAL with
settings based on one of the sets of default settings, and with at least one
output stream that has been registered earlier by the framework. This is sent
to the HAL with <code>ICameraDeviceSession::processCaptureRequest()</code>.
The HAL must block the return of this call until it is ready for the next
request to be sent.</li>
<li>The framework continues to submit requests and calls
<code>ICameraDeviceSession::constructDefaultRequestSettings()</code> to get
default settings buffers for other use cases as necessary.</li>
<li>When the capture of a request begins (sensor starts exposing for the
capture), the HAL calls <code>ICameraDeviceCallback::notify()</code> with
the SHUTTER message, including the frame number and the timestamp for start
of exposure. This notify callback does not have to happen before the first
<code>processCaptureResult()</code> call for a request, but no results are
delivered to an application for a capture until after
<code>notify()</code> for that capture is called.
</li>
<li>After some pipeline delay, the HAL begins to return completed captures to
the framework with <code>ICameraDeviceCallback::processCaptureResult()</code>.
These are returned in the same order as the requests were submitted. Multiple
requests can be in flight at once, depending on the pipeline depth of the
camera HAL device.</li>
</ol>
<p>After some time, one of the following will occur:</p>
<ul>
<li>The framework may stop submitting new requests, wait for
the existing captures to complete (all buffers filled, all results
returned), and then call <code>ICameraDeviceSession::configureStreams()</code>
again. This resets the camera hardware and pipeline for a new set of
input/output streams. Some streams may be reused from the previous
configuration. The framework then continues from the first capture request
to the HAL, if at least one
registered output stream remains. (Otherwise,
<code>ICameraDeviceSession::configureStreams()</code> is required first.)</li>
<li>The framework may call <code>ICameraDeviceSession::close()</code>
to end the camera session. This may be called at any time when no other calls
from the framework are active, although the call may block until all
in-flight captures have completed (all results returned, all buffers
filled). After the <code>close()</code> call returns, no more calls to
<code>ICameraDeviceCallback</code> are allowed from the HAL. Once the
<code>close()</code> call is underway, the framework may not call any other
HAL device functions.</li>
<li>In case of an error or other asynchronous event, the HAL must call
<code>ICameraDeviceCallback::notify()</code> with the appropriate
error/event message.
After returning from a fatal device-wide error notification, the HAL should
act as if <code>close()</code> had been called on it. However, the HAL must
either cancel
or complete all outstanding captures before calling <code>notify()</code>,
so that once
<code>notify()</code> is called with a fatal error, the framework will not
receive further callbacks from the device. Methods besides
<code>close()</code> should return
-ENODEV or NULL after the <code>notify()</code> method returns from a fatal
error message.</li>
</ul>
<img src="images/camera-ops-flow.png" alt="Camera operations flow" id="figure4" width="485"/>
<p class="img-caption">
<strong>Figure 4.</strong> Camera operational flow
</p>
<h2 id="hardware-levels">Hardware levels</h2>
<p>Camera devices can implement several hardware levels depending on their
capabilities. For more information, see
<a href="https://developer.android.com/reference/android/hardware/camera2/CameraCharacteristics#INFO_SUPPORTED_HARDWARE_LEVEL">supported hardware level</a>.</p>
<h2 id="interaction">Interaction between the application capture request, 3A
control, and the processing pipeline</h2>
<p>Depending on the settings in the 3A control block, the camera pipeline ignores
some of the parameters in the application's capture request and uses the values
provided by the 3A control routines instead. For example, when auto-exposure is
active, the exposure time, frame duration, and sensitivity parameters of the
sensor are controlled by the platform 3A algorithm, and any app-specified values
are ignored. The values chosen for the frame by the 3A routines must be reported
in the output metadata. The following table describes the different modes of the
3A control block and the properties that are controlled by these modes. See
the <a href="https://android.googlesource.com/platform/system/media/+/master/camera/docs/docs.html">platform/system/media/camera/docs/docs.html</a> file for definitions of these properties.</p>
<table>
<tr>
<th>Parameter</th>
<th>State</th>
<th>Properties controlled</th>
</tr>
<tr>
<td>android.control.aeMode</td>
<td>OFF</td>
<td>None</td>
</tr>
<tr>
<td></td>
<td>ON</td>
<td>android.sensor.exposureTime
android.sensor.frameDuration
android.sensor.sensitivity
android.lens.aperture (if supported)
android.lens.filterDensity (if supported)</td>
</tr>
<tr>
<td></td>
<td>ON_AUTO_FLASH</td>
<td>Everything is ON, plus android.flash.firingPower, android.flash.firingTime, and android.flash.mode</td>
</tr>
<tr>
<td></td>
<td>ON_ALWAYS_FLASH</td>
<td>Same as ON_AUTO_FLASH</td>
</tr>
<tr>
<td></td>
<td>ON_AUTO_FLASH_RED_EYE</td>
<td>Same as ON_AUTO_FLASH</td>
</tr>
<tr>
<td>android.control.awbMode</td>
<td>OFF</td>
<td>None</td>
</tr>
<tr>
<td></td>
<td>WHITE_BALANCE_*</td>
<td>android.colorCorrection.transform. Platform-specific adjustments if android.colorCorrection.mode is FAST or HIGH_QUALITY.</td>
</tr>
<tr>
<td>android.control.afMode</td>
<td>OFF</td>
<td>None</td>
</tr>
<tr>
<td></td>
<td>FOCUS_MODE_*</td>
<td>android.lens.focusDistance</td>
</tr>
<tr>
<td>android.control.videoStabilization</td>
<td>OFF</td>
<td>None</td>
</tr>
<tr>
<td></td>
<td>ON</td>
<td>Can adjust android.scaler.cropRegion to implement video stabilization</td>
</tr>
<tr>
<td>android.control.mode</td>
<td>OFF</td>
<td>AE, AWB, and AF are disabled</td>
</tr>
<tr>
<td></td>
<td>AUTO</td>
<td>Individual AE, AWB, and AF settings are used</td>
</tr>
<tr>
<td></td>
<td>SCENE_MODE_*</td>
<td>Can override all parameters listed above. Individual 3A controls are disabled.</td>
</tr>
</table>
<p>The controls in the Image Processing block in Figure 2 all operate on a
similar principle, and generally each block has three modes:</p>
<ul>
<li>OFF: This processing block is disabled. The demosaic, color correction, and
tone curve adjustment blocks cannot be disabled.</li>
<li>FAST: In this mode, the processing block may not slow down the output frame
rate compared to OFF mode, but should otherwise produce the best-quality
output it can given that restriction. Typically, this would be used for
preview or video recording modes, or burst capture for still images. On some
devices, this may be equivalent to OFF mode (no processing can be done without
slowing down the frame rate), and on some devices, this may be equivalent to
HIGH_QUALITY mode (best quality still does not slow down frame rate).</li>
<li>HIGH_QUALITY: In this mode, the processing block should produce the best
quality result possible, slowing down the output frame rate as needed.
Typically, this would be used for high-quality still capture. Some blocks
include a manual control which can be optionally selected instead of FAST or
HIGH_QUALITY. For example, the color correction block supports a color
transform matrix, while the tone curve adjustment supports an arbitrary global
tone mapping curve.</li>
</ul>
<p>The maximum frame rate that can be supported by a camera subsystem is a function
of many factors:</p>
<ul>
<li>Requested resolutions of output image streams</li>
<li>Availability of binning/skipping modes on the imager</li>
<li>The bandwidth of the imager interface</li>
<li>The bandwidth of the various ISP processing blocks</li>
</ul>
<p>Since these factors can vary greatly between different ISPs and sensors, the
camera HAL interface tries to abstract the bandwidth restrictions into as simple
model as possible. The model presented has the following characteristics:</p>
<ul>
<li>The image sensor is always configured to output the smallest resolution
possible given the application's requested output stream sizes. The smallest
resolution is defined as being at least as large as the largest requested
output stream size.</li>
<li>Since any request may use any or all the currently configured output streams,
the sensor and ISP must be configured to support scaling a single capture to
all the streams at the same time. </li>
<li>JPEG streams act like processed YUV streams for requests for which they are
not included; in requests in which they are directly referenced, they act as
JPEG streams.</li>
<li>The JPEG processor can run concurrently to the rest of the camera pipeline but
cannot process more than one capture at a time.</li>
</ul>
</body>
</html>
|
