The Android framework offers a variety of graphics rendering APIs for 2D and 3D that interact with manufacturer implementations of graphics drivers, so it is important to have a good understanding of how those APIs work at a higher level. This page introduces the graphics hardware abstraction layer (HAL) upon which those drivers are built.
Application developers draw images to the screen in two ways: with Canvas or OpenGL. See System-level graphics architecture for a detailed description of Android graphics components.
android.graphics.Canvas is a 2D graphics API and is the most popular graphics API among developers. Canvas operations draw all the stock and custom android.view.Views in Android. In Android, hardware acceleration for Canvas APIs is accomplished with a drawing library called OpenGLRenderer that translates Canvas operations to OpenGL operations so they can execute on the GPU.
Beginning in Android 4.0, hardware-accelerated Canvas is enabled by default. Consequently, a hardware GPU that supports OpenGL ES 2.0 is mandatory for Android 4.0 and later devices. See the Hardware Acceleration guide for an explanation of how the hardware-accelerated drawing path works and the differences in its behavior from that of the software drawing path.
In addition to Canvas, the other main way that developers render graphics is by using OpenGL ES to directly render to a surface. Android provides OpenGL ES interfaces in the android.opengl package that developers can use to call into their GL implementations with the SDK or with native APIs provided in the Android NDK.
Android implementers can test OpenGL ES functionality using the drawElements Quality Program, also known as deqp.
No matter what rendering API developers use, everything is rendered onto a "surface." The surface represents the producer side of a buffer queue that is often consumed by SurfaceFlinger. Every window that is created on the Android platform is backed by a surface. All of the visible surfaces rendered are composited onto the display by SurfaceFlinger.
The following diagram shows how the key components work together:
Figure 1. How surfaces are rendered
The main components are described below:
An image stream producer can be anything that produces graphic buffers for consumption. Examples include OpenGL ES, Canvas 2D, and mediaserver video decoders.
The most common consumer of image streams is SurfaceFlinger, the system service that consumes the currently visible surfaces and composites them onto the display using information provided by the Window Manager. SurfaceFlinger is the only service that can modify the content of the display. SurfaceFlinger uses OpenGL and the Hardware Composer to compose a group of surfaces.
Other OpenGL ES apps can consume image streams as well, such as the camera app consuming a camera preview image stream. Non-GL applications can be consumers too, for example the ImageReader class.
The Android system service that controls a window, which is a container for views. A window is always backed by a surface. This service oversees lifecycles, input and focus events, screen orientation, transitions, animations, position, transforms, z-order, and many other aspects of a window. The Window Manager sends all of the window metadata to SurfaceFlinger so SurfaceFlinger can use that data to composite surfaces on the display.
The hardware abstraction for the display subsystem. SurfaceFlinger can delegate certain composition work to the Hardware Composer to offload work from OpenGL and the GPU. SurfaceFlinger acts as just another OpenGL ES client. So when SurfaceFlinger is actively compositing one buffer or two into a third, for instance, it is using OpenGL ES. This makes compositing lower power than having the GPU conduct all computation.
The Hardware Composer HAL conducts the other half of the work and is the central point for all Android graphics rendering. The Hardware Composer must support events, one of which is VSYNC (another is hotplug for plug-and-playHDMI support).
The graphics memory allocator (Gralloc) is needed to allocate memory requested by image producers. For details, see Gralloc HAL.
See the following diagram for a depiction of the Android graphics pipeline:
Figure 2. Graphic data flow through Android
The objects on the left are renderers producing graphics buffers, such as the home screen, status bar, and system UI. SurfaceFlinger is the compositor and Hardware Composer is the composer.
BufferQueues provide the glue between the Android graphics components. These are a pair of queues that mediate the constant cycle of buffers from the producer to the consumer. Once the producers hand off their buffers, SurfaceFlinger is responsible for compositing everything onto the display.
See the following diagram for the BufferQueue communication process.
Figure 3. BufferQueue communication process
BufferQueue contains the logic that ties image stream producers and image stream consumers together. Some examples of image producers are the camera previews produced by the camera HAL or OpenGL ES games. Some examples of image consumers are SurfaceFlinger or another app that displays an OpenGL ES stream, such as the camera app displaying the camera viewfinder.
BufferQueue is a data structure that combines a buffer pool with a queue and uses Binder IPC to pass buffers between processes. The producer interface, or what you pass to somebody who wants to generate graphic buffers, is IGraphicBufferProducer (part of SurfaceTexture). BufferQueue is often used to render to a Surface and consume with a GL Consumer, among other tasks. BufferQueue can operate in three different modes:
Synchronous-like mode - BufferQueue by default operates in a synchronous-like mode, in which every buffer that comes in from the producer goes out at the consumer. No buffer is ever discarded in this mode. And if the producer is too fast and creates buffers faster than they are being drained, it will block and wait for free buffers.
Non-blocking mode - BufferQueue can also operate in a non-blocking mode where it generates an error rather than waiting for a buffer in those cases. No buffer is ever discarded in this mode either. This is useful for avoiding potential deadlocks in application software that may not understand the complex dependencies of the graphics framework.
Discard mode - Finally, BufferQueue may be configured to discard old buffers rather than generate errors or wait. For instance, if conducting GL rendering to a texture view and drawing as quickly as possible, buffers must be dropped.
To conduct most of this work, SurfaceFlinger acts as just another OpenGL ES client. So when SurfaceFlinger is actively compositing one buffer or two into a third, for instance, it is using OpenGL ES.
The Hardware Composer HAL conducts the other half of the work. This HAL acts as the central point for all Android graphics rendering.
Since Android graphics offer no explicit parallelism, vendors have long implemented their own implicit synchronization within their own drivers. This is no longer required with the Android graphics synchronization framework. See the Explicit synchronization section for implementation instructions.
The synchronization framework explicitly describes dependencies between different asynchronous operations in the system. The framework provides a simple API that lets components signal when buffers are released. It also allows synchronization primitives to be passed between drivers from the kernel to userspace and between userspace processes themselves.
For example, an application may queue up work to be carried out in the GPU. The GPU then starts drawing that image. Although the image hasn’t been drawn into memory yet, the buffer pointer can still be passed to the window compositor along with a fence that indicates when the GPU work will be finished. The window compositor may then start processing ahead of time and hand off the work to the display controller. In this manner, the CPU work can be done ahead of time. Once the GPU finishes, the display controller can immediately display the image.
The synchronization framework also allows implementers to leverage synchronization resources in their own hardware components. Finally, the framework provides visibility into the graphics pipeline to aid in debugging.