From f16c42333aa6b2de30a344dd68246d4a33d93e7d Mon Sep 17 00:00:00 2001 From: Android Partner Docs Date: Tue, 22 Aug 2017 10:41:24 -0700 Subject: Docs: Changes to source.android.com - 166080694 Devsite localized content from translation request a3d5a7... by Android Partner Docs - 166079245 Remove duplicate TOC entry to oob-users.html. by mheco - 166002955 Update builds for Oreo by Android Partner Docs - 165977566 Fixing bad conversion by hvm - 165977199 Edit links to point to public source files in AOSP. by cqn - 165962883 Add codename to CTS downloads page. by gdimino - 165955117 Integration of O branch into mainline. by gdimino - 165638251 Update July public Android security bulletin to remove QC... by Android Partner Docs - 165638198 Update June public Android security bulletin to remove QC... by Android Partner Docs - 165638174 Update May public Android security bulletin to remove QC ... by Android Partner Docs - 165638096 Update April public Android security bulletin to remove Q... by Android Partner Docs - 165528993 Update to Keymaster 2 and remove requirements language by daroberts - 165511119 Add Bluetooth verification / debug information by cqn - 165491345 Fixed link broken by file rename. by cqn - 165381648 Fixed broken image paths and renamed HCI Requirements file. by cqn - 165365185 Created high-level Bluetooth directory and added HTML ver... by cqn - 165335694 Devsite localized content from translation request 66a39c... by Android Partner Docs - 165246927 Update August 2017 bulletin with CVE-2017-0687 by daroberts PiperOrigin-RevId: 166080694 Change-Id: I2d3a8d77fa6a66c2099f13ba2e864545328fd17a --- en/devices/architecture/kernel/hardening.html | 100 +++ en/devices/architecture/kernel/index.html | 80 +++ en/devices/architecture/kernel/lldb-debug.html | 110 +++ .../architecture/kernel/modular-kernels.html | 751 +++++++++++++++++++++ en/devices/architecture/kernel/releases.html | 180 +++++ .../architecture/kernel/reqs-interfaces.html | 273 ++++++++ en/devices/architecture/kernel/squashfs.html | 114 ++++ 7 files changed, 1608 insertions(+) create mode 100644 en/devices/architecture/kernel/hardening.html create mode 100644 en/devices/architecture/kernel/index.html create mode 100644 en/devices/architecture/kernel/lldb-debug.html create mode 100644 en/devices/architecture/kernel/modular-kernels.html create mode 100644 en/devices/architecture/kernel/releases.html create mode 100644 en/devices/architecture/kernel/reqs-interfaces.html create mode 100644 en/devices/architecture/kernel/squashfs.html (limited to 'en/devices/architecture/kernel') diff --git a/en/devices/architecture/kernel/hardening.html b/en/devices/architecture/kernel/hardening.html new file mode 100644 index 00000000..28b6d87e --- /dev/null +++ b/en/devices/architecture/kernel/hardening.html @@ -0,0 +1,100 @@ + + + Kernel Hardening + + + + + +

+Android 8.0 added kernel hardening features to help mitigate kernel +vulnerabilities and find bugs in kernel drivers. The features are in kernel/common in +branches android-3.18, android-4.4, and android-4.9. +

+

Implementation

+

+To acquire these features, device manufacturers and SOCs should merge all +hardening patches from kernel/common to their kernel tree and +enable the following kernel configuration options: +

+
    +
  • Hardened usercopy: CONFIG_HARDENED_USERCOPY=y
    • +
  • PAN emulation - arm64: CONFIG_ARM64_SW_TTBR0_PAN=y
    • +
  • PAN emulation - arm: CONFIG_CPU_SW_DOMAIN_PAN=y
    • +
  • KASLR - 4.4 and later kernels: + CONFIG_RANDOMIZE_BASE=y
    • +
+

+KASLR also requires bootloader support for passing hardware entropy through +either the device tree node /chosen/kaslr-seed or by implementing +EFI_RNG_PROTOCOL. +

+

+Also ensure existing hardening features are enabled: +

+
    +
  • Stack buffer overflow mitigation: + CONFIG_CC_STACKPROTECTOR_STRONG=y
    • +
  • Internal memory protection: CONFIG_DEBUG_RODATA=y or + CONFIG_STRICT_KERNEL_RWX=y
    • +
  • Restrict user-space access from kernel - x86 (enabled by default): + CONFIG_X86_SMAP=y
    • +
+

Testing

+

+To test your implementation, add CONFIG_LKDTM=y to the kernel +configuration and confirm that each of the following commands lead to a kernel +panic: +

+ +
+echo ACCESS_USERSPACE > /sys/kernel/debug/provoke-crash/DIRECT
+echo EXEC_USERSPACE > /sys/kernel/debug/provoke-crash/DIRECT
+echo WRITE_RO > /sys/kernel/debug/provoke-crash/DIRECT
+echo WRITE_RO_AFTER_INIT > /sys/kernel/debug/provoke-crash/DIRECT
+echo WRITE_KERN > /sys/kernel/debug/provoke-crash/DIRECT
+echo EXEC_STACK > /sys/kernel/debug/provoke-crash/DIRECT
+echo EXEC_RODATA > /sys/kernel/debug/provoke-crash/DIRECT
+echo EXEC_KMALLOC > /sys/kernel/debug/provoke-crash/DIRECT
+echo EXEC_VMALLOC > /sys/kernel/debug/provoke-crash/DIRECT
+echo CORRUPT_STACK > /sys/kernel/debug/provoke-crash/DIRECT
+
+

+For android-4.9: +

+ +
+echo USERCOPY_HEAP_SIZE_TO > /sys/kernel/debug/provoke-crash/DIRECT
+echo USERCOPY_HEAP_SIZE_FROM > /sys/kernel/debug/provoke-crash/DIRECT
+
+ +

Common issues

+

+These changes are likely to expose bugs in kernel drivers, which need to be +fixed either by the device manufacturer or the owner of the kernel driver. +

+
    +
  • Hardened usercopy exposes incorrect bounds checking when copying data + to/from user space. These should be fixed like any other memory corruption bugs.
    • +
  • PAN emulation exposes direct user space access from the kernel, which is not + allowed. Drivers attempting to access user space memory need to be changed to + use the standard copy_to_user()/copy_from_user() + functions instead.
    • +
+ + diff --git a/en/devices/architecture/kernel/index.html b/en/devices/architecture/kernel/index.html new file mode 100644 index 00000000..456662f7 --- /dev/null +++ b/en/devices/architecture/kernel/index.html @@ -0,0 +1,80 @@ + + + Kernel + + + + + + +

The Linux kernel is an extremely important part of the software on nearly +every Android device. This section describes Linux kernel development and +release models (below), +stable and long-term +supported (LTS) kernels (including why all Android devices should use stable +releases instead of cherry picking patches), kernel +configuration and +hardening, +requirements for +interfaces and +the +modular kernels +(introduced in Android O), kernel +debugging and +network testing, +and SquashFS.

+ +

Linux kernel development

+

+The Linux kernel is the largest collaborative software project ever. In 2016, +over 4,000 different developers from over 450 different companies contributed to +the project and there were 6 releases, each containing between 12,000 and 16,000 +different changes. At the end of 2016 the size of the Linux kernel was just over +56 thousand files, consisting of 22 million lines of code, build scripts, and +documentation (kernel release 4.9). (For full Linux development statistics, +refer to +https://kernelnewbies.org/DevelopmentStatistics.) +

+

+While the Linux kernel contains code for all the different chip architectures +and hardware drivers it supports, an individual system runs only a fraction of +the codebase. An average laptop uses around 2 million lines of kernel code from +5 thousand files to function properly, while the Pixel phone uses 3.2 million +lines of kernel code from 6 thousand files (due to the increased complexity of +an SoC). +

+ +

Linux kernel releases

+

The Linux kernel uses a release model that differs substantially from +standard AOSP releases. With the release of the 2.6 kernel in December of 2003, +the kernel developer community switched from the previous model of having a +separate development and stable kernel branch, and moved to a stable +only branch model. In this model, a new release occurred every 2 to 3 +months, and that release was declared stable and recommended for all +users to run. This change in development model was due to the very long +release cycle prior to the 2.6 kernel (almost 3 years), and the struggle to +maintain two different branches of the codebase at the same time.

+ +

The numbering of the kernel releases began at 2.6.x, where x was an +incrementing number that changed on every release (the value of the number has +no meaning, other than it is newer than the previous kernel release). The kernel +version since then has now moved to 4.x accounting for 2 major version changes. +These version numbers are chosen by the maintainer(s) only to avoid confusion +among users caused by higher minor release numbers.

+ + + diff --git a/en/devices/architecture/kernel/lldb-debug.html b/en/devices/architecture/kernel/lldb-debug.html new file mode 100644 index 00000000..0c019fd5 --- /dev/null +++ b/en/devices/architecture/kernel/lldb-debug.html @@ -0,0 +1,110 @@ + + + Kernel Enhancements to LLDB/C++ Debugging + + + + + + +

+The Android 8.0 release includes kernel enhancements that help developers create +better applications by improving their debugging experience. +

+

+The arm64 Android kernels support setting watchpoints on memory addresses that +are not 4/8-byte aligned, and reporting all accesses to those addresses. +

+

Implementation

+

+The feature runs on any ARM 64-bit device. Adding the relevant support for +32-bit hardware/kernels is optional. All necessary kernel modifications have +already been done. +

+

+This feature is included in the current 3.10, 3.18, 4.4, and 4.9 kernel branches. +To add it to a kernel that does not already include it, cherry pick the +necessary CLs into your kernel build. Choose the patchset according to the +version your kernel is based on, as the patches required some adjustments due to +the kernel codebase evolving over time: +

+ + + +

+This feature implements an extension of the ptrace API to support +writing a debugger. +

+

Validation

+

+There is a kernel selftest in the upstream repository, a CTS test, and there +are CTS tests added exercising the new kernel API: +

+

+https://android.googlesource.com/platform/bionic/+/master/tests/sys_ptrace_test.cpp +

+ + diff --git a/en/devices/architecture/kernel/modular-kernels.html b/en/devices/architecture/kernel/modular-kernels.html new file mode 100644 index 00000000..98ac0528 --- /dev/null +++ b/en/devices/architecture/kernel/modular-kernels.html @@ -0,0 +1,751 @@ + + + Modular Kernel Requirements + + + + + + +

In Android O, the device kernel splits into System-on-Chip (SoC), device, and +board-specific deliverables. This sets up the kernel and Android such that +Original Device Manufacturers (ODMs) and Original Equipment Manufacturers (OEMs) +can work in isolated board–specific trees for board–specific features, drivers, +etc., enabling them to override common kernel configuration, add new drivers in +the form of kernel modules, etc.

+ +

This page provides details on requirements for:

+ +
    +
  • Platform support for independent SoC and OEM/ODM kernel development. Android +O recommends all board–specific code to be built and shipped as kernel modules +in devices. As a result: + +
    • +
  • Application binary interface (ABI)/application programming interface (API) +tests in Vendor Test Suite (VTS) to +ensure a given kernel can run the Android Open Source Project (AOSP) +framework.
    • +
  • Minimum kernel version per Android release and support for generating +Android Vendor Interface +(VINTF) kernel objects.
    • +
+ +

Loadable kernel modules

+

All SoC kernels should support loadable kernel modules. As a starting point, +the following kernel-config options (or their kernel-version equivalent) have +been added to +android-base.cfg +in all common kernels and must be enabled in all device kernels:

+ +
+CONFIG_MODULES=y
+CONFIG_MODULE_UNLOAD=y
+CONFIG_MODVERSIONS=y
+
+ +

All kernel modules are subject to module load/unload testing to ensure the +correctness of the driver/module.

+ + + +

Module signing

+

Optionally, ODMs can enable module signing in their own kernel configuration +by enabling following kernel config options:

+ +
+CONFIG_MODULE_SIG=y
+CONFIG_MODULE_SIG_FORCE=y
+
+ +

On devices required to support verified boot, Android requires the kernel +modules to be in the partitions that have dm-verity enabled. Module signing is +not mandatory and will not be tested against; however, if desired, an ODM can +enable module signing as long as they have the key signing and other +infrastructure required to ensure independent kernel and filesystem OTA updates +in the future.

+ +

File locations

+

While Android 7.x and earlier do not mandate against kernel modules (and +include support for insmod and rmmod), Android O +recommends the use of kernel modules in the ecosystem. The following table shows +potential board–specific peripheral support required across three Android boot +modes:

+ + + + + + + + + + + + + + + + + + + + + + + +> + + + + + + + + + + + + + + + + + + + + + + + + + + +
Boot ModeStorageDisplayKeypadBatteryPMICTouchscreenNFC, Wi-Fi,
Bluetooth		SensorsCamera
Recovery
Charger
Android
+ +

In addition to availability in Android boot modes, kernel modules may also be +categorized by who owns them (the SoC vendor or the ODM). If kernel modules are +being used, requirements for their placement in filesystem are as follows:

+ +
    +
  • All kernels should have built-in support for booting and mounting partitions. +
    • +
  • Kernel modules should be loaded from a read-only partition.
    • +
  • For devices required to have verified boot, kernel modules should be loaded +from verified partitions.
    • +
  • Kernel modules should not be located in /system.
    • +
  • Kernel modules from the SoC vendor that are required for full Android or +Charger modes should be located in /vendor/lib/modules.
    • +
  • If an ODM partition exists, kernel modules from the ODM that are required +for full Android or Charger modes should be located in +/odm/lib/modules. Otherwise, these modules should be located in +/vendor/lib/modules.
    • +
  • Kernel modules from both the SoC vendor and ODM that are required for +Recovery mode should be located in the recovery ramfs at +/lib/modules.
    • +
  • If a kernel module is required for both Recovery mode and full Android or +Charger modes, it should exist both in the recovery rootfs and +either the /vendor or /odm partitions +(as described above).
    • +
  • Kernel modules used in Recovery mode should not depend on modules located +only in /vendor or /odm, as those partitions are not +mounted in Recovery mode.
    • +
  • SoC vendor kernel modules should not depend on ODM kernel modules.
    • +
+ +

In Android 7.x and earlier, /vendor and /odm +partitions are not mounted early. In Android O, to make module +loading from these partitions possible, provisions have been made to mount +partitions early for both +non-A/B and A/B +devices. This also ensures the partitions are mounted in both Android and +Charger modes.

+ +

Android build system support

+

In BoardConfig.mk, the Android build defines a +BOARD_VENDOR_KERNEL_MODULES variable that provides a full list of +the kernel modules intended for the vendor image. The modules listed in this +variable are copied into the vendor image at /lib/modules/, and, +after being mounted in Android, appear in /vendor/lib/modules (in +accordance with the above requirements). Example configuration of the vendor +kernel modules:

+ +
+vendor_lkm_dir := device/$(vendor)/lkm-4.x
+BOARD_VENDOR_KERNEL_MODULES := \
+  $(vendor_lkm_dir)/vendor_module_a.ko \
+  $(vendor_lkm_dir)/vendor_module_b.ko \
+  $(vendor_lkm_dir)/vendor_module_c.ko
+
+ +

… where a vendor kernel module pre-built repository is mapped into the +Android build at the location listed above.

+ +

The recovery image is likely to contain a subset of the vendor modules. The +Android build defines the variable BOARD_RECOVERY_KERNEL_MODULES +for these modules. Example:

+ +
+vendor_lkm_dir := device/$(vendor)/lkm-4.x
+BOARD_RECOVERY_KERNEL_MODULES := \
+  $(vendor_lkm_dir)/vendor_module_a.ko \
+  $(vendor_lkm_dir)/vendor_module_b.ko
+
+ +

The Android build takes care of running depmod to generate the +required modules.dep files in /vendor/lib/modules and +/lib/modules (recovery ramfs).

+ +

Module loading & versioning

+

We recommend loading all kernel modules in one pass from +init.rc* by invoking modprobe -a. This avoids the +overhead of repeatedly initializing the C runtime environment for the +modprobe binary. The early-init event can be modified +to invoke modprobe:

+ +
+on early-init
+    exec u:r:modprobe:s0 -- /vendor/bin/modprobe -a -d \
+        /vendor/lib/modules module_a module_b module_c ...
+
+ +

The kernel image may be updated separately from the vendor image, meaning +that kernel modules may be used with kernels other than the one they were +originally compiled against. To allow for this, and to protect against ABI +breakages, module versioning is used. Module versioning is enabled by +CONFIG_MODVERSIONS=y (one of the required kernel configuration +options mentioned above) and is documented in the kernel tree at +Documentation/kbuild/modules.txt.

+ +

Mounting partitions early +(first stage mount)

+ REQUIRED  +

All Treble-enabled devices must enable first stage mount to make sure +init can load SELinux policy fragments that are spread across +system and vendor partitions (this also enables +loading of kernel modules as soon as possible after kernel boot).

+ + + +

Android must have access to the filesystem(s) on which the modules reside. To +enable, Android O supports mounting /system, /vendor, +or /odm as early as init's first stage (i.e before +selinux is initialized). Device makers can use +device tree overlays to +specify fstab entries for early mounted partitions.

+ + + +

Early mounting partitions, VBoot +1.0

+

Requirements to early mount partitions with vboot 1.0 include:

+
    +
  1. Device node paths must use their by-name symlinks in +fstab and device tree entries. For example, instead of specifying +partitions using /dev/block/mmcblk0pX, ensure partitions are named +and the device node is /dev/block/…./by-name/{system,vendor,odm}. +
    2. +
  2. Paths given for PRODUCT_{SYSTEM,VENDOR}_VERITY_PARTITION and +CUSTOM_IMAGE_VERITY_BLOCK_DEVICE in the device configuration for +the product (i.e. in +device/oem/project/device.mk) must match the +corresponding block device nodes specified by-name in the +fstab/device tree entries. Example: +
    +PRODUCT_SYSTEM_VERITY_PARTITION := /dev/block/…./by-name/system
+PRODUCT_VENDOR_VERITY_PARTITION := /dev/block/…./by-name/vendor
+CUSTOM_IMAGE_VERITY_BLOCK_DEVICE := /dev/block/…./by-name/odm
+
    +
    3. +
  3. Entries provided via device tree overlays must not repeat in the +fstab file fragments. For example, when specifying an entry to +mount /vendor in the device tree, the fstab file must +not repeat that entry.
    4. +
  4. Only /system, /odm, or /vendor can be +mounted early. Android does not include support to mount any other partitions in +init first stage.
    5. +
  5. Partitions requiring verifyatboot must not be +early mounted (doing so is unsupported).
    6. +
  6. The verity mode/state for verified partitions must be specified in kernel +cmdline using androidboot.veritymode option (existing +requirement).
    7. +
+ +

Early mounting device tree, VBoot +1.0

+

In Android O, init parses the device tree and creates +fstab entries to mount the partition early during its first stage. +An fstab entry takes the form:

+ +
src mnt_point type mnt_flags fs_mgr_flags
+ +

Device tree properties are defined to mimic that format:

+ +
    +
  • fstabentries must be under /firmware/android/fstab +in the device tree and must have compatible string set to +android,fstab.
    • +
  • Each node under /firmware/android/fstab is treated as a single +early mount fstab entry. A node must have the following properties +defined: +
      +
    • dev. Must point to the device node representing the + partition by-name.
      • +
    • type. Must be the filesystem type (as in the + fstab files).
      • +
    • mnt_flags. Must be the comma-separated list of mount + flags (as in fstab files).
      • +
    • fsmgr_flags. Must be the list of Android fs_mgr + flags (as in fstab files). +
        +
      • A/B partitions must have slotselect fs_mgroption.
        • +
      • dm-verity enabled partitions must have verify fs_mgr option. +
        • +
      +
      • +
    +
    • +
+ +

Example: /system and /vendor on N6P

+

The following example shows device tree early mount for system +and vendor partitions on Nexus 6P:

+ +
+/ {
+  firmware {
+    android {
+      compatible = "android,firmware";
+  fstab {
+    compatible = "android,fstab";
+    system {
+      compatible = "android,system";
+      dev = "/dev/block/platform/soc.0/f9824900.sdhci/by-name/system";
+      type = "ext4";
+      mnt_flags = "ro,barrier=1,inode_readahead_blks=8";
+      fsmgr_flags = "wait,verify";
+    };
+    vendor {
+      compatible = "android,vendor";
+      dev = "/dev/block/platform/soc.0/f9824900.sdhci/by-name/vendor";
+      type = "ext4";
+      mnt_flags = "ro,barrier=1,inode_readahead_blks=8";
+      fsmgr_flags = "wait";
+    };
+      };
+    };
+  };
+};
+
+ +

Example: /vendor on Pixel

+

The following example shows device tree early mount for +/vendor on Pixel (remember to add slotselect for +partitions subject to A/B):

+ +
+/ {
+  firmware {
+    android {
+      compatible = "android,firmware";
+      fstab {
+        compatible = "android,fstab";
+        vendor {
+          compatible = "android,vendor";
+          dev = "/dev/block/platform/soc/624000.ufshc/by-name/vendor";
+          type = "ext4";
+          mnt_flags = "ro,barrier=1,discard";
+          fsmgr_flags = "wait,slotselect,verify";
+        };
+      };
+    };
+  };
+};
+
+ +

Early mounting partitions, VBoot +2.0

+

VBoot 2.0 is +Android +Verified Boot (AVB). The requirements to early mount partitions with VBoot +2.0 are:

+
    +
  1. The device node paths must use their by-name symlinks in +fstab and device tree entries. For example, instead of specifying +partitions using /dev/block/mmcblk0pX, ensure the partitions are +named and the device node is +/dev/block/…./by-name/{system,vendor,odm}.
    2. +
  2. Build system variables (such as +PRODUCT_{SYSTEM,VENDOR}_VERITY_PARTITION and +CUSTOM_IMAGE_VERITY_BLOCK_DEVICE) used for VBoot 1.0 are NOT +required for VBoot 2.0. Instead, new build variables introduced in VBoot 2.0 +(including BOARD_AVB_ENABLE := true) should be defined; for a full +configuration, refer to +Build-System-Integration +for AVB.
    3. +
  3. Entries provided via device tree overlays must not repeat in the +fstab file fragments. For example, if you specify an entry to mount +/vendor in the device tree, the fstab file must not +repeat that entry.
    4. +
  4. Only /system, /odm, or /vendor can be +mounted early. Android does not include support to mount any other partitions in +init first stage.
    5. +
  5. VBoot 2.0 does not support verifyatboot, regardless of whether +early mount is enabled or not.
    6. +
  6. The verity mode/state for verified partitions must be specified in kernel +cmdline using androidboot.veritymode option (existing requirement). +Make sure to include the following fixes for AVB: + +
    7. +
+ +

Early mounting device tree, VBoot +2.0

+

The configuration in device tree for VBoot 2.0 is the same as that in +VBoot 1.0, with the +following exceptions:

+
    +
  • The fsmgr_flag is switched from verify to +avb.
    • +
  • All partitions with AVB metadata must be in the vbmeta entry in device tree, +even when the partition isn't mounting early (e.g., /boot).
    • +
+ +

Example: /system and /vendor on N5X

+

The following example shows device tree early mount for system +and vendor partitions on Nexus 5X. Note that:

+
    +
  • /system is mounted with AVB and /vendor is mounted +without integrity verification.
    • +
  • As the Nexus 5X has no /vbmeta partition, so the top-level +vbmeta resides at the end of /boot partition (for details, refer to +the +AOSP +changelist). + +
    +/ {
+  firmware {
+    android {
+      compatible = "android,firmware";
+      vbmeta {
+      compatible = "android,vbmeta";
+      parts = "boot,system,vendor";
+      };
+  fstab {
+    compatible = "android,fstab";
+    system {
+      compatible = "android,system";
+      dev = "/dev/block/platform/soc.0/f9824900.sdhci/by-name/system";
+      type = "ext4";
+      mnt_flags = "ro,barrier=1,inode_readahead_blks=8";
+      fsmgr_flags = "wait,avb";
+        };
+    vendor {
+      compatible = "android,vendor";
+      dev = "/dev/block/platform/soc.0/f9824900.sdhci/by-name/vendor";
+      type = "ext4";
+      mnt_flags = "ro,barrier=1,inode_readahead_blks=8";
+      fsmgr_flags = "wait";
+        };
+      };
+    };
+  };
+};
+
    +
    • +
+ +

Example: /vendor on Pixel

+

The following example shows mounting /vendor early on a Pixel. +Note that:

+ +
    +
  • More partitions are specified in the vbmeta entry because those partitions +are +protected +by AVB.
    • +
  • All AVB partitions must be included, even if only /vendor is +early mounted.
    • +
  • Remember to add slotselect for partitions +subject to A/B. +
    +/ {
+  vbmeta {
+      compatible = "android,vbmeta";
+  parts = "vbmeta,boot,system,vendor,dtbo";
+  };
+  firmware {
+    android {
+      compatible = "android,firmware";
+      fstab {
+        compatible = "android,fstab";
+        vendor {
+          compatible = "android,vendor";
+          dev = "/dev/block/platform/soc/624000.ufshc/by-name/vendor";
+          type = "ext4";
+          mnt_flags = "ro,barrier=1,discard";
+          fsmgr_flags = "wait,slotselect,avb";
+        };
+      };
+    };
+  };
+};
+
    +
    • +
+ +

Device tree overlay support +(Bootloader)

+

+Device Tree Overlay (DTO) was +designed to extend the existing +flattened +device-tree (FDT) implementation so that the initial device-tree data in +kernel can be modified by userspace at runtime by loading additional overlay +FDTs that amend the original data. Android does not require runtime updates of +DT blobs from user space, but instead recommends that vendors add the device +tree patching in the bootloader with the help of +libfdt/libufdt.

+ +

In Android 7.x and earlier, Android did not require device tree support and +did not provide recommendations regarding how vendors pass DT blobs to the +kernel or where they store them. Android O recommends such support to keep the +board–specific and SoC-only parts of the kernel separate.

+ +

Partitioning requirements

+

Most Android devices today append the DT blob to the kernel at build time, +which the bootloader knows how to read from. As Android has no specific +requirements for how to build/store DT blobs (which is considered as part of the +SoC kernel), the DT blob can be appended to the kernel or stored in a separate +partition. The only assumption is that the bootloader already knows how and +where to load the DT blob from.

+ +

Requirements for Device Tree Overlay support (if used):

+
    +
  • Device should have new device tree blob for overlay (DTBO) partion per +kernel image for board–specific DT overlay (for details on the partition format, +see DTB/DTBO Partitions. +The assumption is that bootloader already knows where and how to load the +SoC–specific DTB.
    • +
  • Overlay DT partition should be +A/B-ed +for A/B devices. For these devices, the recovery kernel is the same as Android +kernel, but the partition must be A/B-ed as it can be updated via OTA.
    • +
  • Partition size is board–specific. +
      +
    • The DT overlay partition size depends on the device and the amount of + changes needed on top of the main SoC kernel DT blob.
      • +
    • The size of the DTBO partition is a function of number of changes needed to + make the SoC kernel. Choose a size with room to grow for future updates + (typically, 8MB partition size is more than enough).
      • +
    +
    • +
+ +

Bootloader requirements

+

Requirements for bootloader include the following:

+
    +
  • Bootloader should know how and where (considering the boot slot for A/B +devices) to load the SoC–specific DT blob from in a vendor-specific way. This is +typically extracted from the end of the kernel image as blobs are appended to +the kernel.
    • +
  • Bootloader should know how and where (considering the boot slot for A/B +devices) to load the overlay DT blob from in a vendor-specific way.
    • +
  • Bootloader must patch the main DT blob with the overlay before passing the +combined device tree to the kernel.
    • +
+ +

For more details about adding support for DTO in bootloader, see +Device Tree Overlays.

+ +

Core kernel requirements

+

Android O mandates a minimum kernel version and kernel configuration and +checks them both in VTS as well as during an OTA. Android device kernels must +enable the kernel .config support along with the option to read the +kernel configuration at runtime through procfs.

+ +

Kernel .config support

+

All device kernels must enable the entirety of +android-base.cfg, +which must include the following kernel–config options (or their kernel–version +equivalent):

+ +
+CONFIG_IKCONFIG=y
+CONFIG_IKCONFIG_PROC=y
+
+ +

Kernel version

+

Kernel version requirements:

+
    +
  • All SoCs productized in 2017 must launch with kernel 4.4 or newer.
    • +
  • All other SoCs launching new Android devices running Android O must use +kernel 3.18 or newer.
    • +
  • Regardless of launch date, all SoCs with device launches on Android O remain +subject to kernel changes required to enable Treble.
    • +
  • Older Android devices released prior to Android O but that will be upgraded +to Android O can continue to use their original base kernel version if +desired.
    • +
+ +

Device tree support

+

Device tree support in the kernel must be enabled and bootloaders must pass +the hardware description in the form of device tree to the kernel (unless the +platform supports ACPI). The device tree must also be available for Android to +read and be able to pass vendor/odm specific parameters to Android. +CONFIG_OF (along with all other device/subsystem specific +CONFIG_OF_* kernel config options) are mandatory.

+ +

CONFIG_PROC_DEVICETREE is required on kernels prior to 3.15 so +Android can access vendor/odm specific configuration very early during boot. On +kernels 3.15 and later, the functionality of this option is merged into +CONFIG_OF.

+ +
+CONFIG_OF=y
+CONFIG_PROC_DEVICETREE=y (kernels prior to 3.15)
+
+ +

For an example of using device tree to early mount +vendor/odm partitions, refer to the +AOSP +changelist.

+ +

DebugFS

+

The implementation of the vendor interface should not rely on +debugfs. It may be enabled, but VTS testing may be done with +debugfs unmounted.

+ +

Beyond Android O

+

Android O recommends any board–specific kernel functionality to be in the +form of loadable kernel modules and device–tree overlays. The rest of the kernel +is treated monolithically with respect to Android (whether or not is it is +actually a monolithic kernel, or parts of it are compiled as kernel modules). +

+ +

This monolithic kernel is an SoC kernel that can boot on the SoC vendor's +reference hardware but nothing beyond that. Today, SoC kernels are treated +similar to the common kernel; they are also heavily replicated in board–specific +repos. This distribution model causes them to be fixed differently for the same +bug in each branch, delaying future updates to the kernel due to cherry–picking +at different times or fixing the same bug differently. To counter this, the SoC +kernels must be a separate deliverable, with everyone who uses the SoC +contributing to the same SoC kernel.

+ +

Figure 1 (below) illustrates a common example of how SoC kernels get +fragmented over time, across Android releases, and across ODMs.

+ + +
Figure 1. Device kernel replication.
+ +

Figure 1 shows the following:

+
    +
  1. It takes a significant amount of effort and time for everyone to cross-merge +across board–specific branches/tags.
    2. +
  2. While waiting for the cross-merge, Android device manufacturers patch their +own kernel for bugs/security fixes.
    3. +
  3. Divergence from the ancestor make future upgrades/merges really +difficult.
    4. +
+

The proposed model for a common SoC kernel addresses problems created by +upmerging changes (SoC-specific bug fixes, LTS upgrades, security fixes, etc.). +Figure 2 (below) illustrates how the workflow will change in an ideal, +unified–per–SoC–kernel scenario:

+ + +
Figure 2. Android O and higher device +kernels.
+ +

This is intended to solve the problem of fragmented kernel repos by +recommending and working with device manufacturers to stay up to date with the +common SoC kernel. Android O provides all possible options to ODMs to help them +avoid maintaining their own SoC kernels and instead rely on the common SoC +kernel for LTS upgrades/bug fixes/security patches/etc.

+ +

As a start, we want to facilitate all ODMs/vendors using a single kernel +source for an SoC. In the future, we want to move towards a single binary +distribution of kernel per-SoC.

+ +

Upstreaming

+

To make updating to newer kernel versions much easier and almost automatic, +and to provide a more secure and reliable platform for ODMs to build a product +with, it is strongly recommended that SoC vendors work to upstream their kernel +changes and get them accepted into the main kernel.org repository. While doing +so requires additional, up front efforts in time and engineering resources, it +is well documented to save both time and money in the long run. It has also been +documented that merged code is of a much higher quality with fewer bugs and +security issues (and usually smaller) than code that has not been reviewed by +the community.

+ +

If full support for the SoC is merged upstream, the community can make needed +API changes as the internal kernel API evolves over time, automatically +extending the longevity of the platform. The kernel can also be automatically +tested for any regressions in development and stable releases by adding the +hardware platform to one of the many community-managed kernel test platforms +(such as +kernelci.org). +

+ +

For help working with the Linux kernel community to upstream your code, refer +to the following resources:

+
    +
  • Documentation/process +(Documentation/development-process in 4.9 and earlier)
    • +
  • Documentation/CodingStyle
    • +
  • Documentation/SubmittingPatches
    • +
+ +

The community uses a minimal review process to accept stand-alone drivers and +filesystems into the staging portion of the kernel, where the community works to +improve code quality.

+ + + diff --git a/en/devices/architecture/kernel/releases.html b/en/devices/architecture/kernel/releases.html new file mode 100644 index 00000000..612922d9 --- /dev/null +++ b/en/devices/architecture/kernel/releases.html @@ -0,0 +1,180 @@ + + + Stable Kernel Releases & Updates + + + + + + +

+The Linux kernel stable release model started in 2005, when it was determined +that the existing kernel development model (a new release every 2-3 months) was +not meeting the needs of most users. Users wanted bugfixes made during those 2-3 +months, and Linux distributions found it difficult to keep kernels up to date +without feedback from the kernel community. In general, attempts to keep +individual kernels secure and with the latest bugfixes was a large and confusing +effort by lots of different individuals. +

+

+Stable kernel releases are based directly on Linus Torvalds' releases, and are +released every week or so, depending on various external factors (time of year, +available patches, maintainer workload, etc.). The numbering of the stable +releases starts with the number of the kernel release, and an additional number +is added to the end of it. For example, the 4.4 kernel is released by Linus, and +then the stable kernel releases based on this kernel are numbered 4.4.1, 4.4.2, +4.4.3, and so on. This sequence is usually shortened with the number 4.4.y when +referring to a stable kernel release tree. Each stable kernel release tree is +maintained by a single kernel developer, who is responsible for picking the +needed patches for the release and managing the review/release process. +

+

+Stable kernels are maintained for the length of the current development cycle. +After Linus releases a new kernel, the previous stable kernel release tree is +stopped and users must move to the newer released kernel. +

+ +

Long-term stable kernels

+

+After a year of this new stable release process, it was determined that many +different users of Linux wanted a kernel to be supported for longer than just a +few months. In response, the Long Term Supported (LTS) kernel release was +created, with the first LTS kernel (2.6.16) released in 2006. Since then, a new +LTS kernel has been selected once a year and kernel community maintains that +kernel for a minimum of 2 years. +

+

At the time of this writing, the LTS kernels are the 4.4.y and 4.9.y +releases, and a new kernel is released weekly. Due to the needs of some users +and distributions, a few additional older kernels are maintained by kernel +developers at a slower release cycle. Information about all long-term stable +kernels, who is in charge of them, and how long they will be maintained, can be +found on the +kernel.org +releases page.

+

+LTS kernel releases average 6-8 patches accepted per day, while the normal +stable kernel releases contain 10-15 patches per day. The number of patches +fluctuates per release given the current time of the corresponding development +kernel release, and other external variables. The older a LTS kernel is, the +less patches are applicable to it as many recent bugfixes are not relevant to +older kernels. However, the older a kernel is, the harder it is to backport the +changes that are needed to be applied, due to the changes in the codebase. So +while there might be a lower number of overall patches being applied, the effort +involved in maintaining a LTS kernel is greater than maintaining the normal +stable kernel. +

+ +

Stable kernel patch rules

+

The rules for what can be added to a stable kernel release have remained +almost identical since its introduction and are summarized below:

+
    +
  • Must be obviously correct and tested. +
  • Must not be bigger than 100 lines. +
  • Must fix only one thing. +
  • Must fix something that has been reported to be an issue. +
  • Can be a new device id or quirk for hardware, but not add major new +functionality. +
  • Must already be merged into Linus Torvalds' tree.
    • +
+ + +

The last rule, "Must already be merged into Linus Torvalds' tree", prevents +the kernel community from losing fixes. The community never wants a fix to go +into a stable kernel release that is not already in Linus Torvalds' tree, so +that anyone who upgrades should never see a regression. This prevents many +problems that other projects who maintain a stable and development branch can +have.

+ +

Kernel updates

+

The Linux kernel community has promised its userbase that no upgrade will +ever break anything that is currently working in a previous release. That +promise still holds true today. Regressions do happen, but those are the highest +priority bugs and are either quickly fixed, or the change that caused the +regression is quickly reverted from the Linux kernel tree.

+ +

This promise holds true for both the incremental stable kernel updates, as +well as the larger major updates that happen every three months. However, the +kernel community can only make this promise for the code that is merged into the +Linux kernel tree. Any code that is merged into a device's kernel that is not in +the kernel.org releases is unknown and +interactions with it can never be planned for, or even considered.

+ +

Devices based on Linux that have large patch sets can have major issues when +updating to newer kernels, because of the large number of changes between each +release (10-14 thousand changes per release). SoC patchsets are especially known +to have issues with updating to newer kernels due to their large size and heavy +modification of architecture specific, and sometimes core, kernel code. As a +result, most SoC vendors are starting to standardize on using the LTS releases +for their devices, enabling those devices to receive bug and security updates +directly from the Linux kernel community.

+ +

Security

+

When doing kernel releases, the Linux kernel community almost never declares +specific changes as security fixes. This is due to the basic problem of +the difficulty in determining if a bugfix is a security fix or not at the time +of creation. Also, many bugfixes are only determined to be security related +after much time has passed, so the kernel community strongly recommends always +taking all bugfixes that are released.

+ + + +

+When security problems are reported to the kernel community, they are fixed as +soon as possible and pushed out publically to the development tree and the +stable releases. As described above, the changes are almost never described as +a "security fix", but rather look like any other bugfix for the kernel. This is +done to allow affected parties the ability to update their systems before the +reporter of the problem announces it. +

+ +

For details on reporting security bugs to the kernel community to get +them resolved and fixed as soon as possible, refer to +Security +bugs in The Linux kernel user's and administrator's guide at +www.kernel.org.

+ +

+Because security bugs are not announced to the public by the kernel team, CVE +numbers for Linux kernel-related issues are usually released weeks, months, and +sometimes years after the fix was merged into the stable and development +branches. +

+

Keeping a secure system

+

When deploying a device that uses Linux, it is strongly recommended that all +LTS kernel updates be taken by the manufacturer and pushed out to their users +after proper testing shows the update works well. This has several advantages: +

+
    +
  • Releases have been reviewed by the kernel developers as a whole, not in +individual parts.
    • +
  • It is hard, if not impossible, to determine which patches fix "security" +issues and which do not. Almost every LTS release contains at least one known +security fix, and many yet "unknown".
    • +
  • If testing shows a problem, the kernel developer community will react +quickly to resolve the issue.
    • +
  • Attempts to filter out only the changes you run will result in a kernel +tree that is impossible to merge correctly with future upstream releases.
    • +
+ + + diff --git a/en/devices/architecture/kernel/reqs-interfaces.html b/en/devices/architecture/kernel/reqs-interfaces.html new file mode 100644 index 00000000..82349f7b --- /dev/null +++ b/en/devices/architecture/kernel/reqs-interfaces.html @@ -0,0 +1,273 @@ + + + Interface Requirements + + + + + + +

This page describes a subset of the Linux kernel interfaces on which Android +relies to function properly. The presence and correctness of these interfaces is +tested as part of the Vendor Test Suite +(VTS). This subset will grow over time to contain a larger portion of +Android kernel interfaces.

+ +

System calls

+

System calls are expected to provide the same signatures and semantics as in +the upstream Linux kernel.

+ +

ARM64 system calls required by bionic per +bionic/libc/SYSCALLS.txt:

+ + + + +
accept4, acct, adjtimex, bind, brk, capget, capset, chdir, chroot, clock_adjtime, clock_getres, clock_gettime, clock_nanosleep, clock_settime, close, connect, delete_module, dup3, dup, epoll_create1, epoll_ctl, epoll_pwait, eventfd2, execve, exit, exit_group, faccessat, fadvise64, fallocate, fchdir, fchmodat, fchmod, fchownat, fchown, fcntl, fdatasync, fgetxattr, flistxattr, flock, fremovexattr, fsetxattr, fstat, newfstatat, fstatfs, fsync, ftruncate, getcpu, getcwd, getdents64, getegid, geteuid, getgid, getgroups, getitimer, getpeername, getpgid, getpid, getppid, getpriority, getresgid, getresuid, getrlimit, getrusage, getsid, getsockname, getsockopt, gettimeofday, getuid, getxattr, init_module, inotify_add_watch, inotify_init1, inotify_rm_watch, ioctl, kill, syslog, lgetxattr, linkat, listen, listxattr, llistxattr, lremovexattr, lseek, lsetxattr, madvise, mincore, mkdirat, mknodat, mlockall, mlock, mmap, mount, mprotect, mremap, msync, munlockall, munlock, munmap, nanosleep, openat, personality, pipe2, ppoll, prctl, pread64, preadv, prlimit64, process_vm_readv, process_vm_writev, pselect6, ptrace, pwrite64, pwritev, quotactl, readahead, readlinkat, read, readv, reboot, recvfrom, recvmmsg, recvmsg, removexattr, renameat, rt_sigaction, rt_sigpending, rt_sigprocmask, rt_sigqueueinfo, rt_sigsuspend, rt_sigtimedwait, sched_getaffinity, sched_getparam, sched_get_priority_max, sched_get_priority_min, sched_getscheduler, sched_rr_get_interval, sched_setaffinity, sched_setparam, sched_setscheduler, sched_yield, sendfile, sendmmsg, sendmsg, sendto, setdomainname, setfsgid, setfsuid, setgid, setgroups, sethostname, setitimer, setns, setpgid, setpriority, setregid, setresgid, setresuid, setreuid, setrlimit, setsid, setsockopt, set_tid_address, settimeofday, setuid, setxattr, shutdown, sigaltstack, signalfd4, socketpair, socket, splice, statfs, swapoff, swapon, symlinkat, sync_file_range, sync, sysinfo, tee, tgkill, timer_create, timer_delete, timerfd_create, timerfd_gettime, timerfd_settime, timer_getoverrun, timer_gettime, timer_settime, times, truncate, umask, umount2, uname, unlinkat, unshare, utimensat, vmsplice, wait4, waitid, write, writev
+ +

ARM32 system calls required by bionic per +bionic/libc/SYSCALLS.txt:

+ + + + +
accept4, acct, adjtimex, arm_fadvise64_64, bind, brk, cacheflush, capget, capset, chdir, chroot, clock_adjtime, clock_getres, clock_gettime, clock_nanosleep, clock_settime, close, connect, delete_module, dup3, dup, epoll_create1, epoll_ctl, epoll_pwait, eventfd2, execve, exit, exit_group, faccessat, fallocate, fchdir, fchmodat, fchmod, fchownat, fchown32, fcntl64, fdatasync, fgetxattr, flistxattr, flock, fremovexattr, fsetxattr, fstat64, fstatat64, fstatfs64, fsync, ftruncate64, getcpu, getcwd, getdents64, getegid32, geteuid32, getgid32, getgroups32, getitimer, getpeername, getpgid, getpid, getppid, getpriority, getresgid32, getresuid32, ugetrlimit, getrusage, getsid, getsockname, getsockopt, gettimeofday, getuid32, getxattr, init_module, inotify_add_watch, inotify_init1, inotify_rm_watch, ioctl, kill, syslog, lgetxattr, linkat, listen, listxattr, llistxattr, _llseek, lremovexattr, lseek, lsetxattr, madvise, mincore, mkdirat, mknodat, mlockall, mlock, mmap2, mount, mprotect, mremap, msync, munlockall, munlock, munmap, nanosleep, openat, personality, pipe2, ppoll, prctl, pread64, preadv, prlimit64, process_vm_readv, process_vm_writev, pselect6, ptrace, pwrite64, pwritev, quotactl, readahead, readlinkat, read, readv, reboot, recvfrom, recvmmsg, recvmsg, removexattr, renameat, rt_sigaction, rt_sigpending, rt_sigprocmask, rt_sigqueueinfo, rt_sigsuspend, rt_sigtimedwait, sched_getaffinity, sched_getparam, sched_get_priority_max, sched_get_priority_min, sched_getscheduler, sched_rr_get_interval, sched_setaffinity, sched_setparam, sched_setscheduler, sched_yield, sendfile64, sendfile, sendmmsg, sendmsg, sendto, setdomainname, setfsgid, setfsuid, setgid32, setgroups32, sethostname, setitimer, setns, setpgid, setpriority, setregid32, setresgid32, setresuid32, setreuid32, setrlimit, setsid, setsockopt, set_tid_address, settimeofday, set_tls, setuid32, setxattr, shutdown, sigaction, sigaltstack, signalfd4, socketpair, socket, splice, statfs64, swapoff, swapon, symlinkat, sync_file_range2, sync, sysinfo, tee, tgkill, timer_create, timer_delete, timerfd_create, timerfd_gettime, timerfd_settime, timer_getoverrun, timer_gettime, timer_settime, times, truncate64, truncate, umask, umount2, uname, unlinkat, unshare, utimensat, vmsplice, wait4, waitid, write, writev
+ +

The system calls listed below are made by bypassing bionic:

+ + + + + + + + + + + + + + +
All Architecturesgettid, futex, clone, rt_sigreturn, rt_tgsigqueueinfo, restart_syscall, +getrandom, perf_event_open, syncfs, tkill, seccomp
armvfork, sigreturn, pipe, access, stat64, lstat64, open, getdents, eventfd, +epoll_wait, readlink, epoll_create, creat, unlink
arm64pivot_root, ioprio_get, ioprio_set
+ + + +

prctl

+

In addition to the upstream prctl operations for supported +kernel versions, Android relies on additional prctl operations, the +implementation of which can be found in the android-common kernel.

+ +
+PR_SET_TIMERSLACK_PID
+PR_SET_VMA
+
+ +

Filesystems

+

The Linux kernel exports interfaces via several filesystems. Android expects +these interfaces to communicate the same information, in the same format, and +provide the same semantics as in the upstream Linux kernel. For interfaces that +do not exist upstream, the appropriate behavior is dictated by the corresponding +branch of the Android common kernel.

+ +

procfs

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
PathDescription
/proc/cmdlineRead-only file containing command line arguments passed to the kernel. +
/proc/config.gzRead-only file containing kernel build configuration.
/proc/cpuinfoRead-only file containing architecture-specific CPU details.
/proc/kmsgRead-only file showing kernel messages in real time.
/proc/meminfoRead-only file showing memory subsystem details.
/proc/modulesRead-only file containing information about loaded kernel modules.
/proc/mountsSymlink to /proc/self/mounts, which is a read-only file + listing information about the mounted filesystems.
/proc/net/xt_qtaguid/ctrlRead-write file providing information about tagged sockets.
/proc/self/mapsRead-only file containing the currently mapped memory regions and + permissions.
/proc/statRead-only file containing various kernel and system statistics.
/proc/sys/kernel/kptr_restrictRead-write file that determines whether kernel pointers are printed in + proc files and other interfaces.
/proc/sys/kernel/randomize_va_spaceRead-write file that determines the address layout randomization policy + for the system.
/proc/sys/vm/mmap_min_addrRead-write file that determines the minimum address than can be + mmap'd.
/proc/sys/vm/mmap_rnd_bitsRead-write file that specifies the amount of randomness in + mmap'd addresses.
/proc/sys/vm/mmap_rnd_compat_bitsRead-write file that specifies the amount of randomness in + mmap'd addresses.
/proc/sys/vm/overcommit_memoryRead-write file that determines the kernel virtual memory accounting + mode.
/proc/uid_cputime/remove_uid_rangeWrite-only file that, when written, removes UIDs from being shown in + /proc/uid_cputime/show_uid_stat.
/proc/uid_cputime/show_uid_statRead-only file containing the time a UID's processes spent in user and + kernel space.
/proc/versionRead-only file containing a string describing the kernel version.
/proc/vmallocinfoRead-only file containing vmalloc'd ranges.
/proc/zoneinfoRead-only file containing information about memory zones.
+ +

dev

+ + + + + + + + + + + + + + + + + + + + + + + + + + +
PathDescription
/dev/ashmemAnonymous shared memory device file.
/dev/binderBinder device file.
/dev/hwbinderHardware binder device file.
/dev/tunUniversal TUN/TAP device file.
/dev/xt_qtaguidQTAGUID netfilter device file.
+ +

sysfs

+ + + + + + + + + + + + + + + + + +
PathDescription
/sys/devices/system/cpu/onlineRead-only file showing ranges of CPUs that are currently online.
/sys/kernel/wakeup_reasons/last_resume_reasonRead-only file showing a textual description of why the system exited the + last instance of suspend.
/sys/devices/system/cpu/kernel_maxRead-only file showing the maximum CPU index supported by the kernel. +
+ +

selinuxfs

+

The framework mounts selinuxfs at /sys/fs/selinux. +

+ + + + + + + + + + + + + + + + + + +
PathDescription
/sys/fs/selinux/checkreqprotRead/write file containing a binary flag that determines how selinux + protections are checked on mmap and mprotect calls. +
/sys/fs/selinux/nullRead/write null device for use by selinux.
/sys/fs/selinux/policyRead-only file containing the selinux policy in binary form.
+ + + + + diff --git a/en/devices/architecture/kernel/squashfs.html b/en/devices/architecture/kernel/squashfs.html new file mode 100644 index 00000000..93b97ad0 --- /dev/null +++ b/en/devices/architecture/kernel/squashfs.html @@ -0,0 +1,114 @@ + + + Optimizing SquashFS at the Kernel Level + + + + + + +

+ SquashFS is a compressed read-only filesystem for Linux. The file system is + read-only by design and thus suitable for use on the system partition. Many + Android devices may benefit from using this file system for their system + partition, for example, the following: +

    +
  • Devices with a low capacity storage such as Android Watch. +
  • Devices with a slow flash storage (compression reduces the number of block + I/Os).
+

+ Unfortunately the performance of SquashFS lags behind ext4. +

+

Optimizations

+

+ The following optimizations have been implemented to improve the performance of + SquashFS. +

+

Reduce the memory usage and + memcpy

+

+ When reading a block (default 128K), SquashFS tries to grab all the pages + covering this block. +

+

+ If a single page is up-to-date or locked, it falls back to allocating a full + block, submitting a read request, and then copying its content to the pages. +

+

+ This approach is very ineffective; after some time the page cache is likely to + contain pages that are up-to-date even if the adjacent pages are not. +

+

+ The code is now able to handle blocks with holes (=missing pages). This + improves performance in the following ways: +

    +
  • Reduces the number of memcpy calls +
  • Decreases memory allocations
+

Asynchronous reads

+

+ SquashFS still uses the deprecated ll_rw_block() function. There + are two problems with this approach: +

    +
  • As the name implies, the function waits for the reads to complete before + returning. This is redundant since .readpage() already waits on the + page's lock. Moreover, we need an asynchronous mechanism to efficiently + implement .readpages(). +
  • Merging the read requests entirely depends on the I/O scheduler. + ll_rw_block() simply creates one request per buffer. SquashFS has + more information than the I/O scheduler about what should be merged. Moreover, + merging the request means that we rely less on the I/O scheduler.
+

+ For these reasons, the ll_rw_block() function has been replaced + with submit_bio(). +

+

Readpages (prefetching)

+

+ SquashFS does not implement .readpages(), so the kernel repeatedly + calls .readpage(). +

+

+ Now that our read requests are asynchronous, the kernel can truly prefetch pages + using its asynchronous read-ahead mechanism. +

+

Optimize reading uncompressed + blocks

+

+ Modern systems such as Android contain a lot of files that are already + compressed. As a consequence, the image contains a lot of blocks that can't be + compressed. +

+

+ SquashFS handles compressed and uncompressed blocks using the same logic: when + asked to read a single page, it actually reads a full block (default 128k). + While this is necessary for compressed blocks, it is just a waste of resources + for uncompressed blocks. +

+

+ Instead of reading a full block, SquashFS now just reads what is advised by the + readahead algorithm. +

+

+ This greatly improves the performance of random reads. +

+

Code

+

+ SquashFS code optimizations are available in AOSP: +

+ + -- cgit v1.2.3