Kernel Address Sanitizer (KASAN) helps kernel developers and testers find runtime memory-related bugs, such as out-of-bound read or write operations, and use-after-free issues. While KASAN isn't enabled on production builds due to its runtime performance penalties and memory usage increment, it is still a valuable tool for testing debug builds.

When used with another runtime tool called Kernel Coverage (KCOV), KASAN-sanitized and KCOV-instrumented code helps developers and testers to detect runtime memory errors and obtain code coverage information. In the scenario of kernel fuzz testing, e.g. via syzkaller, KASAN helps to determine the root cause of crashes, and KCOV provides code coverage information to the fuzzing engine to help in test-case or corpus deduplication.

This page does not discuss the inner workings or mechanics of KASAN. Rather, it serves as a guide to build and modify the Android Open Soure Project (AOSP) and Pixel's kernel source to boot with KASAN and KCOV turned on.

Setting up your build environment

Follow the steps in the Downloading and Building section to set up your build environment.

Building AOSP

Download the Android source code. For the purpose of building KASAN images, choose a stable build that is not in active development. Often, the latest available release/stable branch is a good choice. More information about build and branch can be found at Source Code Tags and Builds.

After you have successfully checked out the source code, download the necessary device blobs that correspond to the device and branch you are using from Driver Binaries for Nexus and Pixel Devices. Download both the vendor image and the set of binaries from the System-on-Chip (SOC) manufacturer. Then, unarchive the downloaded tarballs, run the scripts they contain, and accept the licenses.

Then clean up, set up your build environment, and choose your build target, following the steps in Preparing to Build.

To establish a working base, make your first build without modifications: 

make -j48

Flash your build result to a test device (for example, marlin) and let it boot: 

cd out/target/product/marlin
ANDROID_PRODUCT_OUT=`pwd` fastboot flashall -w

After booting to the homescreen, you might see a pop-up that says:

There's an internal problem with your device. Contact your manufacturer for details. This pop-up likely means that the build fingerprint of your vendor and your system partition do not match. Because this build is just for development and testing, and not a release build, just ignore it.

Building the kernel

To build the kernel, you need to check out the correct source code, cross-compile it, and then build the kernel image in the correct AOSP directory.

Checking out kernel source code

Create a directory to store the kernel source code and clone the AOSP kernel git repository to your local storage. 

mkdir ~/src/marlin-kernel-src
cd ~/src/marlin-kernel-src
git clone https://android.googlesource.com/kernel/msm

After you are done, there should be an empty directory named msm.

Enter the msm directory and git checkout the branch that corresponds to the source code you are building. For the list of available branches and tags, see the Android msm kernel source tree. 

cd msm
git checkout TAG_NAME

After completing this step, the msm directory should have content.

Performing cross compilation

Next you need to compile the Android kernel.

Setting up your cross-compiler

To build the kernel, you need to set up a cross-compiler. The current recommended and tested toolchain is Android's NDK toolchain latest stable version. To download the Android NDK, visit the official Android NDK website. Download the appropriate zip archive for your platform, and unzip it. This results in a directory resembling android-ndk-NDK_VERSION.

Downloading the LZ4c tool

The Pixel kernel uses LZ4 compression, so the lz4c tool is required when you build your kernel. If you use Ubuntu, install the lz4c tool by: 

sudo apt-get install liblz4-tool

Building your kernel

Set up your build environment from the marlin-kernel-src/msm directory with: 

export ARCH=arm64
export CROSS_COMPILE=PATH_TO_NDK/android-ndk-NDK_VERSION/toolchains/aarch64-linux-android-TOOLCHAIN_VERSION/prebuilt/linux-x86_64/bin/aarch64-linux-android-

Then build an unmodified version of your kernel to establish a working base: 

make marlin_defconfig
make -j48

The result of the build process can be found at: arch/arm64/boot/Image.lz4-dtb

Rebuilding the boot image in AOSP

After you have built the kernel image, copy the result over into AOSP's device/google/marlin-kernel directory, with: 

cp ${marlin-kernel-src}/msm/arch/arm64/boot/Image.lz4-dtb device/google/marlin-kernel
source build/envsetup.sh
lunch aosp_marlin-userdebug
make -j48

After a successful build, flash the target device with: 

cd out/target/product/marlin
fastboot flashall -w

After flashing, your device should boot. Verify the image you flashed to the device is the kernel image you built by checking Kernel version under Settings -> System -> About phone after the device finishes booting.

Modifying the kernel

Enabling KASAN and KCOV compile options

KASAN and KCOV codes are guarded by compilation flags, which are not turned on for normal builds. To enable them, add KASAN and KCOV options to the config file, but drop the LZ4 config.

To do this, make a copy of the default config file, for example, marlin_defconfig: 

cd arch/arm64/configs
cp marlin_defconfig marlin-kasan_defconfig

In the new config file, remove this flag CONFIG_KERNEL_LZ4=y and add these flags: 

CONFIG_KASAN=y
CONFIG_KASAN_INLINE=y
CONFIG_KCOV=y
CONFIG_SLUB=y
CONFIG_SLUB_DEBUG=y

Recompiling the kernel with new configuration

After you've finished modifying your copy of the config file, recompile the kernel.

Reconfiguring the kernel

Set up your build environment. Build your modified defconfig and check if the newly added flags are present in the produced .config file. 

make marlin-kasan_defconfig
grep KASAN .config
CONFIG_HAVE_ARCH_KASAN=y
CONFIG_KASAN=y
# CONFIG_KASAN_OUTLINE is not set
CONFIG_KASAN_INLINE=y

You should see the KASAN flags. Compile your kernel: 

make -j48

Checking the modified kernel image

After a successful compilation, navigate to the arch/arm64/boot directory to view the compilation results. Generally, the Image.gz-dtb is about 23MB and larger than that of a standard build. 

cd arch/arm64/boot
ls -lh Image.gz-dtb
-rw-r--r-- 1 username groupname 23M Aug 11 13:59 Image.gz-dtb

To see if KCOV was properly compiled, perform additional analysis on the produced vmlinux at the root of the kernel source tree. If you run an objdump on vmlinux, you should see numerous calls to __sanitizer_cov_trace_pc(). 

sh -c '${CROSS_COMPILE}objdump -d vmlinux' | grep sanitizer
ffffffc000082030:    94040658    bl    ffffffc000183990 <__sanitizer_cov_trace_pc>
ffffffc000082050:    94040650    bl    ffffffc000183990 <__sanitizer_cov_trace_pc>
ffffffc000082078:    94040646    bl    ffffffc000183990 <__sanitizer_cov_trace_pc>
ffffffc000082080:    94040644    bl    ffffffc000183990 <__sanitizer_cov_trace_pc>
ffffffc0000820ac:    94040639    bl    ffffffc000183990 <__sanitizer_cov_trace_pc>

Modifying AOSP code

Before plugging in the new boot image, you need to adjust certain parameters in AOSP's source code that govern how the device boots. This is mainly necessary to ensure the new (inflated) image boots properly.

Adjusting board parameters

Adjust the boot parameters defined in the device's BoardConfig.mk file. It is located at device/google/marlin/marlin relative to the root of your AOSP source code. 

cd device/google/marlin/marlin
vim BoardConfig.mk

If you do not wish to modify the BoardConfig.mk file, you could instead create a new lunch target that contains the name marlin_kasan. For more information about this process, see Adding a New Device.

Adjusting the kernel target in the local Makefile

The new kernel uses LZ4 compression to improve speed, but KASAN requires gzip for better compression ratio. To accommodate this, tell the build machinery which kernel to bundle into the final target by modifying where the LOCAL_KERNEL variable points to in device/google/marlin/device-common.mk.

Rebuilding the boot image

To rebuild the boot image, copy the new kernel image into the AOSP tree in the device-specific folder (e.g. device/google/marlin-kernel). Make sure this is where the build system expects the kernel target image to be at, according to how you modified it earlier.

Next, rebuild your flashable images, similar to how you built AOSP earlier. Upon successful build, flash all built images as usual.

Booting your device with a modified kernel image

You should now have a build that boots and enters the home screen. From here, check the device's dmesg output for a "KernelAddressSanitizer initialized" message in the very early boot stage. That message means KASAN is initialized during boot time. Also, you can confirm /sys/kernel/debug/kcov is present on the device (you will have to be root to do that).

Troubleshooting

You can experiment with different kernel versions, starting with a standard build as a working base, before turning on KASAN+KCOV compile options. When things break, first check if the bootloader and baseband version on your device matches those required by the new build. Finally, you might have to catch up with a newer branch of the Android tree altogether if you venture too far ahead with the kernel version.