This section summarizes useful tools and related commands for debugging,
tracing, and profiling native Android platform code when developing
platform-level features. This page covers use of debuggerd, a
daemon process for collecting error information after applications crash, and
the GNU Project debugger (GDB).
Other pages in this section explore system services with Dumpsys, viewing native memory, network, and RAM usage, using AddressSanitizer to detect memory bugs in native code, evaluating performance issues (includes systrace), and several other debugging tools.
The debuggerd process dumps registers and unwinds the stack.
When a dynamically linked executable starts, several signal handlers are
registered that connect to debuggerd (or debuggerd64)
in the event that signals (such as SIGSEGV or SIGABRT) are sent to the process.
It's possible for debuggerd to attach only if nothing else is
already attached, which means using tools such as strace or
gdb will prevent debuggerd from working. You can also
explicitly prevent debuggerd from attaching by calling
prctl(PR_SET_DUMPABLE, 0), which can be useful when you need to
opt out of crash reporting.
Example debuggerd output (with timestamps and extraneous
information removed):
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'Android/aosp_angler/angler:7.1.1/NYC/enh12211018:eng/test-keys'
Revision: '0'
ABI: 'arm'
pid: 17946, tid: 17949, name: crasher >>> crasher <<<
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xc
r0 0000000c r1 00000000 r2 00000000 r3 00000000
r4 00000000 r5 0000000c r6 eccdd920 r7 00000078
r8 0000461a r9 ffc78c19 sl ab209441 fp fffff924
ip ed01b834 sp eccdd800 lr ecfa9a1f pc ecfd693e cpsr 600e0030
backtrace:
#00 pc 0004793e /system/lib/libc.so (pthread_mutex_lock+1)
#01 pc 0001aa1b /system/lib/libc.so (readdir+10)
#02 pc 00001b91 /system/xbin/crasher (readdir_null+20)
#03 pc 0000184b /system/xbin/crasher (do_action+978)
#04 pc 00001459 /system/xbin/crasher (thread_callback+24)
#05 pc 00047317 /system/lib/libc.so (_ZL15__pthread_startPv+22)
#06 pc 0001a7e5 /system/lib/libc.so (__start_thread+34)
Tombstone written to: /data/tombstones/tombstone_06
The last line of debuggerd output dumps a summary to the log and
writes a full tombstone to disk. The tombstone is simply a file with
extra data about the crashed process; it contains information that can be
helpful in debugging a crash, in particular the stack traces for all threads in
the crashing process (not just the thread that caught the signal) and a full
memory map.
Assuming the unstripped binaries can be found, you can get a more detailed
unwind with line number information by pasting the above example into
development/scripts/stack:
Tip: For convenience, if you've lunched
stack will be on your $PATH already so you don't need to give the
full path.
$ development/tools/stack
Reading native crash info from stdin
03-02 23:53:49.477 17951 17951 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
03-02 23:53:49.477 17951 17951 F DEBUG : Build fingerprint: 'Android/aosp_angler/angler:7.1.1/NYC/enh12211018:eng/test-keys'
03-02 23:53:49.477 17951 17951 F DEBUG : Revision: '0'
03-02 23:53:49.477 17951 17951 F DEBUG : ABI: 'arm'
03-02 23:53:49.478 17951 17951 F DEBUG : pid: 17946, tid: 17949, name: crasher >>> crasher <<<
03-02 23:53:49.478 17951 17951 F DEBUG : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xc
03-02 23:53:49.478 17951 17951 F DEBUG : r0 0000000c r1 00000000 r2 00000000 r3 00000000
03-02 23:53:49.478 17951 17951 F DEBUG : r4 00000000 r5 0000000c r6 eccdd920 r7 00000078
03-02 23:53:49.478 17951 17951 F DEBUG : r8 0000461a r9 ffc78c19 sl ab209441 fp fffff924
03-02 23:53:49.478 17951 17951 F DEBUG : ip ed01b834 sp eccdd800 lr ecfa9a1f pc ecfd693e cpsr 600e0030
03-02 23:53:49.491 17951 17951 F DEBUG :
03-02 23:53:49.491 17951 17951 F DEBUG : backtrace:
03-02 23:53:49.492 17951 17951 F DEBUG : #00 pc 0004793e /system/lib/libc.so (pthread_mutex_lock+1)
03-02 23:53:49.492 17951 17951 F DEBUG : #01 pc 0001aa1b /system/lib/libc.so (readdir+10)
03-02 23:53:49.492 17951 17951 F DEBUG : #02 pc 00001b91 /system/xbin/crasher (readdir_null+20)
03-02 23:53:49.492 17951 17951 F DEBUG : #03 pc 0000184b /system/xbin/crasher (do_action+978)
03-02 23:53:49.492 17951 17951 F DEBUG : #04 pc 00001459 /system/xbin/crasher (thread_callback+24)
03-02 23:53:49.492 17951 17951 F DEBUG : #05 pc 00047317 /system/lib/libc.so (_ZL15__pthread_startPv+22)
03-02 23:53:49.492 17951 17951 F DEBUG : #06 pc 0001a7e5 /system/lib/libc.so (__start_thread+34)
03-02 23:53:49.492 17951 17951 F DEBUG : Tombstone written to: /data/tombstones/tombstone_06
Reading symbols from /huge-ssd/aosp-arm64/out/target/product/angler/symbols
Revision: '0'
pid: 17946, tid: 17949, name: crasher >>> crasher <<<
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xc
r0 0000000c r1 00000000 r2 00000000 r3 00000000
r4 00000000 r5 0000000c r6 eccdd920 r7 00000078
r8 0000461a r9 ffc78c19 sl ab209441 fp fffff924
ip ed01b834 sp eccdd800 lr ecfa9a1f pc ecfd693e cpsr 600e0030
Using arm toolchain from: /huge-ssd/aosp-arm64/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/bin/
Stack Trace:
RELADDR FUNCTION FILE:LINE
0004793e pthread_mutex_lock+2 bionic/libc/bionic/pthread_mutex.cpp:515
v------> ScopedPthreadMutexLocker bionic/libc/private/ScopedPthreadMutexLocker.h:27
0001aa1b readdir+10 bionic/libc/bionic/dirent.cpp:120
00001b91 readdir_null+20 system/core/debuggerd/crasher.cpp:131
0000184b do_action+978 system/core/debuggerd/crasher.cpp:228
00001459 thread_callback+24 system/core/debuggerd/crasher.cpp:90
00047317 __pthread_start(void*)+22 bionic/libc/bionic/pthread_create.cpp:202 (discriminator 1)
0001a7e5 __start_thread+34 bionic/libc/bionic/clone.cpp:46 (discriminator 1)
Note: Some system libraries are built with
LOCAL_STRIP_MODULE := keep_symbols to provide usable backtraces
directly from debuggerd without taking up anywhere near as much
space as an unstripped version.
You can also stack an entire tombstone. Example:
$ stack < FS/data/tombstones/tombstone_05
This is useful if you've just unzipped a bugreport in the current directory. For more information about diagnosing native crashes and tombstones, see Diagnosing Native Crashes.
You can also use debuggerd on a running process. From the
command line, invoke debuggerd using a process ID (PID) to dump the
full tombstone to stdout. To get just the stack for every thread in
the process, include the -b or --backtrace flag.
The GNU Project debugger (GDB) is a commonly used Unix debugger.
To connect to an already-running app or native daemon, use
gdbclient with a PID. For example, to debug the process with PID
1234, run:
$ gdbclient 1234
The script sets up port forwarding, starts the appropriate
gdbserver on the device, starts the appropriate gdb on
the host, configures gdb to find symbols, and connects
gdb to the remote gdbserver.
To debug a process as it starts, use gdbserver or
gdbserver64 (for 64-bit processes). For example:
$ adb shell gdbserver :5039 /system/bin/my_test_app Process my_test_app created; pid = 3460 Listening on port 5039
Next, identify the application PID from the gdbserver output and
use it in another terminal window:
$ gdbclient <app pid>
Finally, enter continue at the gdb prompt.
Note: If you use the wrong
gdbserver, you'll get an unhelpful error message (such as
"Reply contains invalid hex digit 59").
If you want debuggerd to suspend crashed processes so you can
attach gdb, set the appropriate property:
# Android 7.0 Nougat and later. $ adb shell setprop debug.debuggerd.wait_for_gdb true # Android 6.0 Marshmallow and earlier. $ adb shell setprop debug.db.uid 999999
At the end of the usual crash output, debuggerd provides
instructions on how to connect gdb using the command:
$ gdbclient <pid>
For 32-bit ARM, if you don’t have symbols, gdb can get confused
about the instruction set it is disassembling (ARM or Thumb). To specify the
instruction set chosen as the default when symbol information is missing, set
the following property:
$ set arm fallback-mode arm # or thumb