GDB - The GNU Project Debugger

The GNU Project Debugger (GDB) is a powerful tool that allows you to analyze the execution flow of a process. GDB can help to find bugs, uncover crash errors or track the source code during execution of a development cluster. It can also be used to debug Teuthology test runs.

GET STARTED WITH GDB

Basic usage with examples can be found here. GDB can be attached to a running process. For instance, after deploying a development cluster, the process number (PID) of a ceph-osd daemon can be found in:

$ cd build
$ cat out/osd.0.pid

Attaching gdb to the process:

$ gdb ./bin/ceph-osd -p <pid>

Note

It is recommended to compile without any optimizations (-O0 gcc flag) in order to avoid elimination of intermediate values.

Stopping for breakpoints while debugging may cause timeouts, so the following configuration options are suggested:

[osd]
osd_op_thread_timeout = 1500
osd_op_thread_suicide_timeout = 1500

Debugging Teuthology Tests

src/script/ceph-debug-docker.sh can be used to analyze Teuthology failures:

$ ./ceph-debug-docker.sh <branch-name>

Refer to the script header for more information.