Sometimes applications simply crash and then it's often hard to submit a useful bug report to get the issue resolved quickly. This article walks through the steps to get a meaningful backtrace, which can be submitted along with a bug report. It is meant as a gentle introduction to debugging on (Debian) GNU/Linux. It also discusses how to get a core dump from a system service when using systemd.
This article covers only a portion of possible bug causes: more specifically crashes of programs written in C, C++ and similar languages. It also mentions some Debian/Ubuntu-specific procedures, but can be used for other GNU/Linux systems as well.
Motivation
It all began with a Debian bug squashing party in Salzburg, hosted by Conova (thanks). At the event, a group of Debian developers, maintainers and other folks gathered at a large table and started hacking. While I researched the list of bugs, it turned out that some bugs simply lack enough information to get fixed easily. So I started on these "low hanging fruits".
Reproducing The Fault
Getting a core dump is slightly different for ordinary user programs and system services. Both scenarios are discussed below.
Ordinary User Programs
Before we can start reproducing the fault, we need to prepare our shell to produce a so called core dump, as soon as the program crashes. A core dump is an image of the whole memory, which was used by the process during the crash. You can configure your shell session temporarily by entering the following:
$ ulimit -c unlimited
Running the faulty program in this "prepared" environment will lead to the creation of a new file called "core".
Now you need to find out the name of the program and run it from the prepared shell session. For instance, let's look at the program BRLTTY. This particular program needs root privileges and is a daemon service. So to run it from the shell, I would type:
sudo service brltty stop # stop the running deamon
sudo brltty
Now the steps to get the program to crash have to be performed. After the crash, the shell should show something like this:
zsh: segmentation fault sudo brltty
The output might vary slightly, depending on the shell in use. This indicates some memory issue, i.e. an invalid pointer operation and it is now time to collect information for the bug report.
System Services
Obtaining a core dump from a system service is slightly more
complicated, because they are not started from a user shell and hence
the core dumping property must be set elsewhere. If systemd is used, the
service unit (the file which tells systemd how to handle the service)
needs to be extended. It is best to initialize a directory under
/etc/systemd/system/ with the name of the service unit and .d as
suffix, e.g.
sudo mkdir /etc/systemd/system/brltty.service.d
Within this directory, any file on .conf will be parsed and will
extend the service unit file. For BRLTTY, I did enter the following:
echo -e "[Service]\nLimitCORE=infinity\n" > /etc/systemd/system/brltty.service.d/core.conf
Afterwards, the configuration has to be reloaded and the service restarted:
systemctl daemon-reload
systemctl restart brltty
After a crash, the core file will reside in the current working
directory of the daemon. That's most likely /.
If you want to have more control about where core files end up and to
which program they belong, add the following line to /etc/sysctl.conf:
kernel.core_pattern = /tmp/core-%e-%p
This will tell the kernel to put core dumps into /tmp and name it with the binary name and its pid. To activate the new setting, just type
sudo sysctl --system
Installing Debugging Symbols
Debug symbols are essentially names within the program, which help the programmer to identify a particular part of the source code. These are normally stripped of the binary for performance and size reasons. Therefore these need to be installed (or compiled in) additionally.
For Debian and Ubuntu, the debugging symbols can be found inadditional packages, which can be easily installed using APT.
Note: The following paragraph is slightly different on Ubuntu, please see the Ubuntu Wiki.
Add a new entry in /etc/apt/sources.list, which looks like this:
deb http://deb.debian.org/debian-debug stretch-debug main
and alter the relese information, as appropriate.
From here on, the instructions apply for Ubuntu again.
After the repository has been added, updating the package list with
sudo apt update is required. Now it is possible to search for the
package with the debugging symbols, i.e. packages ending on dbgsym or
dbg. For the brltty case, that would be aptitude search brltty and
then sudo apt install brltty-dbg.
Getting A Core Dump
To reconstruct what happened when the program crashed, it is possible to
save a full memory backup (core dump) of the running program. This can
then be examined with GDB, to
figure out, where the program faulted. If you have not yet installed
GDB, do so using sudo apt install gdb.
When you have a core dump in place, you can examin it with the file utility, which would output something like this:
core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from
'/sbin/brltty', real uid: 1000, effective uid: 1000, real gid: 1000, effective gid:
1000, execfn: 'sudo brltty', platform: 'x86_64'
Note that the full path of the executable is contained in the above output. This is helpful for running GDB, as explained in the next section.
Obtaining A Backtrace
It is not a good idea to attach the core dump to a bug report, because
it might get really big and it might also contain private data, even
passwords and other credentials. For a lot of cases, the programmer just
wants to know where an error occurred and this information is contained
in a back trace. The GNU Debugger (GDB) is able to retrieve the
backtrace, if you provide the full path to the binary and the core file
as first and second argument, respectively. This is
gdb /sbin/brltty core for the BRLTTY example. After launching GDB, it
will display a command prompt and await more commands. The following
command will print the desired output:
thread apply all bt full
The output can then be attached to a bug report.
Where To Go Now?
With this knowledge, you can now provide quite useful bug reports. It is not always necessary to be a programmer to help out and programmers on the other hand appreciate people, who took some care before reporting a bug.
If you plan to use your knowledge for the next Debian Bug Squashing Party, you might want to read an introduction to the Debian Bug Tracking System or an introduction to a helper program called reportbug, which eases reporting bugs greatly.
Comments