Blog posts about free software,
usage, configuration and my involvement in free software
Posted by Sebastian Humenda at 11.10.2016, 12:17
Sometimes applications simply crash and then it's often hard to submit a useful bug report to get the issue resolved quickly. This article walks through the steps to get a meaningful backtrace, which can be submitted along with a bug report. It is meant as a gentle introduction to debugging on (Debian) GNU/Linux. It also discusses how to get a core dump from a system service when using systemd.
This article covers only a portion of possible bug causes: more specifically crashes of programs written in C, C++ and similar languages. It also mentions some Debian/Ubuntu-specific procedures, but can be used for other GNU/Linux systems as well.
It all began with a Debian bug squashing party in Salzburg, hosted by Conova (thanks). At the event, a group of Debian developers, maintainers and other folks gathered at a large table and started hacking. While I researched the list of bugs, it turned out that some bugs simply lack enough information to get fixed easily. So I started on these "low hanging fruits".
Getting a core dump is slightly different for ordinary user programs and system services. Both scenarios are discussed below.
Before we can start reproducing the fault, we need to prepare our shell to produce a so called core dump, as soon as the program crashes. A core dump is an image of the whole memory, which was used by the process during the crash. You can configure your shell session temporarily by entering the following:
$ ulimit -c unlimited
Running the faulty program in this "prepared" environment will lead to the creation of a new file called "core".
Now you need to find out the name of the program and run it from the prepared shell session. For instance, let's look at the program BRLTTY. This particular program needs root privileges and is a daemon service. So to run it from the shell, I would type:
sudo service brltty stop # stop the running deamon sudo brltty
Now the steps to get the program to crash have to be performed. After the crash, the shell should show something like this:
zsh: segmentation fault sudo brltty
The output might vary slightly, depending on the shell in use. This indicates some memory issue, i.e. an invalid pointer operation and it is now time to collect information for the bug report.
Obtaining a core dump from a system service is slightly more complicated, because they are not started from a user shell and hence the core dumping property must be set elsewhere. If systemd is used, the service unit (the file which tells systemd how to handle the service) needs to be extended. It is best to initialize a directory under
/etc/systemd/system/ with the name of the service unit and
.d as suffix, e.g.
sudo mkdir /etc/systemd/system/brltty.service.d
Within this directory, any file on
.conf will be parsed and will extend the service unit file. For BRLTTY, I did enter the following:
echo -e "[Service]\nLimitCORE=infinity\n" > /etc/systemd/system/brltty.service.d/core.conf
Afterwards, the configuration has to be reloaded and the service restarted:
systemctl daemon-reload systemctl restart brltty
After a crash, the core file will reside in the current working directory of the daemon. That's most likely
If you want to have more control about where core files end up and to which program they belong, add the following line to
kernel.core_pattern = /tmp/core-%e-%p
This will tell the kernel to put core dumps into /tmp and name it with the binary name and its pid. To activate the new setting, just type
sudo sysctl --system
Debug symbols are essentially names within the program, which help the programmer to identify a particular part of the source code. These are normally stripped of the binary for performance and size reasons. Therefore these need to be installed (or compiled in) additionally.
For Debian and Ubuntu, the debugging symbols can be found inadditional packages, which can be easily installed using APT.
Note: The following paragraph is slightly different on Ubuntu, please see the Ubuntu Wiki.
Add a new entry in
/etc/apt/sources.list, which looks like this:
deb http://deb.debian.org/debian-debug stretch-debug main
and alter the relese information, as appropriate.
From here on, the instructions apply for Ubuntu again.
After the repository has been added, updating the package list with
sudo apt update is required. Now it is possible to search for the package with the debugging symbols, i.e. packages ending on
dbg. For the brltty case, that would be
aptitude search brltty and then
sudo apt install brltty-dbg.
To reconstruct what happened when the program crashed, it is possible to save a full memory backup (core dump) of the running program. This can then be examined with GDB, to figure out, where the program faulted. If you have not yet installed GDB, do so using
sudo apt install gdb.
When you have a core dump in place, you can examin it with the file utility, which would output something like this:
core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/sbin/brltty', real uid: 1000, effective uid: 1000, real gid: 1000, effective gid: 1000, execfn: 'sudo brltty', platform: 'x86_64'
Note that the full path of the executable is contained in the above output. This is helpful for running GDB, as explained in the next section.
It is not a good idea to attach the core dump to a bug report, because it might get really big and it might also contain private data, even passwords and other credentials. For a lot of cases, the programmer just wants to know where an error occurred and this information is contained in a back trace. The GNU Debugger (GDB) is able to retrieve the backtrace, if you provide the full path to the binary and the core file as first and second argument, respectively. This is
gdb /sbin/brltty core for the BRLTTY example. After launching GDB, it will display a command prompt and await more commands. The following command will print the desired output:
thread apply all bt full
The output can then be attached to a bug report.
With this knowledge, you can now provide quite useful bug reports. It is not always necessary to be a programmer to help out and programmers on the other hand appreciate people, who took some care before reporting a bug.
If you plan to use your knowledge for the next Debian Bug Squashing Party, you might want to read an introduction to the Debian Bug Tracking System or an introduction to a helper program called reportbug, which eases reporting bugs greatly.