Subscribe to this blog

Sometimes applications simply crash and then it's often hard to submit a useful bug report to get the issue resolved quickly. This article walks through the steps to get a meaningful backtrace, which can be submitted along with a bug report. It is meant as a gentle introduction to debugging on (Debian) GNU/Linux. It also discusses how to get a core dump from a system service when using systemd.

This article covers only a portion of possible bug causes: more specifically crashes of programs written in C, C++ and similar languages. It also mentions some Debian/Ubuntu-specific procedures, but can be used for other GNU/Linux systems as well.


It all began with a Debian bug squashing party in Salzburg, hosted by Conova (thanks). At the event, a group of Debian developers, maintainers and other folks gathered at a large table and started hacking. While I researched the list of bugs, it turned out that some bugs simply lack enough information to get fixed easily. So I started on these "low hanging fruits".

Reproducing The Fault

Getting a core dump is slightly different for ordinary user programs and system services. Both scenarios are discussed below.

Ordinary User Programs

Before we can start reproducing the fault, we need to prepare our shell to produce a so called core dump, as soon as the program crashes. A core dump is an image of the whole memory, which was used by the process during the crash. You can configure your shell session temporarily by entering the following:

$ ulimit -c unlimited

Running the faulty program in this "prepared" environment will lead to the creation of a new file called "core".

Now you need to find out the name of the program and run it from the prepared shell session. For instance, let's look at the program BRLTTY. This particular program needs root privileges and is a daemon service. So to run it from the shell, I would type:

sudo service brltty stop # stop the running deamon
sudo brltty

Now the steps to get the program to crash have to be performed. After the crash, the shell should show something like this:

zsh: segmentation fault  sudo brltty

The output might vary slightly, depending on the shell in use. This indicates some memory issue, i.e. an invalid pointer operation and it is now time to collect information for the bug report.

System Services

Obtaining a core dump from a system service is slightly more complicated, because they are not started from a user shell and hence the core dumping property must be set elsewhere. If systemd is used, the service unit (the file which tells systemd how to handle the service) needs to be extended. It is best to initialize a directory under /etc/systemd/system/ with the name of the service unit and .d as suffix, e.g.

sudo mkdir /etc/systemd/system/brltty.service.d

Within this directory, any file on .conf will be parsed and will extend the service unit file. For BRLTTY, I did enter the following:

echo -e "[Service]\nLimitCORE=infinity\n" > /etc/systemd/system/brltty.service.d/core.conf

Afterwards, the configuration has to be reloaded and the service restarted:

systemctl daemon-reload
systemctl restart brltty

After a crash, the core file will reside in the current working directory of the daemon. That's most likely /.

If you want to have more control about where core files end up and to which program they belong, add the following line to /etc/sysctl.conf:

kernel.core_pattern = /tmp/core-%e-%p

This will tell the kernel to put core dumps into /tmp and name it with the binary name and its pid. To activate the new setting, just type

sudo sysctl --system

Installing Debugging Symbols

Debug symbols are essentially names within the program, which help the programmer to identify a particular part of the source code. These are normally stripped of the binary for performance and size reasons. Therefore these need to be installed (or compiled in) additionally.

For Debian and Ubuntu, the debugging symbols can be found inadditional packages, which can be easily installed using APT.

Note: The following paragraph is slightly different on Ubuntu, please see the Ubuntu Wiki.

Add a new entry in /etc/apt/sources.list, which looks like this:

deb stretch-debug main

and alter the relese information, as appropriate.

From here on, the instructions apply for Ubuntu again.

After the repository has been added, updating the package list with sudo apt update is required. Now it is possible to search for the package with the debugging symbols, i.e. packages ending on dbgsym or dbg. For the brltty case, that would be aptitude search brltty and then sudo apt install brltty-dbg.

Getting A Core Dump

To reconstruct what happened when the program crashed, it is possible to save a full memory backup (core dump) of the running program. This can then be examined with GDB, to figure out, where the program faulted. If you have not yet installed GDB, do so using sudo apt install gdb.

When you have a core dump in place, you can examin it with the file utility, which would output something like this:

core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from
'/sbin/brltty', real uid: 1000, effective uid: 1000, real gid: 1000, effective gid:
1000, execfn: 'sudo brltty', platform: 'x86_64'

Note that the full path of the executable is contained in the above output. This is helpful for running GDB, as explained in the next section.

Obtaining A Backtrace

It is not a good idea to attach the core dump to a bug report, because it might get really big and it might also contain private data, even passwords and other credentials. For a lot of cases, the programmer just wants to know where an error occurred and this information is contained in a back trace. The GNU Debugger (GDB) is able to retrieve the backtrace, if you provide the full path to the binary and the core file as first and second argument, respectively. This is gdb /sbin/brltty core for the BRLTTY example. After launching GDB, it will display a command prompt and await more commands. The following command will print the desired output:

thread apply all bt full

The output can then be attached to a bug report.

Where To Go Now?

With this knowledge, you can now provide quite useful bug reports. It is not always necessary to be a programmer to help out and programmers on the other hand appreciate people, who took some care before reporting a bug.

If you plan to use your knowledge for the next Debian Bug Squashing Party, you might want to read an introduction to the Debian Bug Tracking System or an introduction to a helper program called reportbug, which eases reporting bugs greatly.