Ubuntu Server with Systemd – How to Get a Backtrace or Coredump

debuggingsystemdubuntu-18.04

I'm using Ubuntu 18.04 Server with systemd. Recently a program my department has developed crashed twice within a day with the following error:

Jun 07 06:33:07 xxx systemd[1]: xxx.service: Main process exited, code=killed, status=11/SEGV
Jun 07 06:33:07 xxx systemd[1]: xxx.service: Failed with result 'signal'.

I gather the next step is to get a backtrace or coredump, however I'm not sure how to do this on Ubuntu Server with systemd.

I'm not sure if I should pursue using systemd-coredump, coredumpctl, or some other utility.

Also, I'm not sure what commands to issue. For the above utilities there is plenty of documentation on various features, etc. but I can't find a concise example along the lines of:

sudo apt-get install xyz

(run x, y, z commands to get core dump)

Can anybody provide a concise example or a tutorial website that explains this well? I don't need or want to use various elaborate features, I'm just trying to get a basic core dump.

Best Answer

Take for example a relatively simple service, the chrony NTP daemon.

Install debug symbols with the dbgsym packages. Unfortunately, the ddebs repo is not in sources files by default. Also there is not great scripts to find the packages, so start by appending -dbgsym to package name.

sudo apt install chrony-dbgsym

You probably need to think about how to handle core dumps on modern Linux servers and in their case, they are considering going back to just core dump files. Personally, I get nothing useful out of apport on a server, but find coredumpctl to be useful. So, the systemd approach on Ubuntu 18.04: 

sudo systemctl stop apport
sudo systemctl mask --now apport
sudo apt install systemd-coredump
# Verify this changed the core pattern to a pipe to systemd-coredump
sysctl kernel.core_pattern

# Reproduce the crash.
sudo killall -s SIGSEGV chronyd

# List collected dumps.
coredumpctl

# Invoke debugger on the latest one.
sudo coredumpctl gdb
# systemd >= 239  the gdb verb was renamed debug. Also, select core by PID.
sudo coredumpctl debug 5809

# In GDB, the basic thing to get is a stack trace. Ask the developer what else they want.
(gdb) thread apply all bt

Starting the debugger session might look like this:

John@coredump:~$ coredumpctl
TIME                            PID   UID   GID SIG COREFILE  EXE
Sat 2019-06-08 12:55:16 UTC    5809   111   115  11 error     /usr/sbin/chronyd
John@coredump:~$ sudo coredumpctl gdb
           PID: 5809 (chronyd)
           UID: 111 (_chrony)
           GID: 115 (_chrony)
        Signal: 11 (SEGV)
     Timestamp: Sat 2019-06-08 12:55:16 UTC (1h 19min ago)
  Command Line: /usr/sbin/chronyd
    Executable: /usr/sbin/chronyd
 Control Group: /system.slice/chrony.service
          Unit: chrony.service
         Slice: system.slice
       Boot ID: c9a0a69a73d245c1ae5dfe7d491ead0a
    Machine ID: d2934a6e67f81ae0097be31003da0b31
      Hostname: coredump
       Storage: /var/lib/systemd/coredump/core.chronyd.111.c9a0a69a73d245c1ae5dfe7d491ead0a.5809.1559998516000000.lz4
       Message: Process 5809 (chronyd) of user 111 dumped core.

                Stack trace of thread 5809:
                #0  0x00007eff1ce1403f __GI___select (libc.so.6)
                #1  0x00005597867eb3be n/a (chronyd)
                #2  0x00005597867e1071 n/a (chronyd)
                #3  0x00007eff1cd1eb97 __libc_start_main (libc.so.6)
                #4  0x00005597867e127a n/a (chronyd)

GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/chronyd...Reading symbols from /usr/lib/debug/.build-id/89/dcd398c87777f4c869bfd0831215eeb8b6c7fe.debug...done.
done.
[New LWP 5809]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/chronyd'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007eff1ce1403f in __GI___select (nfds=4, readfds=readfds@entry=0x7ffc0fd73c80,
    writefds=writefds@entry=0x0, exceptfds=exceptfds@entry=0x0, timeout=timeout@entry=0x7ffc0fd73be0)
    at ../sysdeps/unix/sysv/linux/select.c:41
41      ../sysdeps/unix/sysv/linux/select.c: No such file or directory.
(gdb) thread apply all bt

Thread 1 (Thread 0x7eff1df14740 (LWP 5809)):
#0  0x00007eff1ce1403f in __GI___select (nfds=4, readfds=readfds@entry=0x7ffc0fd73c80,
    writefds=writefds@entry=0x0, exceptfds=exceptfds@entry=0x0, timeout=timeout@entry=0x7ffc0fd73be0)
    at ../sysdeps/unix/sysv/linux/select.c:41
#1  0x00005597867eb3be in SCH_MainLoop () at sched.c:747
#2  0x00005597867e1071 in main (argc=<optimized out>, argv=0x7ffc0fd73fb8) at main.c:605

In this contrived example, caught it in select() as I rudely sent a signal to a task waiting on I/O.

More complex software probably lacks other symbols, install those and maybe source code and continue debugging.

Related Topic