How to run gdb against a daemon in the background?

Question

I'm trying to debug a server I wrote with gdb as it segfaults under very specific and rare conditions.

Is there any way I can make gdb run in the background (via quiet or batch mode?), follow children (as my server is a daemon and detaches from the main PID) and automatically dump the core and the backtrace (to a designated file) once the program crashes?

Solution 1

Why not just run the process interactively in a persistent screen session? Why must it be a daemon when debugging? Or just run gdb in the screen session and attach it to the running process (e.g. gdb /path/to/binary -p PID_of_binary) after it forks.

Solution 2

Assuming you have appropriate permissions, you can have gdb attach to any process. You can do it on the command line with:

gdb /path/to/binary _pid_

or from within gdb with the attach command:

attach _pid_

So, once your daemon has started, you can use either of these techniques to attach to the final PID your daemon is running as. Attaching gdb stops the process which you are tracing so you will need to issue a "continue" to restart it.

I don't know a direct way to get gdb to run arbitrary commands when the program crashes. Here is one workaround I can think of:

1. Create and register a signal handlers for SIGSEGV.
2. Tell gdb not to stop on that signal (handle SIGSEGV nostop)
3. Set a breakpoint at the first line of your signal handler.
4. Assign commands to the breakpoint from step 3

Solution 3

First, I'd setup your shell / environment to give you a core dump. In bash:

ulimit -c unlimited

Once you have the core dump, you can use gdb to examine the stack trace:

gdb /path/to/app /path/to/core/file

Solution 4

I'm not really a gdb expert but two things come to mind

1. Tracepoints which might give you the necessary information as your program runs or
2. Use gdb's remote debugging facility to debug your program while it's running as a daemon.

Solution 5

How to generate a stacktrace when my gcc C++ app crashes answer for this question should do what you want. (assuming you can make changes in your code)

View more solutions

Comments

[newpxsn]

Note that having the core file isn't the same thing as having the same process stopped under the debugger. The core file doesn't preserve information about open file descriptors or memory mapping state. So that isn't always a useful suggestion.

[Jason Orendorff]

And you can't call functions defined in the program.

[David Titarenco]

This is actually a great idea, no idea why I didn't think of this :P Thanks for the elementary solution!