How to use debug version of libc

Question

Short version of question: How can I get gdb to use the debugging symbols for libc?

Longer version: I am debugging a program with gdb and I want to see information about a futex used by libc. However, at some point during debugging I get output such as:

Catchpoint 2 (call to syscall futex), 0x00007ffff772b73e in ?? () from /lib/libc.so.6
(gdb) bt
#0  0x00007ffff772b73e in ?? () from /lib/libc.so.6
#1  0x00007ffff767fb90 in ?? () from /lib/libc.so.6
#2  0x00007ffff767a4c0 in vfprintf () from /lib/libc.so.6
#3  0x00007ffff768565a in printf () from /lib/libc.so.6
....

When I run info sharedlibrary in gdb at the breakpoint I see:

(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0x00007ffff7dddaf0  0x00007ffff7df6704  Yes (*)     /lib64/ld-linux-x86-64.so.2
0x00007ffff7bc53e0  0x00007ffff7bd1388  Yes (*)     /lib/libpthread.so.0
0x00007ffff79ba190  0x00007ffff79bd7d8  Yes (*)     /lib/librt.so.1
0x00007ffff76538c0  0x00007ffff7766c60  Yes (*)     /lib/libc.so.6
0x00007ffff6c1fd80  0x00007ffff6c303c8  Yes (*)     /lib/libgcc_s.so.1
(*): Shared library is missing debugging information.

And when I run ldd I see:

linux-vdso.so.1 =>  (0x00007ffff7fde000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007ffff7dbf000)
librt.so.1 => /lib/librt.so.1 (0x00007ffff7bb6000)
libc.so.6 => /lib/libc.so.6 (0x00007ffff7833000)
/lib64/ld-linux-x86-64.so.2 (0x00007ffff7fdf000)

I am using Ubuntu 10.04 and I think that the version of libc with debug symbols is in /usr/lib/debug/lib. I tried setting my LD_LIBRARY_PATH variable to have this at the front of the path but that did not seem to make a difference.

I'm not completely clear on how the program chooses which shared libraries to load, whether this is set at runtime or compile time (I sort of assumed runtime but now I'm not sure). So information on how to get gdb to use the debug version of libc is appreciated.

Solution 1

I think that the version of libc with debug symbols is in /usr/lib/debug/lib. I tried setting my LD_LIBRARY_PATH variable to have this at the front of the path but that did not seem to make a difference.

These are not the droids you are looking for.

The libraries in /usr/lib/debug are not real libraries. Rather, they contain only debug info, but do not contain .text nor .data sections of the real libc.so.6. You can read about the separate debuginfo files here.

The files in /usr/lib/debug come from libc6-dbg package, and GDB will load them automatically, so long as they match your installed libc6 version. If your libc6 and libc6-dbg do not match, you should get a warning from GDB.

You can observe the files GDB is attempting to read by setting set verbose on. Here is what you should see when libc6 and libc6-dbg do match:

(gdb) set verbose on
(gdb) run
thread_db_load_search returning 0
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/lib/ld-2.11.1.so...done.
thread_db_load_search returning 0
done.
thread_db_load_search returning 0
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from system-supplied DSO at 0x7ffff7ffb000...done.
WARNING: no debugging symbols found in system-supplied DSO at 0x7ffff7ffb000.
thread_db_load_search returning 0
Reading in symbols for dl-debug.c...done.
Reading in symbols for rtld.c...done.
Reading symbols from /lib/librt.so.1...Reading symbols from /usr/lib/debug/lib/librt-2.11.1.so...done.
thread_db_load_search returning 0
... etc ...

Update:

For instance I see
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done

That implies that your GDB is not searching /usr/lib/debug. One way that could happen is if you set debug-file-directory in your .gdbinit incorrectly.

Here is the default setting:

(gdb) show debug-file-directory
The directory where separate debug symbols are searched for is "/usr/lib/debug".

Solution 2

Make sure you've installed the debug symbols for libc:

sudo apt-get install libc6-dbg

And if you're on an x64 system debugging x86 code:

sudo apt-get install libc6:i386
sudo apt-get install libc6-dbg:i386

Comments

[Gabriel Southern]

thanks for the answer to my question. It looks like I have some other problem because when I turn on verbose output I don't see gdb searching /usr/lib/debug for the symbols for libc. For instance I see Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. So I've still got some problem, but your answer is very helpful for understanding what the real problem is.

[Gabriel Southern]

thanks for the update. I had downloaded and installed gdb 7.4 from source in my home directory because it had a bug fix that solved a problem I was having with gdb 7.1, but I don't have permission to update the version of gdb on my system. Anyway the debug-files-directory was not set correctly but setting it correctly in .gdbinit seems to fix the problem.

[Craig M. Brandenburg]

Also, to step into libc functions in gdb, be sure to download the glibc source code and unpack it. Then run the gdb directory command with the path to the source. If you're running a distro that patches libc (such as Debian), then be sure to apply the same patches (such as by running debuild), else the source line numbers mayn't match.