I need help debugging apache with gdb (no debugging symbols found)

Question

I'm trying to debug a segfault in PHP/Apache running on RHEL4.

This is a production server, so I'm trying to install a separate copy of Apache and PHP, and run apache through gdb.

When I load httpd through gdb and then do run -X, I get the error:

(no debugging symbols found)...
Error while reading shared library symbols:
Dwarf Error: Cannot handle DW_FORM_strp in DWARF reader.

The Apache manual says something about setting EXTRA_CFLAGS to -g, and I've tried configuring with --enable-maintainer-mode, but I don't seem to be getting far.

Solution 1

It would be difficult to debug a complex multi-process/threaded network daemon application like Apache, complete with modules (such as potentially including mod_PHP) via gdb.

Why do you think the error lies within Apache's httpd, and not either an application (i.e. a PHP script) or module (e.g. mod_PHP)?

What is the exact segfault error message?

Does the segfault occur on startup, after a random while, after a certain web app is run, or a particular amount of time (about every 48 hours)?

Have you increase or added the Apache httpd Log Level directive in the Apache configuration file to get more information? Try info or debug for the most information.

I've been using Apache httpd for nearly 15 years, and I have never had to debug the Apache web server daemon itself, so I'm surprised you appear to need to debug it using gdb.

---

(no debugging symbols found)

This means that the debugging symbols are not included in the binary executable. Typically the compiler didn't have debugging enabled (-g flag with gcc), and/or the install process stripped the executable file, to make it occupy less memory while running.

Error while reading shared library symbols:

This error is about the shared (dynamically linked) libraries debug symbols.

Dwarf Error: Cannot handle DW_FORM_strp in DWARF reader.

I haven't seen this error before, it may be occurring because Red Hat Enterprise Linux 4 is built differently. DWARF is a debugging information format. The other common format found on Linux systems is Stabs. Basically seems to be suggesting the shared libraries are also "stripped" of debugging information.

Correction: It means that debugger (gdb) cannot read / find the string pointer (strp). It may be due to GDB Bug 7358 or a compiler / debugger version mismatch.

The shared libraries often have the debugging information (symbols) in a separate package. For example, I believe RHEL the GNU Standard C Library (glibc) debug info is in a package called glibc-debug. You will need to install the debug info of the libraries that Apache httpd uses. Check the Apache rpm package's libraries dependencies for a list of libraries the executable depends upon -- rpm -qR package-name or rpm -qpR <rpm-file>.

---

i've tried configuring with --enable-maintainer-mode, but I don't seem to be getting far.

I'm sorry, but that's too vague to help you with. You need to include more useful details.

Update: Usage of automake's maintainer-mode is mostly for packaging or development, not debugging proposes.

---

Consider using a PHP debugger, like Xdebug with the PHP application or script.

---

Summarize comments

I'll try to summarize the information from the comments here:

• Generating a gdb backtrace (from PHP)
• requires PHP built with debugging (symbols included)
• PHP with debugging can be built by making it using EXTRA_CFLAGS="-g" make or
• Preferably by using configure --enable-debug (ref), followed by make

This appears to have resolved the debugging symbols.

Solution 2

Make sure the debug symbols aren't getting stripped as part of the apache installation process. It's common to run 'strip' on binaries to remove said symbols when they get copied to /usr/local/bin or wherever.

Comments

[kenorb]

Related: How to get current PHP function name in gdb?

[Declan Stewart McPartlin]

What are these packages called?

[Declan Stewart McPartlin]

This is the way the PHP documentation recommends debugging segfaults from a PHP script. bugs.php.net/bugs-generating-backtrace.php Basically I have a PHP script which is intermittently crashing, and apache is leaving this in its logs: [notice] child pid 20242 exit signal Segmentation fault (11), possible coredump in /etc/httpd

[Declan Stewart McPartlin]

As far the -g flag for gcc, I've tried to add that either as an environment variable (export EXTRA_CFLAGS), or by adding that to the make command (make EXTRA_CFLAGS="-g"), and I still can't get it to work. Obviously, I don't really know what I'm doing.

[user3789902]

maybe take a closerlook at the php code, and php apache module first.

[mctylr]

From the PHP generating backtrace information, second paragraph titled Important! : To get a backtrace with correct information you must have PHP configured with --enable-debug! So you want to build PHP, not Apache with debug information.

[mctylr]

make EXTRA_CFLAGS="-g" The EXTRA_CFLAGS is an environment variable, not a command parameter, you want to do EXTRA_CFLAGS="-g" make

[Declan Stewart McPartlin]

Yes, you are correct...I do need to compile PHP with --enable-debug. I did just that, and I recompiled apache as well. After that, I followed the instructions, and got that message from GDB. I figured EXTRA_CFLAGS was an environment variable, which is why I tried the export command shown in my comment. I also tried make EXTRA_CFLAGS="-g" as well, but that didn't seem to work either.

[Declan Stewart McPartlin]

Tried EXTRA_CFLAGS="-g" make and it worked. Thanks for the help!

[Ryan V. Bissell]

The 'strp' in DW_FORM_strp does not mean stripped; it means that the symbol attribute comes in the form of a pointer to a string (pointer to elsewhere within the symbol data.) I would -1 except that I have insufficient reputation.

[mctylr]

@RyanV.Bissell I'm glad you didn't have the reputation to downvote, because as others have indicated the answer is not incorrect, lacking of merit. Finally, as I have shown, I am also willing to fix mistakes. Thank you for the correction, it's much more productive.

[Ryan V. Bissell]

@mctylr, my downvote would have been temporary, pending your edit. Sorry for not being clear about that.