How can I read a crash file from /var/crash

Question

php-fpm crashed on us and dumped a file in

/var/crash/_usr_sbin_php5-fpm.1002.crash

There is some info in that file but what I'm after is in the section called CoreDump in a base64 encoded format. How can I read what was running at the time of the crash?

Solution 1

In case if you don't want to install a bunch of sub-dependencies for apport-retrace tool, you can unpack apport format into separate files and to use only CoreDump dump with gdb as usual.

1. apport-unpack systemGeneratedCrashReportPath.crash yourNewUnpackDirectoryHere

2. cd yourNewUnpackDirectoryHere/

3. gdb `cat ExecutablePath` CoreDump (pay attention to backticks here!)

4. bt (output actual back-trace)

   Note: apport-unpack will sometimes crash itself on unpack operation (apport seems broken all around... xD), but your CoreDump and other files will be there, just ignore it and delete all .crash files in /var/crash after you move them elsewhere in order to allow system to output new crash reports from same apps there.

Solution 2

There is a tool called apport-retrace that reads the .crash files and allows you to either fill it with a fully-symbolic stack trace or run a gdb session using the core dump. To start a gdb session, run apport-retrace -g CRASHFILE.crash. Note that you need to have the -dbg packages installed to get a good stack trace.

That being said (I'm not an expert on PHP), it might actually be something that you wrote in one of your files that is causing the crash.

Comments

[user76369]

Doesnt seem to work: # apport-retrace -g _usr_sbin_php5-fpm.1002.crash ERROR: report file does not contain one of the required fields: CoreDump DistroRelease Package ExecutablePath` # grep CoreDump _usr_sbin_php5-fpm.1002.crash CoreDump: base64

[DarkNeuron]

Edit the crash file and add the field "Package: 0" under ExecutableTimestamp.

[greuze]

Worked like a charm, without requiring to install apport-retrace, thanks!

[stamster]

@digital_infinity no?! See it for yourself. There's a large difference between echo and cat...

[digital_infinity]

@stamster You are right. Sorry I missed that there is a file with name ExecutablePath. I thought the reader must fill the executable path there.

[XavierStuvw]

See wiki.ubuntu.com/DebuggingProgramCrash for documentation

[mxmlnkn]

Instead of manually editing the crash file, the --rebuild-package-info should also work as long as you are on the same system the crash happened, I think? I get errors because of missing deb-src URIs and can't be bothered to get it to work, I'll try the apport-unpack version.

[Branton Davis]

Great answer. Just a slight correction (for other silly people who get hung up on details) that step 3 contains no tildes. I assume that you mean the backticks.

[stamster]

@BrantonDavis - indeed I meant backticks - my bad, now it's changed thanks to someone who submitted the proposal for edit!