Can anything useful be done with a truncated core?

Question

We have processes written in a mix of Python, Java and C++ that core dump from time to time. They allocate more memory in chunks as needed during runtime, and are known to crash when their allocation tips over 4G (I guess the return value of the malloc() isn't checked).

However the core dumps produced are truncated, according to GDB - they are unlimited in size in the OS, and on-disk they vary between 2-3.8G in size.

GDB observes that the size doesn't match what it expects (presumably including the failed alloc?) and gives up - but in 3.8G of data there surely must be something of interest? Possibly even the entire stack I need for a backtrace!

How can I persuade GDB to at least try, or is there an alternate tool that can extract something from a truncated core?

Solution

This blurb on the Sun Studio 12 website would seem to imply that they're basically useless.

excerpt - http://docs.oracle.com/cd/E19205-01/819-5257/blabs/index.html

If Your Core File Is Truncated

If you have problems loading a core file, check whether you have a truncated core file. If you have the maximum allowable size of core files set too low when the core file is created, then dbx cannot read the resulting truncated core file. In the C shell, you can set the maximum allowable core file size using the limit command (see the limit(1) man page). In the Bourne shell and Korn shell, use the ulimit command (see the limit(1) man page). You can change the limit on core file size in your shell start-up file, re-source the start-up file, and then rerun the program that produced the core file to produce a complete core file.

If the core file is incomplete, and the stack segment is missing, then stack trace information is not available. If the runtime linker information is missing, then the list of loadobjects is not available. In this case, you get an error message about librtld_db.so not being initialized. If the list of LWPs is missing, then no thread information, lwp information, or stack trace information is available.If you run the where command, you get an error saying the program was not “active.”

Comments

[Alen Milakovic]

What is your purpose here? Wouldn't you be better off directly debugging the process? For example, I've heard rumors one can attach gdb to a running process. How often do these crashes happen?

[JVG]

It is probably not possible to predict, as the memory that's actually needed depends on the datasets that are being processed, and there isn't a linear relationship between the size of the data and the memory needed to process it. My motivation here is to offer my users as much assistance I can when their overnight batch jobs fail, all I can tell them right now is you're trying to use too much memory but I can't help them track down which function exactly is doing it (many of the C++ and Java libraries come from elsewhere, my users are writing Python glue code mostly).

[Alen Milakovic]

I see. You are not running or writing the code yourself? When you say "my users", what is your role/function in this? Without hands-on contact, I think these problems can be difficult to resolve. Consider hanging out in the chat room. This can be a reasonable place to discuss problems; when people are around and paying attention, that is.

[JVG]

I am a sysadmin, my users are writing mainly numerical code; they are good at maths but not necessarily "technical". The code is run using a distributed filesystem and a job scheduler. The problem I am asking here is, is there anything useful I can get from a 3.8G core dump, when GDB thinks it is truncated? I am comfortable enough writing C myself if necessary.

[Alen Milakovic]

I'm not familar with core files myself, so I can't offer a suggestion there. I was suggesting alternative methods of debugging. Generally it is possible to avoid exhausting memory by freeing memory periodically. It should be possible to figure out where the problems lie without having to look at core dumps, unless it is very complicated. Again, feel free to stop by our chat room, to, er, chat. Stack Overflow might have more programming expertise, but we're possibly friendlier. :-)

[JVG]

This would seem to be the important caveat the stack segment is missing, then stack trace information is not available. If the runtime linker information is missing, then the list of loadobjects is not available <--- just by sheer probability there should be some of this information in a 3.8G core file!