Thread specific data from linux core dump

linux gdb coredump

11,751

Solution 1

If you're debugging a live program, you can:

print pthread_getspecific(i)

If you have access to the pthread_t of the thread, you can:

print ((struct pthread*)pth)->specific[i/32][i%32]

where i in the index you want and pth is the pthread_t. See nptl/pthread_getspecific.c in the glibc sources.

To do this without calling a function, you need to find the struct pthread. On x86-64, it's stored in the fs base, which is set using arch_prctl(ARCH_SET_FS_BASE, ...). I don't know how to access this from gdb, but you can get it with eu-readelf. Run eu-readelf --notes core_file and look through the records for fs.base. That number is the pthread_t value. (To figure out which one it is, you can match up the pid field in the same record with the LWP shown in gdb's info threads command.)

Good luck!

Solution 2

As far as I know, there is no command in gdb to get a pointer to data stored via pthread_setspecific(). However, there are a few options to obtain the memory address:

Examine each thread's backtrace, checking each frame to see if the result of pthread_getspecific() is still on the stack.
Modify existing code to log both thread id and the result of pthread_getspecific().
Locate the thread-specific data list within the threads internal data, then use the key to find the record that will contain the address. This approach is dependent on the pthread library implementation; thus, it requires either knowledge of the pthread implementation being used or reverse engineering with a bit of patience.

Below is an demonstration with a simple program on a 32-bit machine:

g++ (GCC) 4.1.2 20080704 (Red Hat 4.1.2-48)
libpthread-2.5.so
GNU gdb Red Hat Linux (6.5-25.el5rh)

$cat example.cpp
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

void* the_thread( void* ); 
void get_position();

struct position_t
{
  int x;
  int y;
};

namespace {
  pthread_key_t position_key;

  enum {
    NUMBER_OF_THREADS = 2
  };
} // unnamed

int main(int argc, char **argv)
{
  int result = pthread_key_create( &position_key, NULL );
  printf( "pthread_key_create -- key: %u, result: %i\n",
          position_key, result );

  pthread_t threads[NUMBER_OF_THREADS];
  for (unsigned int i = 0; i < NUMBER_OF_THREADS; ++i )
  {
     // Allocate a position per threads.
     position_t* position = new position_t();

     // Set position values.
     position->x = ( 1 + i ) * 11;
     position->y = ( 1 + i ) * 13;

     // Create the thread.
     result = pthread_create( &threads[i], NULL, the_thread, position );
  }

  // Give time for threads to enter their forever loop.
  sleep( 5 );

  // Abort.
  abort();
  return 0; 
}

void* the_thread( void* position )
{
   int result = pthread_setspecific( position_key, position );

   printf( "Thread: 0x%.8x, key: %u, value: 0x%.8x, result: %i\n",
           pthread_self(), position_key, position, result );

   get_position();
   return 0;
}

void get_position()
{
   position_t* position =
        reinterpret_cast< position_t* >( pthread_getspecific( position_key ) );

   printf( "Thread: 0x%.8x, key: %u, position: 0x%.8x, x: %i, y: %i\n",
           pthread_self(), position_key, position, position->x, position->y );

   // Wait forever.
   while( true ) {};
}

$ g++ -g -lpthread example.cpp && gdb -q ./a.out 
Using host libthread_db library "/lib/libthread_db.so.1".
(gdb) r
Starting program: /tmp/a.out 
[Thread debugging using libthread_db enabled]
[New Thread -1209043248 (LWP 17390)]
pthread_key_create -- key: 0, result: 0
[New Thread -1209046128 (LWP 17393)]
Thread: 0xb7ef6b90, key: 0, value: 0x09a35008, result: 0
Thread: 0xb7ef6b90, key: 0, position: 0x09a35008, x: 11, y: 13
[New Thread -1219535984 (LWP 17394)]
Thread: 0xb74f5b90, key: 0, value: 0x09a350b0, result: 0
Thread: 0xb74f5b90, key: 0, position: 0x09a350b0, x: 22, y: 26

Program received signal SIGABRT, Aborted.
[Switching to Thread -1209043248 (LWP 17390)]
0x00377402 in __kernel_vsyscall ()

Using addresses still on the stack:

(gdb) info threads
  3 Thread -1219535984 (LWP 17394)  get_position () at example.cpp:71
  2 Thread -1209046128 (LWP 17393)  get_position () at example.cpp:71
* 1 Thread -1209043248 (LWP 17390)  0x00377402 in __kernel_vsyscall ()
(gdb) thread 3
[Switching to thread 3 (Thread -1219535984 (LWP 17394))]#0  get_position () 
 at example.cpp:71
71         while( true ) {};
(gdb) list get_position
57
58         get_position();
59         return 0;
60      }
61       
62      void get_position()
63      {
64         position_t* position =
65              reinterpret_cast< position_t* >( pthread_getspecific( 
 position_key ) );
66
(gdb) info locals
position = (position_t *) 0x9a350b0
(gdb) p position->x
$1 = 22
(gdb) p position->y
$2 = 26

Using addresses printed from stdout:

(gdb) p ((position_t*)(0x09a350b0))->x 
$3 = 22
(gdb) p ((position_t*)(0x09a350b0))->y
$4 = 26

Locate the thread-specific data list within the threads internal data:

This approach is much easier if you have the value of key and pthread_t.

I will introduce details about the pthread implementation I am using as they are needed:

pthread struct is thread descriptor structure used internally by pthread.
pthread_create() returns pthread_t, an unsigned int, that contains the address of the associated pthread struct.

First, locate the pthread struct for the thread.

(gdb) info threads
* 3 Thread -1219535984 (LWP 17394)  get_position () at example.cpp:71
  2 Thread -1209046128 (LWP 17393)  get_position () at example.cpp:71
  1 Thread -1209043248 (LWP 17390)  0x00377402 in __kernel_vsyscall ()
(gdb) thread 1
[Switching to thread 1 (Thread -1209043248 (LWP 17390))]#0  0x00377402 in
 __kernel_vsyscall ()
(gdb) bt
#0  0x00377402 in __kernel_vsyscall ()
#1  0x0080ec10 in raise () from /lib/libc.so.6
#2  0x00810521 in abort () from /lib/libc.so.6
#3  0x0804880f in main () at example.cpp:47
(gdb) frame 3
#3  0x0804880f in main () at example.cpp:47
47        abort();
(gdb) info locals
result = 0
threads = {3085921168, 3075431312}
(gdb) p/x threads[1]
$5 = 0xb74f5b90

Ignoring many of the fields, the pthread struct definition looks like:

struct pthread
{
  ...
  pid_t tid; // Thread ID (i.e. this thread descriptor).
  pid_t pid; // Process ID.
  ...
  struct pthread_key_data
  {
    uintptr_t seq;
    void *data;
  } specific_1stblock[PTHREAD_KEY_2NDLEVEL_SIZE];
  struct pthread_key_data *specific[PTHREAD_KEY_1STLEVEL_SIZE];
  ...
};

pthread_key_data.seq : contains the sequence number that should be fairly low and match __pthread_keys[key].seq.
pthread_key_data.data : contains the value provided to pthread_setspecific()
pthread.specific_1stblock is a block that is used to store thread specific data before trying to dynamically allocate more blocks.
pthread is a two-level array for thread-specific data. Index 0 will contain the memory address of pthread.specific_1stblock.
PTHREAD_KEY_2NDLEVEL_SIZE has a size of 32.

The definition gives a fairly good idea as to what to look for in memory:

An integer with a value of the pthread memory address (tid), followed by an integer with the process id (pid). This is helpful to indicate if the memory being examined is a pthread struct.
cancelhandling and flags are flags. The specific values are not important. These fields may be helpful because their values are potentially visibly distinguishable from other fields, such as those containing memory addresses or counters.
specific_1stblock is an array of size 32. If the pthread struct has been zero-initialized, then there should repeating 0s for 62~ words because the example code only has one thread-specific data position_key which has a size of two words.
specific is an array containing memory addresses. If the pthread struct has been zero-initialized, then there should repeating 0s, but the first value should be the memory address of specific_1stblock.

Print a chunk of pthread's memory:

(gdb) p/x *((int*)threads[1])@150
$6 = {0xb74f5b90, 0x9a350c8, 0xb74f5b90, 0x1, 0x377400, 0x7fb99100,
  0xcb40329e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb7ef6bd0,
  0x96b118, 0x43f2, 0x43ee, 0xb74f5be0, 0xffffffec, 0x0, 0x0, 0xb74f5470,
  0x0, 0x1, 0x9a350b0, 0x0 <repeats 62 times>, 0xb74f5bf8,
  0x0 <repeats 31 times>, 0x1000101, 0x0, 0x0, 0x0, 0xc2342345, 0xe0286,
  0x0, 0x0, 0x0, 0x0, 0x0, 0x80486ca, 0x9a350b0, 0x0 <repeats 13 times>, 
  0xb6af5000, 0xa01000}

By analyizing patterns in the memory, some words become good candidates for specific pthread fields:

0xb74f5b90, 0x9a350c8, 0xb74f5b90 (pthread.tid), 0x1, 0x377400 (pthread.pid) ...
0x1, 0x9a350b0, 0x0 <repeats 62 times> (pthread.specific_1stblock) ...
0xb74f5bf8, 0x0 <repeats 31 times>  (pthread.specific)

Some light-weight sanity checks can be done, such as checking if pthread.specific[0] contains the address of pthread.specific_1stblock:

(gdb) p/x *((int*)0xb74f5bf8)@64
$7 = {0x1, 0x9a350b0, 0x0 <repeats 62 times>} ## matches specific_1stblock

Now that pthread.specific has been identified, obtain its memory address by counting the word offset from &pthread. In this case, it is 90:

(gdb) set $specific=(int*)threads[1] + 90

Calculate the first and second index via position_key:

The index into the first array is key / PTHREAD_KEY_2NDLEVEL_SIZE.

The index into the second array is key % PTHREAD_KEY_2NDLEVEL_SIZE.

(gdb) set $index1=position_key/32
(gdb) set $index2=position_key%32

Locate the pthread_key_data for position_key:

(gdb) set $level2=(int*)*($specific + $index1)
(gdb) p/x *($level2 + (2*$index2))@2
$8 = {0x1, 0x9a350b0}

Thus:

pthread_key_data.seq = 1
pthread_key_data.data = 0x9a350b0

The first word is the seq which should match pthread_key_struct[position_key].seq. Due to dealing with raw memory, __pthread_keys will be cast to int* and pointer arithmetic will have to occur to account for the sizeof pthread_key_struct:

(gdb) p *(&((int*)&__pthread_keys)[2*position_key])@2
$9 = {1, 0}

Thus:

pthread_key_struct[position_key].seq = 1
pthread_key_struct[position_key].destr = NULL

The seq numbers match, so everything looks good. pthread_key_data.data contains the value that would be returned from pthread_getspecific( position_key ).

(gdb) set $position=(position_t*)0x9a350b0
(gdb) p $position->x
$10 = 22
(gdb) p $position->y
$11 = 26

It is technically still possible to locate thread-specific data without having knowledge of the key and pthread_t values:

If a destructor function was provided to pthread_key_create(), then its memory address may will reside within the __pthread_keys array. Examine the memory, and calculate the offset and divide by the sizeof pthread_key_struct. This should result in the index, which also happens to be the key:
```
void* destr_fn( void* );
pthread_key_create( key, destr_fn )
__pthread_keys[key].destr == destr_fn
```

If the pthread_t is unknown, it may exists within a register on the thread's stack. This may require examining various different memory addresses trying to locate a section in memory that contains the pthread struct.

(gdb) info thread
  3 Thread -1219535984 (LWP 17394)  get_position () at example.cpp:71
  2 Thread -1209046128 (LWP 17393)  get_position () at example.cpp:71
* 1 Thread -1209043248 (LWP 17390)  0x00377402 in __kernel_vsyscall ()
(gdb) thread 3
[Switching to thread 3 (Thread -1219535984 (LWP 17394))]#0 g
 get_position () at example.cpp:71
71         while( true ) {};
(gdb) bt
#0  get_position () at example.cpp:71
#1  0x0804871d in the_thread (position=0x9a350b0) at example.cpp:58
#2  0x0095c43b in start_thread () from /lib/libpthread.so.0
#3  0x008b3fde in clone () from /lib/libc.so.6
(gdb) frame 2
#2  0x0095c43b in start_thread () from /lib/libpthread.so.0
(gdb) info register
eax            0x3f     63
ecx            0xb74f52ac       -1219538260
edx            0x0      0
ebx            0x96aff4 9875444
esp            0xb74f53c0       0xb74f53c0
ebp            0xb74f54a8       0xb74f54a8
esi            0x0      0
edi            0xb74f5b90       -1219535984
eip            0x95c43b 0x95c43b <start_thread+203>
eflags         0x200286 [ PF SF IF ID ]
cs             0x73     115
ss             0x7b     123
ds             0x7b     123
es             0x7b     123
fs             0x0      0
gs             0x33     51

In this instance, the edi register contains the address of the pthread struct.

References: descr.h, pthread_key_create.c, pthread_setspecific.c, pthreadP.h, internaltypes.h

11,751

Author by

Vishwanath Sungal

Masters Student at USC

Updated on July 20, 2022

Comments

Vishwanath Sungal almost 2 years

How do I get pointer to thread's local storage or thread specific data while analyzing core dump for Linux ?

I use pthread_setspecific to store some data in the pthread's local storage.

my multi threaded program on Linux crashed, and I want to see what is stored in current running thread's local storage.

If I get pointer to thread's local storage I can use key to get the data that is stored.

Is there a command in gdb to get the pointer to thread's local storage?
Vishwanath Sungal almost 12 years

can you please explain how to identify the register which contains pthread struct. from "info register"
Vishwanath Sungal almost 12 years

i am not able to get ptherad_t value, is there any otehr way of getting that value. my initial qustion was how to get the thread struct.
Vishwanath Sungal almost 12 years

the stack is corrupted, so i cant use bt,
Tanner Sansbury almost 12 years

I believe it is implementation specific, so either consult the nptl implementation, print and analyze the memory at each address stored in the registers, or try to analyze the nptl behavior in a mock-up program similiar to example.cpp. Also, keep in mind that the registers may be invalid due to stack corruption.
Paolo Bonzini almost 9 years

Note that -1219535984 in gdb's Thread -1219535984 output is already the address of the pthread_t