Library path order for alternate glibc dynamic linker (ld.so)

Question

I need to use an alternate glibc version, newer than the one installed on my system (2.18 vs 2.15). Several related issues are covered here and here. The specific question I'm asking here is the following:

I set up the library path of the new dynamic linker (ld-2.18.so) so that the new libc (libc-2.18.so) is found ahead of the old libc (libc-2.15.so). However, when I try to run a program with the new ld, the old version of libc is picked up, generating a SEGV. Why is that happening?

Note: I know this can be fixed by using --rpath at compile time or LD_LIBRARY_PATH at run time. However, I would still like to understand why one of these is still needed.

The details follow:

I downloaded glibc-2.18 and built it at /opt/glibc-2.18. By default, the file /opt/glibc-2.18/etc/ld.so.conf is missing. I created it, and updated the library cache of the new glibc as follows. I emphasize that: the new libc is found before the old libc:

$ cat /opt/glibc-2.18/etc/ld.so.conf
/opt/glibc-2.18/lib
/usr/local/lib
/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/mesa
/lib
/usr/lib
$ /opt/glibc-2.18/sbin/ldconfig -v |& grep -E '^[^'$'\t'']|libc\.'
/opt/glibc-2.18/sbin/ldconfig: Path `/opt/glibc-2.18/lib' given more than once
/opt/glibc-2.18/sbin/ldconfig: Can't stat /opt/glibc-2.18/lib64: No such file or directory
/opt/glibc-2.18/sbin/ldconfig: Can't stat /opt/glibc-2.18/libx32: No such file or directory
/opt/glibc-2.18/lib:
        libc.so.6 -> libc-2.18.so
/usr/local/lib:
/lib/x86_64-linux-gnu:
        libc.so.6 -> libc-2.15.so
/usr/lib/x86_64-linux-gnu:
/usr/lib/x86_64-linux-gnu/mesa:
/lib:
/usr/lib:

Then, I created a simple C program:

$ cat <<EOF >a.c
> #include <stdio.h>
> int main()
> {
>     fprintf(stdout, "ok\n");
>     return 0;
> }
> EOF
$ g++ a.c
$ file a.out
a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0x43b8484e3910072375d68418cb6327478266c0e9, not stripped
$ ldd a.out
    linux-vdso.so.1 =>  (0x00007fffd7ffe000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa7c47bd000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa7c4b9b000)
$ readelf -a a.out | grep lib
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
000000601000  000100000007 R_X86_64_JUMP_SLO 0000000000000000 __libc_start_main + 0
     1: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.2.5 (2)
    46: 00000000004005f0     2 FUNC    GLOBAL DEFAULT   13 __libc_csu_fini
    52: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@@GLIBC_
    57: 0000000000400560   137 FUNC    GLOBAL DEFAULT   13 __libc_csu_init
  000000: Version: 1  File: libc.so.6  Cnt: 1
$ objdump -x a.out | grep -A3 Version
Version References:
  required from libc.so.6:
    0x09691a75 0x00 02 GLIBC_2.2.5

As seen above, this program has the old ld hard-coded inside. I can forcefully run it with the new ld, and I expect the path of the new ld to be used (you can see the new ld.so.cache being opened). However, for some reason I'm trying to understand, the old libc is found before the new libc, generating a SEGV:

$ /opt/glibc-2.18/lib/ld-2.18.so ./a.out
Segmentation fault (core dumped)
$ strace /opt/glibc-2.18/lib/ld-2.18.so ./a.out |& grep open
open("./a.out", O_RDONLY|O_CLOEXEC)     = 3
open("/opt/glibc-2.18/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3

I can also compile with the new library and bake-in the new ld as follows:

$ g++ -L/opt/glibc-2.18/lib -Wl,--dynamic-linker=/opt/glibc-2.18/lib/ld-2.18.so a.c -o a.2.18.out
$ file a.2.18.out
a.2.18.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0x25ab43f3d29b49fa21385a15e43325e9fb904e81, not stripped
$ ldd a.2.18.out
    linux-vdso.so.1 =>  (0x00007fffa68da000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9df5cbe000)
    /opt/glibc-2.18/lib/ld-2.18.so => /lib64/ld-linux-x86-64.so.2 (0x00007f9df609c000)
$ readelf -a a.2.18.out | grep lib
      [Requesting program interpreter: /opt/glibc-2.18/lib/ld-2.18.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
000000601000  000100000007 R_X86_64_JUMP_SLO 0000000000000000 __libc_start_main + 0
     1: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.2.5 (2)
    54: 0000000000400600     2 FUNC    GLOBAL DEFAULT   13 __libc_csu_fini
    60: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@@GLIBC_
    65: 0000000000400570   137 FUNC    GLOBAL DEFAULT   13 __libc_csu_init
  000000: Version: 1  File: libc.so.6  Cnt: 1
$ objdump -x a.2.18.out | grep -A3 Version
Version References:
  required from libc.so.6:
    0x09691a75 0x00 02 GLIBC_2.2.5

Still, if I try to run the new program the same thing happens, the old libc is being used instead of the new libc:

$ ./a.2.18.out
Segmentation fault (core dumped)
$ strace ./a.2.18.out |& grep open
open("/opt/glibc-2.18/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3

With either executable, specifying LD_LIBRARY_PATH=/opt/glibc-2.18/lib makes it work. However, my question here is why that is still needed, given that the path of the new ld is configured at the beginning to pick up the new libc ahead of the old libc.

Solution

I got it, the issue was with the OS ABI version. That's the number indicated by file, such as:

$ file /lib/x86_64-linux-gnu/libc-2.15.so | grep -o "for GNU/Linux [0-9.]*"
for GNU/Linux 2.6.24

When glibc is configured with nothing other than --prefix, it builds by default with an ABI version smaller(!!) (in my case, 2.6.16) than the default on the system (2.6.24). So libc-2.18 has ABI version smaller than libc-2.15.

When ldconfig finds 2 versions of libc.so.6 with different ABI numbers, it places them in ld.so.cache in order of descending ABI number, not in order of appearance. This can be checked by swapping their locations, rebuilding the cache (with ldconfig), and listing cache contents (with ldconfig -p). Only when 2 libc.so.6 files have the same ABI version, do they get placed in the cache in order of appearance.

Configuring glibc with --enable-kernel=2.6.24 causes it to use the same ABI version as the system, which in turn fixes the resolution issues in the question statement, without the need for an explicit --rpath or LD_LIBRARY_PATH.