How to use libraries installed by nix at run-time?

Question

I'm using nix in "single-user mode" in a system where I'm not the root (see below for a description of my nix setup).

I wanted to quickly run one of my binaries which is dynamically linked with a library which is absent in the system.

So, I've installed the library with nix:

$ nix-env -qa 'gmp'
gmp-4.3.2
gmp-5.1.3
$ nix-env -i gmp-5.1.3

But the library is still not found by the linker:

$ ldd -r ../valencies 
../valencies: /lib64/libc.so.6: version `GLIBC_2.15' not found (required by ../valencies)
../valencies: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ../valencies)
    linux-vdso.so.1 =>  (0x00007fffbbf28000)
    /usr/local/lib/libsnoopy.so (0x00007f4dcfbdc000)
    libgmp.so.10 => not found
    libffi.so.5 => /usr/lib64/libffi.so.5 (0x00007f4dcf9cc000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f4dcf748000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f4dcf540000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f4dcf33c000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4dcf11f000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f4dced8b000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f4dcfde7000)
undefined symbol: __gmpz_gcd    (../valencies)
undefined symbol: __gmpn_cmp    (../valencies)
undefined symbol: __gmpz_mul    (../valencies)
undefined symbol: __gmpz_fdiv_r (../valencies)
undefined symbol: __gmpz_fdiv_q_2exp    (../valencies)
undefined symbol: __gmpz_com    (../valencies)
undefined symbol: __gmpn_gcd_1  (../valencies)
undefined symbol: __gmpz_sub    (../valencies)
symbol memcpy, version GLIBC_2.14 not defined in file libc.so.6 with link time reference    (../valencies)
undefined symbol: __gmpz_fdiv_q (../valencies)
undefined symbol: __gmpz_fdiv_qr    (../valencies)
undefined symbol: __gmpz_add    (../valencies)
undefined symbol: __gmpz_init   (../valencies)
undefined symbol: __gmpz_ior    (../valencies)
undefined symbol: __gmpz_mul_2exp   (../valencies)
undefined symbol: __gmpz_xor    (../valencies)
undefined symbol: __gmpz_and    (../valencies)
symbol __fdelt_chk, version GLIBC_2.15 not defined in file libc.so.6 with link time reference   (../valencies)
undefined symbol: __gmpz_tdiv_qr    (../valencies)
undefined symbol: __gmp_set_memory_functions    (../valencies)
undefined symbol: __gmpz_tdiv_q (../valencies)
undefined symbol: __gmpz_divexact   (../valencies)
undefined symbol: __gmpz_tdiv_r (../valencies)
$ 

Look, it is present in the filesystem:

$ find / -name 'libgmp.so.10' 2>/dev/null 
/nix/store/mnmzq0qbrvw6dv1k2vj3cwz9ffdh05zr-user-environment/lib/libgmp.so.10
/nix/store/fnww2w81hv5v3dl9gsb7p4llb7z7krzd-gmp-5.1.3/lib/libgmp.so.10
$ 

What do I do so that libraries installed by nix are "visible"?

Probably, the standard user-installation script of nix modifies .bash_profile to add its bin/ into PATH, but does not do something analogous for libraries.

My nix setup:

The only thing I have asked the root to do for me was: mkdir -m 0755 /nix && chown ivan /nix, otherwise I've followed the standard simple nix installation procedure. So now I can use custom programs from nix packages. I couldn't do this nicely without any help from the root at all, i.e., without /nix/, because /nix/ was not available for me; I could of course use another directory, but then the pre-built binary packages wouldn't be valid and all packages would have to be rebuilt, according to the nix documentation. In my case, it was simpler to ask for /nix/ for me.

Another thing I've done is adding to ~/.bash_profile:

export NIX_CONF_DIR=/nix/etc/nix

so that I can edit nix.conf. (It was supposed to be in the root-controlled /etc/ otherwise. I did it because I wanted to build-max-jobs and build-cores settings in it.)

Solution 1

TL;DR

The working solution is using patchelf (if you have to deal with non-matching glibc versions: in the host system and the one nix libs have been linked with), see the second half of my story.

Trying the usual approach

Trying to use LD_LIBRARY_PATH

Well, I have set up an environment variable for this in ~/.bash_profile:

NIX_LINK=/home/ivan/.nix-profile
export LD_LIBRARY_PATH="$NIX_LINK"/lib

but that's not all!

Now there are problems with linking with different versions of libc:

$ ldd -r ../valencies 
../valencies: /lib64/libc.so.6: version `GLIBC_2.15' not found (required by ../valencies)
../valencies: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ../valencies)
../valencies: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /home/ivan/.nix-profile/lib/libgmp.so.10)
    linux-vdso.so.1 =>  (0x00007fff365ff000)
    /usr/local/lib/libsnoopy.so (0x00007f56c72e6000)
    libgmp.so.10 => /home/ivan/.nix-profile/lib/libgmp.so.10 (0x00007f56c7063000)
    libffi.so.5 => /usr/lib64/libffi.so.5 (0x00007f56c6e54000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f56c6bd0000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f56c69c7000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f56c67c3000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f56c65a6000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f56c6211000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f56c74f1000)
symbol memcpy, version GLIBC_2.14 not defined in file libc.so.6 with link time reference    (/home/ivan/.nix-profile/lib/libgmp.so.10)
symbol memcpy, version GLIBC_2.14 not defined in file libc.so.6 with link time reference    (../valencies)
symbol __fdelt_chk, version GLIBC_2.15 not defined in file libc.so.6 with link time reference   (../valencies)
$ 

Sorting out 2 versions of glibc

The most surprizing error here is:

symbol memcpy, version GLIBC_2.14 not defined in file libc.so.6 with link time reference    (/home/ivan/.nix-profile/lib/libgmp.so.10)

because nix must have installed the version of glibc which is used by its libgmp!

And indeed, the glibc from nix is there:

$ ldd -r /home/ivan/.nix-profile/lib/libgmp.so.10
    linux-vdso.so.1 =>  (0x00007fff0f1ff000)
    /usr/local/lib/libsnoopy.so (0x00007f06e9919000)
    libc.so.6 => /nix/store/93zfs0zzndi7pkjkjxawlafdj8m90kg5-glibc-2.20/lib/libc.so.6 (0x00007f06e957c000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f06e9371000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f06e9da7000)
symbol _dl_find_dso_for_object, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference (/nix/store/93zfs0zzndi7pkjkjxawlafdj8m90kg5-glibc-2.20/lib/libc.so.6)
/home/ivan/.nix-profile/lib/libgmp.so.10: error while loading shared libraries: __vdso_time: invalid mode for dlopen(): Invalid argument
$ 

Probably, glibc was not available to the user, so when I ran my binary, the system's glibc was loaded first. Proof:

$ ls ~/.nix-profile/lib/*libc*
ls: cannot access /home/ivan/.nix-profile/lib/*libc*: No such file or directory
$ 

Ok, we can try to make glibc visible to the user, too:

$ nix-env -i glibc

Then everything is bad:

$ ldd -r ../valencies 
/bin/bash: error while loading shared libraries: __vdso_time: invalid mode for dlopen(): Invalid argument
$ /bin/echo ok
/bin/echo: error while loading shared libraries: __vdso_time: invalid mode for dlopen(): Invalid argument
$ 

So, it seems to be not an easy job if you want to load libraries from nix when running your own binaries...

For now, I'm commenting out

export LD_LIBRARY_PATH="$NIX_LINK"/lib

and doing in the shell session:

$ unset LD_LIBRARY_PATH
$ export LD_LIBRARY_PATH

Need to think more. (Read about __vdso_time: invalid mode for dlopen(): having another glibc in LD_LIBRARY_PATH is expected to crash, because your ld-linux-x86-64.so.2 will not match your libc.so.6. Having multiple versions of glibc on a single system is possible, but slightly tricky, as explained in this answer.)

The needed solution: patchelf

So, the path to the dynamic linker is hard-coded in the binary. And the dynamic linker being used is from the system (from the host glibc), not from nix. And because the dynamic linker doesn't match the glibc which we want and need to use, it doesn't work.

A simple and working solution is patchelf.

patchelf --set-interpreter /home/ivan/.nix-profile/lib/ld-linux-x86-64.so.2 ../valencies

After that, it works. You still need to fiddle with LD_LIBRARY_PATH though.

$ LD_LIBRARY_PATH=/home/ivan/.nix-profile/lib:/lib64/:/usr/lib64/ ../valencies

If--like in my imperfect case--some of the libraries are taken from nix, but some are taken from the host system (because I haven't installed them with nix-env -i), you have to specify both the path to the nix libs, and to your host system libs in LD_LIBRARY_PATH (it completely overrides the default search path).

additional step: patchelf for the library search path

(from the patchelf page)

Likewise, you can change the RPATH, the linker search path embedded into executables and dynamic libraries:

patchelf --set-rpath /opt/my-libs/lib:/foo/lib program

This causes the dynamic linker to search in /opt/my-libs/lib and /foo/lib for the shared libraries needed by program. Of course, you could also set the environment variable LD_LIBRARY_PATH, but that’s often inconvenient as it requires a wrapper script to set up the environment.

Solution 2

In addition to the Nix's “single-user mode”, I provide an answer for NixOS users. You usually can't run binary files on NixOS.

If you install packages locally using nix-env -i, all your .so files are stored in ~/.nix-profile/lib/.

If you install packages globally by specifying them in /etc/nixos/configuration.nix, your corresponding .so files can be found in /nix/var/nix/profiles/system/sw/lib/. More correctly, only symlinks to corresponding files somewhere in /nix/store/ are in that directory.

So if you install packages globally, Ivan Zakharyaschev's solution becomes:

$ patchelf --set-interpreter /nix/var/nix/profiles/system/sw/lib/ld-linux-x86-64.so.2 ./YOUREXECUTABLE
$ LD_LIBRARY_PATH=/nix/var/nix/profiles/system/sw/lib ./YOUREXECUTABLE

For the first command to work you'll have to install glibc globally. You can also modify the second command if you have packages installed both globally and per-user:

$ LD_LIBRARY_PATH=/home/YOURUSERNAME/.nix-profile/lib:/nix/var/nix/profiles/system/sw/lib ./YOUREXECUTABLE

It might be that the needed .so file simply isn't installed in the system, so you'll have an error like:

./YOUREXECUTABLE: error while loading shared libraries: libX11.so.6: cannot open shared object file: No such file or directory

I'm not sure how to find a corresponding package for a missing file in general, but you can google the name of the .so file and install the corresponding package and try to run your executable with a custom LD_LIBRARY_PATH again.