Why does chroot get ENOENT on an existing file?

Question

;TL-DR - Answer: because the dynamic linker ld-linux-x86-64.so.2 was missing.

I have mounted a squashfs (shouldn't matter) filesystem -ro,loop at /mnt/foo.

It contains among other things the following (/mnt/foo is the mount point):

-rwxr-xr-x 1 root    root     110088 jan 17  2013 /mnt/foo/bin/ls
-rw-r--r-- 1 root    root       5212 jul 23 09:35 /mnt/foo/etc/ld.so.cache
-rw-r--r-- 1 root    root          5 jul 23 09:35 /mnt/foo/etc/ld.so.conf
-rw-r--r-- 1 root    root      31168 maj 23  2013 /mnt/foo/lib/libacl.so.1
-rw-r--r-- 1 root    root      18624 maj 20  2013 /mnt/foo/lib/libattr.so.1
-rwxr-xr-x 1 root    root    1853400 okt 12  2013 /mnt/foo/lib/libc.so.6
-rw-r--r-- 1 root    root      14664 okt 12  2013 /mnt/foo/lib/libdl.so.2
-rw-r--r-- 1 root    root     256224 mar 11  2013 /mnt/foo/lib/libpcre.so.3
-rwxr-xr-x 1 root    root     135757 okt 12  2013 /mnt/foo/lib/libpthread.so.0
-rw-r--r-- 1 root    root      31760 okt 12  2013 /mnt/foo/lib/librt.so.1
-rw-r--r-- 1 root    root     134224 maj 23  2013 /mnt/foo/lib/libselinux.so.1

/mnt/foo/etc/ld.so.conf contains a single row (including newline) with just /lib on it.

Before creating the file system, I ran ldconfig -r ${TOPDIR} where ${TOPDIR} resolved to the place that is now mounted at /mnt/foo. A strings on /mnt/foo/etc/ld.so.cache shows it contains strings like /lib/libpcre.so.3 and so on, so I don't think it is a problem with the shared libraries.

I figure it must be something silly I'm overlooking, but I cannot figure out myself why a simple chroot /mnt/foo /bin/ls doesn't work.

readelf -d /mnt/foo/bin/ls | grep NEEDED shows these libraries as needed:

 0x0000000000000001 (NEEDED)             Shared library: [libselinux.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libacl.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

Finally, strace shows this:

chroot("/mnt/foo")                      = 0
chdir("/")                              = 0
execve("/bin/ls", ["/bin/ls"], [/* 32 vars */]) = -1 ENOENT (No such file or directory)

This is the full strace chroot:

# strace -f chroot /mnt/foo /bin/ls
execve("/usr/sbin/chroot", ["chroot", "/mnt/foo", "/bin/ls"], [/* 32 vars */]) = 0
brk(0)                                  = 0x1985000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc115ac8000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=96457, ...}) = 0
mmap(NULL, 96457, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc115ab0000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\36\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1853400, ...}) = 0
mmap(NULL, 3961912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc1154e0000
mprotect(0x7fc11569d000, 2097152, PROT_NONE) = 0
mmap(0x7fc11589d000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bd000) = 0x7fc11589d000
mmap(0x7fc1158a3000, 17464, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fc1158a3000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc115aaf000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc115aad000
arch_prctl(ARCH_SET_FS, 0x7fc115aad740) = 0
mprotect(0x7fc11589d000, 16384, PROT_READ) = 0
mprotect(0x606000, 4096, PROT_READ)     = 0
mprotect(0x7fc115aca000, 4096, PROT_READ) = 0
munmap(0x7fc115ab0000, 96457)           = 0
brk(0)                                  = 0x1985000
brk(0x19a6000)                          = 0x19a6000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=8463952, ...}) = 0
mmap(NULL, 8463952, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc114ccd000
close(3)                                = 0
chroot("/mnt/foo")                      = 0
chdir("/")                              = 0
execve("/bin/ls", ["/bin/ls"], [/* 32 vars */]) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/en_US/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/en/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/charset.alias", O_RDONLY|O_NOFOLLOW) = -1 ENOENT (No such file or directory)
write(2, "chroot: ", 8chroot: )                 = 8
write(2, "failed to run command \342\200\230/bin/ls"..., 35failed to run command ‘/bin/ls’) = 35
open("/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, ": No such file or directory", 27: No such file or directory) = 27
write(2, "\n", 1
)                       = 1
close(1)                                = 0
close(2)                                = 0
exit_group(127)                         = ?
+++ exited with 127 +++

So, is this ENOENT misleading?

YES - ENOENT is a bit misleading. To me it has always meant "file not found".

When the dynamic linker cannot be found, execve() gets ENOENT.

When the booting kernel tries to load init (or linuxrc or whatever), the error I get is "Failed to execute /linuxrc (error -2). Attempting defaults...".

The problem happened due to a bug in my script that created this initial ramdisk. (it omitted libraries not pointed out by "=>" by ldd).

Two bonus questions to ponder for extra credits:

• what is linux-vdso.so.1 that ldd shows without a path?
• why does ldd show ld-linux.so without a "=>"?

Solution

The problem is /bin/ls don't just need the shared libraries, which you provided. It also needs the program that loads them; the linux loader.

To solve your problem you can copy the loader from your system (usually /lib/ld-linux.so.2) to the location of your chroot (/mnt/foo/lib/ld-linux.so.2).

Comments

[Stéphane Chazelas]

What about the dynamic linker itself (ld-linux.so or other)? Do an ldd to see what ls needs

[MattBianco]

so true, @StéphaneChazelas. It appears that I'm missing ld-linux!

[cuonglm]

Can you add strace -s 1024 to get the full message of strace.

[MattBianco]

-s 1024 only shows more of the libc.so.6 contents.

[cuonglm]

@MattBianco: so -s 0? some lines like execve("/bin/ls", ["/bin/ls"], [/* 32 vars */]) is not printed enough.

[MattBianco]

@Gnouc That's just environment vars. The option is -v to show them (and other details) in linux strace. (-e on solaris truss).

[MattBianco]

Yes. Adding /mnt/foo/lib/ld-linux-x86-64.so.2 solved it.