executing binary file: file not found

Question

I know there are similar questions out there, but I haven't found a solution nor this exact case. The binary was built on Arch Linux using its GCC 4.7. The package works fine on the build system. The commands below were executed on:

Linux vbox-ubuntu 3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

The file in question is located here. It's a Linux 64-bit to Windows 64-bit cross-compiler. Untarring it to ~/ gives a single ~/mingw64 directory which contains everything needed.

When I try to run ~/mingw64/x86_64-w64-mingw32/bin/as this is what I get:

bash: /home/ruben/mingw64/x86_64-w64-mingw32/bin/as: No such file or directory

Running file ~/mingw64/x86_64-w64-mingw32/bin/as gives me:

/home/ruben/mingw64/x86_64-w64-mingw32/bin/as: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0x0b8e50955e7919b76967bac042f49c5876804248, not stripped

Running ldd ~/mingw64/x86_64-w64-mingw32/bin/as gives me:

    linux-vdso.so.1 =>  (0x00007fff3e367000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f2ceae7e000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2ceaac1000)
    /lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f2ceb0a8000)

I am truly at a loss. Any help is much appreciated.

EDIT: Some more details: The build system is Arch Linux (currently glibc 2.16). The output of ls -l is:

-rwxr-xr-x 2 ruben users 1506464 11 aug 23:49 /home/ruben/mingw64/bin/x86_64-w64-mingw32-as

The output of objdump -p is:

Version References:
  required from libz.so.1:
    0x0827e5c0 0x00 05 ZLIB_1.2.0
  required from libc.so.6:
    0x0d696917 0x00 06 GLIBC_2.7
    0x06969194 0x00 04 GLIBC_2.14
    0x0d696913 0x00 03 GLIBC_2.3
    0x09691a75 0x00 02 GLIBC_2.2.5

The output of ldd -v on Ubuntu 12.04 is:

    linux-vdso.so.1 =>  (0x00007fff225ff000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd525c71000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd5258b4000)
/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fd525e9b000)

Version information:
/home/ruben/mingw64/x86_64-w64-mingw32/bin/as:
    libz.so.1 (ZLIB_1.2.0) => /lib/x86_64-linux-gnu/libz.so.1
    libc.so.6 (GLIBC_2.7) => /lib/x86_64-linux-gnu/libc.so.6
    libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
    libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6
    libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libz.so.1:
    libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
    libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
    libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libc.so.6:
    ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
    ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2

The tested other OSes are Fedora 17 (glibc 2.15) and Ubuntu 12.04 (eglibc 2.15). Both zlib and glibc version requirements are met.

Solution

If I run ldd -v as on my system, I get:

./as: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by ./as)
        linux-vdso.so.1 =>  (0x00007fff89ab1000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1e4c81f000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1e4c498000)
        /lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f1e4ca6d000)

So yeah, it looks like these binaries are looking for a GLIBC_2.14 symbol, which you are presumably missing on your system. As svenx pointed out, it looks like it's searching for the memcpy@@GLIBC_2.14 symbol. Some more information on why memcpy was given a new version is described in this bug report.

Installing a new version of glibc on your target system should fix it. If you want to try to rebuild the binary to still work on the old version of glibc, you could try tricks like the one listed here. You could also maybe get by with a shim that just provides the specific version of the memcpy symbol that you need, but that gets to be a bit hacky.

---

After reading your update: you're right, that wasn't your problem. But I think I've found it: your binary is requesting the interpreter /lib/ld-linux-x86-64.so.2, which doesn't exist on Ubuntu 12.04 systems:

$ readelf -a ./as | grep interpreter
      [Requesting program interpreter: /lib/ld-linux-x86-64.so.2]

While ldd knew to find it in /lib64 instead, I suppose the kernel doesn't know that when it tries to run the binary and can't find the file's requested interpreter. You could try just running it through the interpreter manually:

$ pwd
/home/jim/mingw64/x86_64-w64-mingw32/bin
$ ./as --version
-bash: ./as: No such file or directory
$ /lib64/ld-linux-x86-64.so.2 ./as --version
GNU assembler (rubenvb-4.7.1-1-release) 2.23.51.20120808
Copyright 2012 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or later.
This program has absolutely no warranty.
This assembler was configured for a target of `x86_64-w64-mingw32'.

I'm not 100% certain this is working correctly -- on my system, running gcc this way gives a segmentation fault. But that's at least a different problem.

Comments

[tripledes]

is it just a typo or are you actually trying to run '~/mingw64/x86_64-w64-mingw32/as' ... it's missing the 'bin' folder.

[rubenvb]

@tripledes type, and corrected.

[tripledes]

weird, just tried to download, untar under my ~ and I get the exptected result. Could you provide an ls -l ~/mingw64/x86_64-w64-mingw32/bin/as ?

[svenx]

Could it be a libc version mismatch? Try running "objdump -p <path/to/as>" and inspect the "Version References" section. It could look like it depends on GLIBC_2.14, which could be considered fairly new. What's your system version? "readelf -a <path/to/as>|less" and grep for GLIBC, you'll see that it's using memcpy from 2.14. I don't know why the version would vary so much between different library calls.

[rubenvb]

That would have been nice, if this was the case. See update :(

[cheshirecatalyst]

I've updated my answer.

[rubenvb]

Wow, ok. This kind of sucks. I can't think of a decent way to work around this problem (and keep building on Arch). Do you have any brilliant ideas?

[cheshirecatalyst]

Not really. Arch is just being stupid -- the Filesystem Heirarchy Standard clearly says that libraries should be in /lib64 on amd64, and apparently Arch manually patches their gcc sources to change this, thereby guaranteeing incompatibility with every other Linux distribution out there. See the comments of this bug report for their bizarre reasoning. To me, this would be a clear warning sign to stay away from Arch Linux.

[cheshirecatalyst]

That said, you can change the interpreter location using patchelf. See this post for another person running into this problem, who claimed that patchelf worked for them.

[rubenvb]

well, as this is a package meant for redistributing, such a symlink is not really in my control. I thought Linux binaries would be more portable... guess not.

[rubenvb]

I have submitted a reopen request to the Arch bug tracker. This modification is IMHO against Arch philosophy by design, and as we both know, very much incompatible with vanilla Linux/glibc/GCC installs.