How to compile my own glibc C standard library from source and use it?

Question

I am trying to compile my own glibc. I have a directory glibc, which contain the glibc source code I downloaded from the internet. From that directory I typed mkdir ../build-glibc. Now from the build-glibc directory I typed ../glibc/configure, which performed the configuration. Now I'm not sure how to call make. I can't call it from glibc directory since it doesn't have the configuration set, neither I can call it from build-glibc, since makefile is not in that directory. How do I solve this problem?

Solution 1

The Makefile is going to exist in your build-glibc directory if the configure script finishes successfully.

If everything seems to have gone smoothly during configure and still no Makefile, then you probably missed an idiosyncrasy:

While doing a configure for glibc, it is expected that you normally provide an alternative --prefix, because installing to the default location (/usr/local) can potentially cripple the system. If you don't provide one, then you need to switch on --disable-sanity-checks.

If this is not the case either, look for a config.log file, and read its contents.

Solution 2

Setup 1: glibc without dedicated GCC

This setup might work and is quick as it does not recompile the whole GCC toolchain, just glibc.

The only problem I have with this setup is that I haven't found a nice way to use runtime objects such as crt1.o, crti.o, and crtn.o provided by our glibc, and I'm using the host ones for now. This is mentioned at: https://sourceware.org/glibc/wiki/Testing/Builds?action=recall&rev=21#Compile_against_glibc_in_an_installed_location Those objects do early setup that glibc relies on, so I wouldn't be surprised if things crashed in wonderful and awesomely subtle ways. See attempts at solving this below.

Build glibc and install locally:

git clone git://sourceware.org/git/glibc.git
cd glibc
git checkout glibc-2.32
mkdir build
cd build
export glibc_install="$(pwd)/install"
../configure --prefix "$glibc_install"
make -j `nproc`
make install -j `nproc`

Setup 1: verify the build

test_glibc.c

#define _GNU_SOURCE
#include <assert.h>
#include <gnu/libc-version.h>
#include <stdatomic.h>
#include <stdio.h>
#include <threads.h>

atomic_int acnt;
int cnt;

int f(void* thr_data) {
    for(int n = 0; n < 1000; ++n) {
        ++cnt;
        ++acnt;
    }
    return 0;
}

int main(int argc, char **argv) {
    /* Basic library version check. */
    printf("gnu_get_libc_version() = %s\n", gnu_get_libc_version());

    /* Exercise thrd_create from -pthread,
     * which is not present in glibc 2.27 in Ubuntu 18.04.
     * https://stackoverflow.com/questions/56810/how-do-i-start-threads-in-plain-c/52453291#52453291 */
    thrd_t thr[10];
    for(int n = 0; n < 10; ++n)
        thrd_create(&thr[n], f, NULL);
    for(int n = 0; n < 10; ++n)
        thrd_join(thr[n], NULL);
    printf("The atomic counter is %u\n", acnt);
    printf("The non-atomic counter is %u\n", cnt);
}

Compile and run with test_glibc.sh:

#!/usr/bin/env bash
set -eux
gcc \
  -L "${glibc_install}/lib" \
  -I "${glibc_install}/include" \
  -Wl,--rpath="${glibc_install}/lib" \
  -Wl,--dynamic-linker="${glibc_install}/lib/ld-linux-x86-64.so.2" \
  -std=c11 \
  -o test_glibc.out \
  -v \
  test_glibc.c \
  -pthread \
;
ldd ./test_glibc.out
./test_glibc.out

Command adapted from https://sourceware.org/glibc/wiki/Testing/Builds?action=recall&rev=21#Compile_against_glibc_in_an_installed_location

The program outputs the expected:

gnu_get_libc_version() = 2.32
The atomic counter is 10000
The non-atomic counter is 8674

ldd output confirms that the ldd and libraries that we've just built are actually being used as expected:

+ ldd test_glibc.out
        linux-vdso.so.1 (0x00007ffe4bfd3000)
        libpthread.so.0 => /home/ciro/glibc/build/install/lib/libpthread.so.0 (0x00007fc12ed92000)
        libc.so.6 => /home/ciro/glibc/build/install/lib/libc.so.6 (0x00007fc12e9dc000)
        /home/ciro/glibc/build/install/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fc12f1b3000)

The gcc compilation debug output shows that my host runtime objects were used, which is bad as mentioned previously, but I don't know how to work around it, e.g. it contains:

COLLECT_GCC_OPTIONS=/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/crt1.o

Setup 1: modify glibc

Now let's modify glibc with:

diff --git a/nptl/thrd_create.c b/nptl/thrd_create.c
index 113ba0d93e..b00f088abb 100644
--- a/nptl/thrd_create.c
+++ b/nptl/thrd_create.c
@@ -16,11 +16,14 @@
    License along with the GNU C Library; if not, see
    <http://www.gnu.org/licenses/>.  */

+#include <stdio.h>
+
 #include "thrd_priv.h"

 int
 thrd_create (thrd_t *thr, thrd_start_t func, void *arg)
 {
+  puts("hacked");
   _Static_assert (sizeof (thr) == sizeof (pthread_t),
                   "sizeof (thr) != sizeof (pthread_t)");

Then recompile and re-install glibc, and recompile and re-run our program:

cd glibc/build
make -j `nproc`
make -j `nproc` install
./test_glibc.sh

and we see hacked printed a few times as expected.

This further confirms that we actually used the glibc that we compiled and not the host one.

Tested on Ubuntu 20.10.

Setup 1: attempts at using the correct crt* objects

https://sourceware.org/glibc/wiki/Testing/Builds?action=recall&rev=21#Compile_against_glibc_in_an_installed_location suggests adding --sysroot to the gcc command but:

• it does not really change the objects to ours according to logs
• and it makes compilation fail with /usr/bin/ld: cannot find libgcc_s.so.1 presumably because the sysroot gets used for this GCC provided object, which we don't have at that sysroot because we only built glibc

At https://stackoverflow.com/a/66634184/895245 ZeZNiQ provides a workaround that is likely correct, by passing:

-nostartfiles

followed by all the objects. You just need to extract the correct objects from the full command with -nostartfiles and pass them manually.

For example, on my amd64 machine, the objects used were different from his 32-bit command, so this is a bit fiddly.

Bibliography:

• How do I change GCC's default search directory for crti.o?
• https://gcc.gnu.org/legacy-ml/gcc-help/2015-02/msg00016.html
• https://gcc.gnu.org/legacy-ml/gcc-help/2001-11/msg00029.html

Setup 2: crosstool-NG pristine setup

This is an alternative to setup 1, and it is the most correct setup I've achieved far: everything is correct as far as I can observe, including the C runtime objects such as crt1.o, crti.o, and crtn.o.

In this setup, we will compile a full dedicated GCC toolchain that uses the glibc that we want.

The only downside to this method is that the build will take longer. But I wouldn't risk a production setup with anything less.

crosstool-NG is a set of scripts that downloads and compiles everything from source for us, including GCC, glibc and binutils.

Yes the GCC build system is so bad that we need a separate project for that.

This setup is only not perfect because crosstool-NG does not support building the executables without extra -Wl flags, which feels weird since we've built GCC itself. But everything seems to work, so this is only an inconvenience.

Get crosstool-NG, configure and build it:

git clone https://github.com/crosstool-ng/crosstool-ng
cd crosstool-ng
git checkout a6580b8e8b55345a5a342b5bd96e42c83e640ac5
export CT_PREFIX="$(pwd)/.build/install"
export PATH="/usr/lib/ccache:${PATH}"
./bootstrap
./configure --enable-local
make -j `nproc`
./ct-ng x86_64-unknown-linux-gnu
./ct-ng menuconfig
env -u LD_LIBRARY_PATH time ./ct-ng build CT_JOBS=`nproc`

The build takes about thirty minutes to two hours.

The only mandatory configuration option that I can see, is making it match your host kernel version to use the correct kernel headers. Find your host kernel version with:

uname -a

which shows me:

4.15.0-34-generic

so in menuconfig I do:

• Operating System
  • Version of linux

so I select:

4.14.71

which is the first equal or older version. It has to be older since the kernel is backwards compatible.

Setup 2: optional configurations

The .config that we generated with ./ct-ng x86_64-unknown-linux-gnu has:

CT_GLIBC_V_2_27=y

To change that, in menuconfig do:

• C-library
• Version of glibc

save the .config, and continue with the build.

Or, if you want to use your own glibc source, e.g. to use glibc from the latest git, proceed like this:

• Paths and misc options
  • Try features marked as EXPERIMENTAL: set to true
• C-library
  • Source of glibc
    • Custom location: say yes
    • Custom location
      • Custom source location: point to a directory containing your glibc source

where glibc was cloned as:

git clone git://sourceware.org/git/glibc.git
cd glibc
git checkout glibc-2.28

Setup 2: test it out

Once you have built he toolchain that you want, test it out with:

#!/usr/bin/env bash
set -eux
install_dir="${CT_PREFIX}/x86_64-unknown-linux-gnu"
PATH="${PATH}:${install_dir}/bin" \
  x86_64-unknown-linux-gnu-gcc \
  -Wl,--dynamic-linker="${install_dir}/x86_64-unknown-linux-gnu/sysroot/lib/ld-linux-x86-64.so.2" \
  -Wl,--rpath="${install_dir}/x86_64-unknown-linux-gnu/sysroot/lib" \
  -v \
  -o test_glibc.out \
  test_glibc.c \
  -pthread \
;
ldd test_glibc.out
./test_glibc.out

Everything seems to work as in Setup 1, except that now the correct runtime objects were used:

COLLECT_GCC_OPTIONS=/home/ciro/crosstool-ng/.build/install/x86_64-unknown-linux-gnu/bin/../x86_64-unknown-linux-gnu/sysroot/usr/lib/../lib64/crt1.o

Setup 2: failed efficient glibc recompilation attempt

It does not seem possible with crosstool-NG, as explained below.

If you just re-build;

env -u LD_LIBRARY_PATH time ./ct-ng build CT_JOBS=`nproc`

then your changes to the custom glibc source location are taken into account, but it builds everything from scratch, making it unusable for iterative development.

If we do:

./ct-ng list-steps

it gives a nice overview of the build steps:

Available build steps, in order:
  - companion_tools_for_build
  - companion_libs_for_build
  - binutils_for_build
  - companion_tools_for_host
  - companion_libs_for_host
  - binutils_for_host
  - cc_core_pass_1
  - kernel_headers
  - libc_start_files
  - cc_core_pass_2
  - libc
  - cc_for_build
  - cc_for_host
  - libc_post_cc
  - companion_libs_for_target
  - binutils_for_target
  - debug
  - test_suite
  - finish
Use "<step>" as action to execute only that step.
Use "+<step>" as action to execute up to that step.
Use "<step>+" as action to execute from that step onward.

therefore, we see that there are glibc steps intertwined with several GCC steps, most notably libc_start_files comes before cc_core_pass_2, which is likely the most expensive step together with cc_core_pass_1.

In order to build just one step, you must first set the "Save intermediate steps" in .config option for the intial build:

• Paths and misc options
  • Debug crosstool-NG
    • Save intermediate steps

and then you can try:

env -u LD_LIBRARY_PATH time ./ct-ng libc+ -j`nproc`

but unfortunately, the + required as mentioned at: https://github.com/crosstool-ng/crosstool-ng/issues/1033#issuecomment-424877536

Note however that restarting at an intermediate step resets the installation directory to the state it had during that step. I.e., you will have a rebuilt libc - but no final compiler built with this libc (and hence, no compiler libraries like libstdc++ either).

and basically still makes the rebuild too slow to be feasible for development, and I don't see how to overcome this without patching crosstool-NG.

Furthermore, starting from the libc step didn't seem to copy over the source again from Custom source location, further making this method unusable.

Bonus: stdlibc++

A bonus if you're also interested in the C++ standard library: How to edit and re-build the GCC libstdc++ C++ standard library source?

Solution 3

Adding to Ciro's earlier answer/solution https://stackoverflow.com/a/52454710/4726668 :

@CiroSantilli Editing your answer returns "Suggested edit queue is full". The ldd script you are calling in the test_glibc.sh script points to the host dynamic linker: /home/ciro/glibc/build/install/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fc12f1b3000). To fix this, in test_glibc.sh, change ldd to ${glibc_install}/bin/ldd. This will require you to add the built crt*.o files as well to the script:

-nostartfiles \
${glibc_install}/lib/crti.o \
${glibc_install}/lib/crtn.o \
${glibc_install}/lib/crt1.o \

On my GNU/Linux i386/i686 (32-bit x86 arch) machine, following is my working test_glibc.sh:

#!/usr/bin/env bash
set -eux
gcc \
  -L "${glibc_install}/lib" \
  -I "${glibc_install}/include" \
  -Wl,--rpath="${glibc_install}/lib" \
  -Wl,--dynamic-linker="${glibc_install}/lib/ld-linux.so.2" \
  -std=c11 \
  -nostartfiles \
  ${glibc_install}/lib/crti.o \
  ${glibc_install}/lib/crtn.o \
  ${glibc_install}/lib/crt1.o \
  -o test_glibc.out \
  -v \
  test_glibc.c \
  -pthread \
;
${glibc_install}/bin/ldd ./test_glibc.out
./test_glibc.out

Solution 4

Last tested on Ubuntu 20.04 with glibc development version 2.33.9000 (see glibc/version.h) on 27 June 2021.

How to download and build glibc, and run its benchmarks

You can manually obtain the glibc source code here: https://www.gnu.org/software/libc/sources.html:

git clone https://sourceware.org/git/glibc.git
cd glibc
git checkout master

3rd party mirror on GitHub: https://github.com/bminor/glibc/tree/master/benchtests

See also:

1. https://kazoo.ga/a-simple-tool-to-test-malloc-performance/

If you ever wish to manually build glibc and its benchtests, do so as follows:

# IMPORTANT: begin AT THE SAME DIRECTORY LEVEL as the `glibc` source code 
# directory, NOT inside the `glibc` source code dir! In other words, if 
# you are in the correct dir, running `ls` will show the `glibc` source
# code dir (that you just cloned) inside the dir you are in.
mkdir -p glibc-build
mkdir -p glibc-install
cd glibc-build
../glibc/configure --prefix="$(realpath "../glibc-install")"
time make -j8  # build with 8 threads (jobs); on a fast laptop this takes ~3 min.
time make install # (optional: install to the `glibc-install` dir you created)

# Build the benchtests (everything inside the `glibc/benchtests` dir) too;
# see the makefile 'glibc/benchtests/Makefile' for more build commands. 
time make bench-build -j8
# Now you have this executable file you can use for malloc speed tests, for instance!: 
#       ../glibc-build/benchtests/bench-malloc-thread

# To build **and run** all glibc benchtests, do:
time make bench

References:

1. https://www.gnu.org/software/libc/manual/html_node/Configuring-and-compiling.html
2. https://kazoo.ga/a-simple-tool-to-test-malloc-performance/
3. https://github.com/f18m/malloc-benchmarks/blob/master/Makefile#L122-L129 : I learned much of this by studying this makefile target:

   $(glibc_install_dir)/lib/libc.so.6:
   @echo "Building GNU libc... go get a cup of coffee... this will take time!"
   mkdir -p $(glibc_build_dir)
   cd $(glibc_build_dir) && \
       ../glibc/configure --prefix=$(glibc_install_dir) && \
       make $(parallel_flags) && \
       make install
   [ -x $(glibc_build_dir)/benchtests/bench-malloc-thread ] && echo "GNU libc benchmarking utility is ready!" || echo "Cannot find GNU libc benchmarking utility! Cannot collect benchmark results"
   

4. How to compile my own glibc C standard library from source and use it?

_{Keywords: how to build and run glibc and its benchtests, incl. malloc benchtests, from source; build glibc from source on linux ubuntu}

View more solutions

Comments

[niCk cAMel]

is this the longest answer on SO?

[Ciro Santilli OurBigBook.com]

@niCkcAMel these are likely my longest (work intensive) answers so far: stackoverflow.com/questions/18431261/how-does-x86-paging-wor‌​k | stackoverflow.com/questions/22054578/… | stackoverflow.com/questions/26294034/…

[calvin]

Hi @CiroSantilli郝海东冠状病六四事件法轮功 Is there glibc included in gcc-9.3 gcc.gnu.org/gcc-9?

[Ciro Santilli OurBigBook.com]

@calvin I haven't heart about such changes, so AFAIK no.

[ZeZNiQ]

@CiroSantilliTRUMPBANISBAD Editing your answer returns "Suggested edit queue is full". ldd script you are calling in test_glibc.sh points to the host dynamic linker: "/home/ciro/glibc/build/install/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fc12f1b3000)". To fix this, in test_glibc.sh, change "ldd" to "${glibc_install}/bin/ldd". This will require you to add the built crt*.o files as well to the script: -nostartfiles \ ${glibc_install}/lib/crti.o \ ${glibc_install}/lib/crtn.o \ ${glibc_install}/lib/crt1.o \ See my answer: stackoverflow.com/a/66634184/4726668

[Ciro Santilli OurBigBook.com]

Thanks for this suggestion, it does look interesting. When I tried it on Ubuntu 20.10 glibc 2.32 I got errors ${glibc_install}/bin/ldd ./test_glibc.out gives not a dynamic executable and ./test_glibc.out gives ./test_glibc.out: No such file or directory. I didn't stop to investigate further.

[ZeZNiQ]

@Ciro At first glance, that "no such file or directory" error tells me that you probably copy-pasted the script as is, which was tested on 32-bit, and tried it on ur 64-bit arch machine. Please note that the dynamic linker is different for 32-bit vs 64-bit. That's why I had made a note there on my machine config, "GNU/Linux i386/i686 machine", which is a 32-bit arch.

[Ciro Santilli OurBigBook.com]

Ops, yes, now I fixed that and ./test_glibc.out worked. But the ${glibc_install}/bin/ldd ./test_glibc.out still failed for some reason.

[ZeZNiQ]

Maybe the "RTLDLIST" variable in that ldd script is pointing to the incorrect dynamic linker?