Moving the dynamic linker

Or, interpreter witchcraft to achieve a relatively ert nomultilib filesystem hierarchy.

Introduction

When I first began LFS, the presence of a mysterious /lib64 containing two files never sat right with me. It's not even supposed to exist at /usr/lib64!

For the instructions in LFS and BLFS to work correctly, it is imperative that this directory [/usr/lib64/] be non-existent. From time to time you should verify that it does not exist, because it is easy to create it inadvertently, and this will probably break your system.

After bricking my LFS once by naively deleting /lib64, and thereby breaking all dynamically linked binaries, I decided I wanted to get rid of this annoying directory in my root properly. Not only is it confusing, but it clutters /, which of course is absolutely unacceptable.

Setting up the system

Thence began the journey of greping through glibc and gcc source code to divine how I might change this directory. After some investigating, I discovered I could patch libc_cv_rtlddir much like LFS already patches libc_cv_slibdir in glibc. Again, through similar exploration, I set the default linker directory to /usr/lib for gcc via a sed.

# The variable to pass to glibc's configure script, around the same time the
# libc_cv_slibdir variable is passed
libc_cv_rtlddir=/usr/lib

# The gcc sed in question, to be applied around the same time the default 64-bit
# library directory name is
sed -i 's,/lib64/ld-linux,/usr/lib/ld-linux,g' gcc/config/i386/linux64.h

But it's not over yet; we still need to patch ldd to look in the correct place. While LFS also applies a patch, they only remove /usr from /usr/lib64, leaving /lib64.

patch -d "$LFS/usr/bin/" << .
--- ldd
+++ ldd
@@ -29 +29 @@
-RTLDLIST="/usr/lib64/ld-linux-x86-64.so.2 /usr/lib/ld-linux.so.2 /usr/libx32/ld-linux-x32.so.2"
+RTLDLIST="/usr/lib/ld-linux-x86-64.so.2"
.

It would be prudent to run some sanity checks at this point.

echo 'int main(){}' | "$LFS_TGT-gcc" -x c - -v -Wl,--verbose &> dummy.log
readelf -l a.out | grep 'Requesting program interpreter: /usr/lib/ld-linux-x86-64.so.2'
grep -Eo "$LFS/lib.*/S?crt[1in].*succeeded" dummy.log
grep -B3 "^ $LFS/usr/include" dummy.log
grep 'SEARCH.*/usr/lib' dummy.log | sed 's|; |\n|g' | grep '='
grep "/lib.*/libc.so.6 " dummy.log | grep succeeded
grep found dummy.log | grep "$LFS/usr/lib/ld-linux-x86-64.so.2"
rm -v a.out dummy.log

Those checks differ slightly from the ones LFS runs in that, again, we're looking for a program interpreter under /usr/lib instead of /lib64. The tweaks to the toolchain must be applied each time glibc or gcc is rebuilt.

Once we've compiled stage 2 LFS, it's time to chroot in and build stage 3. But there's a problem -- the utilities we cross-compiled from our host system have their dynamic linker hardcoded to /lib64/ld-linux-x86-64.so.2. To fix this, we call upon patchelf, my beloved.

find "$LFS/usr" -type f -executable | while read -r elf; do
    if head -c4 "$elf" | grep -qF ELF; then
        interp="$(patchelf --print-interpreter "$elf" 2>/dev/null || true)";
        if echo "$interp" | grep -qF "/lib64/ld-linux-x86-64.so.2"; then
            patchelf --set-interpreter "/usr/lib/ld-linux-x86-64.so.2" "$elf"
            printf "patched interpreter for %s\n" "$elf"
        fi
    fi
done

Finally we can chroot in and build out the stage 3 system. Once again, it's necessary to continue applying the build tweaks for gcc and glibc from before.

Assuming all has gone smoothly, the final system might look a little like this (note i've also moved /root into /home).

$ cd $LFS

$ ls -l
Octal Permissions Size User Date Modified Name
0777  lrwxrwxrwx     - root  3 Nov 23:26  bin -> usr/bin/
0755  drwxr-xr-x     - root  3 Nov 23:51  dev/
0755  drwxr-xr-x     - root  3 Nov 23:54  etc/
0755  drwxr-xr-x     - root  3 Nov 23:51  home/
0777  lrwxrwxrwx     - root  3 Nov 23:26  lib -> usr/lib/
0755  drwxr-xr-x     - root  3 Nov 23:51  proc/
1777  drwxrwxrwt     - root  3 Nov 23:51  run/
0555  dr-xr-xr-x     - root  3 Nov 23:51  sys/
1777  drwxrwxrwt     - root  3 Nov 23:54  tmp/
0755  drwxr-xr-x     - root  3 Nov 23:40  usr/
0755  drwxr-xr-x     - root  3 Nov 23:51  var/

$ patchelf --print-interpreter bin/ld
/usr/lib/ld-linux-x86-64.so.2

Maintenance

For maintenance, patchelf and libtree are two indispensable utilities for troubleshooting dynamic linking on such a system.

Dynamically linked binaries compiled elsewhere will not work here until their interpreter is patched.

patchelf --set-interpreter /usr/lib/ld-linux-x86-64.so.2

For setting up a rust toolchain with rustup, I found it necessary to briefly create /lib64 to satisfy some internal checks and to use the latest commit version of patchelf to dodge an assertion that rustup trips in patchelf 0.18.0. Once the binaries had been installed, I then overwrote their interpreters with patchelf.