Will Wilson

CEO

At the Mountains of Madness

July 10, 2024

I am forced into speech because men of science have refused to follow my advice without knowing why… Doubt of the real facts, as I must reveal them, is inevitable; yet if I suppressed what will seem extravagant and incredible there would be nothing left… In the end I must rely on the judgment and standing of the few scientific leaders who have, on the one hand, sufficient independence of thought to weigh my data on its own hideously convincing merits or in the light of certain primordial and highly baffling myth-cycles; and on the other hand, sufficient influence to deter the exploring world in general from any rash and overambitious programme in the region of those mountains of madness.

– H.P. Lovecraft, At the Mountains of Madness

tl;dr: we are open-sourcing an internal tool that solves a problem that we think many NixOS shops are likely to run into. The rest of this post is just the story of how we came to write this tool, which is totally a skippable story. But along the way we learned things which have opened up such terrifying vistas of reality that we shall either go mad from the revelation or flee from the light into the peace and safety of a new dark age. We hope you find it interesting too!

A barren landscape filled with strange ruins

We are big fans of Nix, the build system, and we use NixOS literally everywhere at our company.¹ But not everybody in the world runs NixOS, so we are frequently in the business of building native executables and shared libraries that need to run both on NixOS and on the kinds of Linux used by normal people (insofar as any Linux user is a normal person). Does that sound like it should be straightforward? Buckle up and prepare yourself for a descent into the depths of ELF program interpretation and loading. We survived this journey, we even brought back an artifact. But at what cost? At what terrible cost?

I used to think computers were straightforward. I used to think computers made sense. I could do things like compile a program. See? Like this:

$ gcc hello.c
$ ./a.out
> Hello, world!

Amazing! And of course, since I compiled my program on NixOS, when I run ldd on it I’m going to get this wonderful, comforting gobbledygook:

$ ldd a.out
> linux-vdso.so.1 (0x00007ffebf234000)
> libm.so.6 => /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/libm.so.6 (0x00007f12d80e2000)
> libpthread.so.0 => /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/libpthread.so.0 (0x00007f12d80dd000)
> libc.so.6 => /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/libc.so.6 (0x00007f12d7ef0000)
> /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/ld-linux-x86-64.so.2 => /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib64/ld-linux-x86-64.so.2 (0x00007f12d81c7000)

Those paths that start with /nix/store followed by a long pseudorandom string might look ugly to you, but to me they look like salvation. Yes, salvation from the depths of DLL Hell. You see, in ancient times dynamic linking was invented to save a few kilobytes here and there. If a bunch of different programs all needed the same library, the OS could provide just one copy of the library, and everybody could map it into their address space. This seemed like a great idea at the time, and then it led to 40 years of version conflicts, inscrutable error messages, and total dependency resolution insanity.

It was partly in the interest of slaying this monster that containers were invented. One way to think of a container is that you’re just static-linking your entire userspace, which seems excessive unless you remember the world where it was normal to see your process spitting out errors like “cannot open shared object file” when you deployed it to prod, and vowed to become a wandering hermit and never to touch a computer again. Hypothetically. NixOS is a different way to solve the same problem. Instead of wrapping an entire Linux userspace around your process like a warm parka, we just store every version of every dependency, uniquely identified by the hash of all of its dependencies, and then recursively do the same thing to those, and so on. The result is that you can have multiple versions of your entire operating system happily coexisting on the same hard drive, and a lot of other crazy tricks.

If you’ve been around the software packaging block, then you probably know the first error you’ll run into when you try to run our hello world program on Ubuntu. Yes, it has to do with glibc version symbols, the bane of every software packager and maintainer. The problem boils down to the fact that it’s not safe to have a newer version of glibc on your build machine than on your deployment machine (developers who want to build one binary that targets multiple Linuxes generally have to keep around a computer with the lowest common denominator version of glibc).² NixOS tends to have an extremely bleeding edge glibc, so this will always be a problem. Fortunately, it’s trivial on NixOS to summon another universe into existence where every program and every dynamic dependency was built with an old glibc, then do your build in that universe. Cool.

But you do that and then it still doesn’t work.

An explorer happens upon something unexpected

Here I am running ldd on my nice, new binary linked to an ancient and rotting glibc. Notice how everything is now pointing to an older glibc in the Nix store. So why is it still not working?

$ ldd a.out
> linux-vdso.so.1 (0x00007ffebf234000)
> libpthread.so.0 => /nix/store/kpsz7y412y5f95mv468bs4v5c2g9zy67-glibc-2.27/lib/libpthread.so.0 (0x00007f12d80dd000)
> libc.so.6 => /nix/store/kpsz7y412y5f95mv468bs4v5c2g9zy67-glibc-2.27/lib/libc.so.6 (0x00007f12d7ef0000)
> /nix/store/kpsz7y412y5f95mv468bs4v5c2g9zy67-glibc-2.27/lib/ld-linux-x86-64.so.2 => /nix/store/j6mwswpa6zqhdm1lm2lv9iix3arn774g-glibc-2.38-27/lib64/ld-linux-x86-64.so.2 (0x00007f3b6f66a000)

Well, let’s try running it in a nice, normal Ubuntu LTS container:

$ docker run -it -v a.out:/tmp/a.out docker.io/ubuntu:20.04
root@a00e456aafc8:/# /tmp/a.out
> bash: /tmp/a.out: No such file or directory

I beg your pardon?

root@a00e456aafc8:/# ls /tmp
> a.out

Wat.

root@9cd480a40ee7:/# ls -l /tmp
> total 5
> -rwxr-xr-x 1 root root 15504 Jun 26 01:38 a.out

Is it some crazy Docker bug? Did this turn into a character device or something?

root@9cd480a40ee7:/# stat /tmp/a.out 
>  File: /tmp/a.out
>  Size: 15504     	Blocks: 9          IO Block: 15872  regular file
> Device: 26h/38d	Inode: 33547       Links: 1
> Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
> Access: 2024-06-29 02:54:28.411329967 +0000
> Modify: 2024-06-26 01:38:50.576992464 +0000
> Change: 2024-06-26 01:38:50.576992464 +0000
> Birth: -

Maybe… a dependency resolution bug? Maybe it can’t find some shared library?

root@9cd480a40ee7:/# ldd /tmp/a.out
> linux-vdso.so.1 (0x00007fff625c4000)
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb8ed817000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb8ed625000)
> /nix/store/kpsz7y412y5f95mv468bs4v5c2g9zy67-glibc-2.27/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fb8ed98d000)

The clue is actually right there in that ldd output. Can you figure it out? Meditate on this picture of a heroic anteater facing down a nightmarish monster, and then scroll down when you have a guess.

Sometimes systems programming feels like this...

ldd is a tool that tells you about the dynamic dependencies of your program, and what they got resolved to. The library you were looking for is on the left side of each of the arrows, and the actual, resolved library from your load path is on the right side. Since we’re running this in an Ubuntu container, the libraries it finds are all in normal locations under /lib rather than zany locations under /nix/store.

ldd prints the location of ld-linux.so (the Linux ELF program loader) in the same format, but in this case it’s a total lie. When you ran ldd on the program, it was able to find a copy of ld-linux.so for you. But when you run the actual program, ld-linux.so itself is the thing that does library name resolution! It can’t very well find itself, so every ELF native binary on Linux hardcodes the location of the ld-linux.so that it should use. Thus, if you want to know the copy of ld-linux.so that your program will use, ldd gives a very misleading answer. Only the thing on the left side of the arrow matters, the thing on the right-hand side is a lie that will not matter when your program actually runs. And in our case, the hardcoded path to ld-linux.so points to a location in the Nix store that definitely doesn’t exist on Ubuntu, and that’s what leads to the “No such file or directory” error. It’s a little clearer if you look at the output of the file utility:

root@9cd480a40ee7:/# file /tmp/a.out 
> /tmp/a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/kpsz7y412y5f95mv468bs4v5c2g9zy67-glibc-2.27/lib/ld-linux-x86-64.so.2, for GNU/Linux 3.10.0, not stripped

Where does this hardcoded path to ld-linux.so come from? Well, it’s injected in the link phase of your compilation toolchain, based on information provided by your installation of the C standard library. Fortunately, the good people at NixOS have produced a handy utility called patchelf that lets you selectively modify the ELF headers of an existing executable or shared library, and patchelf has an option, --set-interpreter that lets you modify this path. So that’s it, we just need to add a call to patchelf in our toolchain, and we should be good…

…but what do we set the path to? If we set it to a location in the Nix store, this won’t work on any other kind of Linux. If we set it to the FHS standard location, it won’t work on NixOS. We want the same binary to work in both locations. Is this possible?

Yes… if you’re willing to embrace Madness.

Our first thought was that since the FHS location is fixed, we could simply configure our NixOS machines to have a symlink from that standard location to the “real” location of the ld-linux.so we wanted to use. Then, we would simply patch all the binaries we built to have the FHS loader location. This would mean they work out of the box on Ubuntu, and on NixOS they’d just require a line or two in your configuration.nix. In fact this approach is so sensible that in NixOS 24.05 they use something very much like it to print a sensible error message when you try to run a binary compiled on a different system.³

But we want to do better than that! We don’t want you to get an error message when you run a binary with an FHS loader path, we want it to actually work! Should be straightforward right? Just point the symlink at a real copy of ld-linux.so. There’s only one problem…

Which one?

...and sometimes it feels like this.

Remember the part where I said the amazing thing about NixOS is that you can have multiple versions of every library happily coexisting on your computer? On a normal version of Linux, you have one copy and one version of all of your core system libraries. So on a normal Linux you also have one version of ld-linux.so, living in a canonical location, that’s tested and confirmed to work with those libraries. But a moment ago I just showed you that you can have multiple versions of glibc on the same Nix machine, and moreover entire non-overlapping software ecosystems using those different versions of glibc. Can we guarantee that the ld-linux.so we choose will work with all of them?

Unfortunately, this is not some theoretical issue. If you go try to build your own copy of ld-linux.so, you’ll discover that it’s part of the glibc project, and that it’s versioned together with glibc. There’s no documentation anywhere about what the rules are around using mismatched glibc/ld-linux.so versions, but we’ve determined via exhaustive experimentation that it is not guaranteed to work. So if we pick a random version of ld-linux.so out of the Nix store for our canonical symlink to point to, it might not work for some programs. In Nix jargon, this is what’s called an impurity, an arbitrary source of out-of-band state about your system that could cause a program to fail, and that’s a big no-no.

This is a tricky issue. There’s a project which I discovered while I was writing this post, called nix-ld, that is basically trying to deal with this same problem using a similar approach. They also ran into this impurity issue and decided that the best course was to force the user to specify which version of ld-linux.so to use via an environment variable. That’s a totally valid approach, and their program has some other very nice features,⁴ but what if we could somehow scan the binary and just figure out dynamically which ld-linux.so to use?

If we could figure this out dynamically, there would be an easy way to take advantage of the information. You see, there’s a way to override the program loader path hardcoded into a binary, which is just to invoke the loader directly as the first argument of your command (even before the program you’re trying to run). So for instance: $ foo will execute the program foo using the path to ld-linux.so in foo’s ELF headers. But $ /home/will/my_crazy_ld-linux.so foo will ignore the one that’s present in foo’s ELF headers, and directly use /home/will/my_crazy_ld-linux.so. But what this means is that you can “chain” program loaders – it’s possible to write a “meta-loader” which when asked to load a program, invokes some other loader to finish the job. To make this concrete, here’s the simplest possible example of such a meta-loader:

#!/usr/bin/env bash
/lib64/ld-linux-x86-64.so.2 $1

So if we knew what the correct loader should be, then we could have the meta-loader invoke it (and hardcode the path to the meta-loader into our program). But is there a foolproof way to decide what the correct loader is? Again, YES! There’s another special header in your executable which defines the “RPATH” and the “RUNPATH”, think of these as a way of providing hints to the dynamic linker about where to go looking for certain libraries that you’ve linked against. But for a binary compiled on NixOS, these actually contain store paths for (among other things) the glibc that your program was built against. And that same store path will contain a copy of ld-linux.so that is guaranteed (well, I hope) to work with that version of glibc. Best of all, having a bunch of extra RPATHs pointing to nonexistent Nix store locations doesn’t break anything on normal Linux. And that means we can actually have a plan that works.

That moment when you've figured it all out

Let’s consider the 4 possible cases of has/does not have a Nix-style RPATH and running on NixOS/normal linux:

Normal binary, Normal Linux: No problem, the loader is where we expect and library resolution works as normal.
Nix binary, Normal Linux: No problem, we patchelf’d the binary so it works on normal linux. Library resolution using the RPATH will fail, and then it will fall back on the system library path and find what it needs.
Nix binary, NixOS: No problem, our symlink will point it at the meta-loader, which will then read the RPATH and pick the correct “real” loader to use.
Normal binary, NixOS: ALSO NO PROBLEM!!! Assuming that your binary is able to run at all, there is some glibc it will pick up at runtime (perhaps via your shell’s PATH or LD_LIBRARY_PATH). Our symlink will point the binary at the meta-loader, which will fail to read the RPATH, and fall back on the ld-linux.so provided by the glibc installation that your binary would have used. This will exactly match the glibc that gets loaded by the process, so everything will still work.

So great, now we have a design that should work. Can we just write a little shell script to do this and act as our meta-loader?

Lol. Lmao, even.

Remember our simplest possible meta-loader? It takes your program and invokes it again using the real loader. It looked like this:

#!/usr/bin/env bash
/lib64/ld-linux-x86-64.so.2 $1

This minimal meta-loader will totally work if you invoke it directly like $ meta_loader.sh foo, and it will totally not work if you hardcode its path (or a symlink to it) in the ELF headers of a binary. Why? I dunno man, haven’t you learned not to ask these sorts of questions? This is the part of programming that feels more like archaeology. We are descending through geological strata, marveling at perfectly preserved skulls and papyri and stuff, pondering how to run our sewer line through here without triggering a lot of red tape.

Great, no shell script. Let’s write a C program. Yeah, a simple C program. C programs are great. C has a standard library. Unfortunately, if you attempt to use that standard library, your little meta-loader will fail in all sorts of gory ways.⁵ Again, we don’t have an exact diagnosis of why this is. We are in the region of the space-time continuum where we sort of shrug our shoulders and roll with things.

Great, no standard library. We will write a C program like God intended: using raw system calls, intrinsics, and a little inline assembly. Unfortunately that is not the most productive programming environment, so we cheat and have our meta-loader actually just bootstrap the loading process, and defer all the hard work to a… shell script which we exec. This totally works.⁶ Yes, after all that it really genuinely works. This program has never let us down. If you want to try it, we’ve open-sourced it here and turned it into a tiny NixOS module that’s trivial to enable. Enjoy!

We promise it will be good for your computer