Lightning-fast CI with nixbuild.net

Posted on March 16, 2022 by Rickard Nilsson

Today we are announcing support for remote store builds in nixbuild.net. This is an alternative way of remote building in Nix, which can improve build performance greatly for certain types of environments. The biggest performance impact can be had in CI setups, especially in hosted CI services like GitHub Actions.

By switching over our own CI setup (running on GitHub Actions) to this build mode we reduced the best-case build time of our slowest CI job from 20 minutes down to just 20 seconds, an amazing 60x speedup!

No extra setup is needed, everything is already built into Nix and the feature is available to all nixbuild.net users right now.

Read on to find out how to try this out on your own CI builds!

Remote Builders vs Stores

Usually, when Nix distributes builds, it sends them to one or more remote builder. These remote builders are just normal Nix machines, accessed over SSH by the Nix client. Since nixbuild.net tries hard to act just like a normal Nix machine from the outside, you can use nixbuild.net as a drop-in replacement for a remote builder. This is how nixbuild.net has worked from the very launch.

Here is a brief iteration of what happens when Nix uses a remote builder to perform a build:

  1. Nix checks if the build result already is available in the local store or can be fetched from any configured substituter (binary cache).

  2. If Nix needs to run the build, it will first make sure that all dependencies (inputs), and their transitive closures, are in place in the local Nix store. This might mean it has to build the inputs, if there are no existing builds to be found.

  3. Once all inputs are locally available, Nix will select a remote builder machine that can perform the build, and then check if the builder has all build inputs in place in its store. If there are any missing closures, they will either be downloaded from a substituter directly to the remote builder, or uploaded from the local store to the builder.

  4. Now, Nix asks the remote builder to perform the build. Once it is done Nix will download the build result from the builder to its local store.

We have just added a chapter in the nixbuild.net documentation that describes this process in greater detail.

If you use a service such as GitHub Actions, each run will start out with a completely empty environment. This is pretty bad when using Nix, because it means all build inputs have to be fetched on each run. Even for changes that only trigger quick builds, the total size of the build inputs might be considerable. Not only does it take time to fetch the closures from binary caches, it also takes time for Nix to unpack the closure archives (compressed nar-files) to the local file system.

If you have configured a binary cache together with your CI, where all new build results are stored, you might sometimes be lucky and commit code that doesn’t trigger any new builds at all. In this case, no build inputs needs to be fetched. However, the build output and its transitive closure will still be fetched from your binary cache to your CI. This might seem unavoidable and even desireable, but in fact you are often not interested in the build output itself when it comes to CI builds. What you really want to see is if your builds and tests pass. If you are using a remote builder like nixbuild.net, actually fetching the build output to the ephemeral CI runner is just a waste of time.

For some builds, the time spent on transporting Nix closures can add up to a considerable amount. In our own CI for nixbuild.net, we had one particularly slow build. It builds our development shell, including NixOS containers for a complete development deployment of nixbuild.net itself. It contains all software that the nixbuild.net service directly or indirectly uses. The total closure size is just shy of 10 GB. Running this build in GitHub Actions (hooked up to nixbuild.net, of course) took about 20 minutes, even when minimal actual building was required.

Today, the same build takes just 20 seconds!

Nix has a lesser-known way of distributing builds, using remote stores. It is conceptually the same thing as SSHing to a remote Nix machine and running your build on it instead of on your local machine. But Nix makes the process a bit more convenient, allowing the Nix evaluation to happen locally (where you have your sources). To use it with nixbuild.net, all you have to do is to run this:

nix build \
  --eval-store auto \
  --store ssh-ng://eu.nixbuild.net \
  ...

Now, the build process will look roughly like this instead:

  1. Nix evaluates your build as usual. During this phase, derivation files (.drv-files) will be written to your local store, and any sources that are needed for evaluation will be fetched.

  2. Nix will copy over all .drv-files for the build and its inputs to the remote Nix machine (eu.nixbuild.net in this case). Any sources used during evaluation will also be copied over (if they don’t already exist on remote machine).

  3. Now, Nix will tell the remote machine to build the top-level .drv file of the build you requested. During the build, logs will be sent over to your local console so it looks just like an ordinary build, but everything is running remotely.

  4. When the build is done, the build results will be available on the remote machine, but Nix will not fetch them. If you need to, you can fetch them explicitly, with nix copy.

As you can see, all copying of Nix closures is gone, improving performance tremendously for many types of builds. We have created a public demonstration repository where you can see more comparisons between remote store building and traditional building, and practical examples of GitHub Actions. We plan on keeping the repository updated and adding more sample projects.

It is not only CI runners that benefit from using nixbuild.net as a remote store. If you use Nix on underpowered (in terms of network, storage or CPU) machines, you might want to offload as much work as possible from them. Using a remote store for building guarantees that the local machine will only be used for Nix evaluation.

Nix itself has had support for remote store building for some time, but the support for --eval-store (which makes the whole thing practically useful for actual builds) was recently added. We were able to add support for remote store building to nixbuild.net by implementing more pieces of the nix-daemon protocol (than we had already implemented). With this new feature nixbuild.net fits into even more Nix use cases, and as we add support for even more pieces of the nix-daemon protocol we hope to find many more.

If you want to try this new feature out, you need to use Nix 2.4 or newer.

Lastly, we should add that the remote store building feature on nixbuild.net should be considered beta for the time being. There is some polishing needed in the UX (the output you see when running Nix in your shell), some known limitations, and we are also working on further performance improvements. If you try it out, your feedback is very much welcome, either through [email protected] or our issue tracker.