This means that you can create auth tokens that are only usable if you also can provide an OIDC JWT signed by a specific identity provider. In practice, you can now configure your GitHub Actions workflow with a nixbuild.net auth token that can only be used from within GitHub’s infrastructure. You can even lock it down to specific GitHub repositories. This is a great step forward in securing nixbuild.net’s authentication and authorization.

This new feature is not specific to GitHub, but can be used together with any OIDC provider. Additionally, it is implemented on top of the existing Biscuit support in nixbuild.net. This further validates the usefulness and flexibility that nixbuild.net’s Biscuit policies provide.

Read on to find out exactly how it all fits together!

A Usage Example

Let’s illustrate how OIDC works in nixbuild.net by working our way through a complete usage example. The repository used in this example is available for inspection at GitHub.

Our objective is to configure a GitHub Actions workflow that uses nixbuild.net to perform Nix builds. We want to make use of OIDC to secure the nixbuild.net authentication and avoid the impact a workflow auth token leak could have.

We have three steps to go through:

1. Create a new nixbuild.net auth token with the base permissions needed for our GitHub workflow.

2. Derive a new token from the original one, with a Biscuit policy that takes the OIDC context into account. This process is called attenuation and is done offline from nixbuild.net, on your local computer. Arguably, the nixbuild.net UI could offer a way to attach a Biscuit policy directly during token creation, but there is no such functionality yet.

3. Create a GitHub Actions workflow that requests an OIDC JWT from GitHub and passes it on to nixbuild.net.

Step 1: Create a Base Token

Create a new nixbuild.net auth token using the nixbuild.net admin shell or the web UI. This token defines the “upper limit” of what our workflow can do on nixbuild.net. Since you probably want your workflow to be able to run builds on nixbuild.net, add the permissions build:read, build:write, store:read and store:write, but leave out account:read and account:write. You can set the expiration time to whatever makes sense to you. It is possible to set a shorter expiration time on the attenuated token (next step) if you like.

The token generated in this step can be kept entirely offline, and you can then derive new tokens from it whenever you like. Or, you could simply throw away this base token once you have created an attenuated token. Just remember to not revoke the base token in nixbuild.net because that will implicitly revoke all derived tokens too.

If you use the web UI, creating the token should look something like this:

The created token is a long string of randomly-looking characters. Store this string to a file on your computer and treat it as a secret. If anyone gets their hands on it, they will be able to run builds on nixbuild.net, possibly incurring costs to your account.

Step 2: Attenuate the Base Token

Now we want to create a token with more restrictions than the base token. This restricted token will be used by your GitHub workflow. The process of creating a new token from an existing one is called “attenuation”, in Biscuit lingo.

To perform attenuation you need the Biscuit CLI tool, biscuit-cli. It is available in nixpkpgs:

$ nix shell nixpkgs#biscuit-cli

You can use this tool to inspect your base token (stored in the file base.token in this example):

$ biscuit inspect base.token

In the printed output you should be able to see the permissions and expiration time that you selected during token creation.

Now we’ll write a Biscuit authorization policy that restricts token usage to only work from within GitHub Actions workflows for the nixbuild/nixbuild-oidc-auth-example repository:

check if
  jwt_claim("iss", "https://token.actions.githubusercontent.com");

check if
  jwt_claim("sub", $sub),
  $sub.contains("repo:nixbuild/nixbuild-oidc-auth-example");

We add two check statements, which say what must hold true for authorization to succeed. jwt_claim refers to a Biscuit fact that is added by nixbuild.net if it has been provided an OIDC JWT and been able to verify the signature of the JWT. We’ll talk more about how the JWTs are passed to nixbuild.net in the next section.

nixbuild.net will not add all JWT claims as Biscuit facts, since it would make the authorization process computationally more expensive. But it adds the iss and sub claims, which are used above. In our policy, we first verify the iss claim. When nixbuild.net receives a JWT with an iss claim, it will use OIDC Discovery to find out which endpoint to ask for public keys to perform JWT verification. This is what makes the OIDC support in nixbuild.net generic. You could even host your own identity provider, as long as it supports OIDC Discovery.

So, when our policy checks that the iss claim is https://token.actions.githubusercontent.com, we can be sure that the provided JWT was issued by GitHub.

Furthermore, in GitHub’s reference documentation we can see that the sub claim will include the owner and name of the repo that created the OIDC JWT. We use that information to write our second policy check. GitHub encodes more information in the sub claim that can be useful for authouring Biscuit policies. Check out their documentation for details! Also be sure to read through the Biscuit language documentation to find out what functions and operators you can use in your policies! There is also a helpful Bisuit playground available.

Finally, we save our policy to a file (gha-oidc.biscuit in this example) and then use the Biscuit CLI to attach the policy to our base token, creating a new, attenuated, token:

$ biscuit attenuate --block-file gha-oidc.biscuit base.token > gha-oidc.token

If you run biscuit inspect gha-oidc.token you should see that it includes our new policy.

Lets do a quick sanity check to see that the token actually can’t be used outside GitHub. We do this by performing a HEAD request against the /builds endpoint of the nixbuild.net HTTP API, once with the base token and once with the attenuated token:

$ curl -sIH "Authorization: Bearer $(cat base.token)" https://api.nixbuild.net/builds
HTTP/2 200
server: nginx
date: Thu, 28 Aug 2025 09:31:47 GMT
content-type: application/json;charset=utf-8

$ curl -sIH "Authorization: Bearer $(cat gha-oidc.token)" https://api.nixbuild.net/builds
HTTP/2 403
server: nginx
date: Thu, 28 Aug 2025 09:32:02 GMT

Looks good! Now lets see how to use the token in an actual GitHub Actions workflow.

Step 3: Create a GitHub Workflow

The action nixbuild/nixbuild-action is the preferred way of configuring nixbuild.net in GitHub Actions workflows, and since v21 it supports generating an OIDC ID Token and passing it on to nixbuild.net automatically.

Here is a simple workflow that does this:

name: oidc-test

on: push

jobs:

  oidc-example:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - name: Install Nix
        uses: nixbuild/nix-quick-install-action@v33

      - name: Configure nixbuild.net
        uses: nixbuild/nixbuild-action@v21
        with:
          nixbuild_token: ${{ secrets.nixbuild_token }}
          oidc: true

      - name: Run a simple test build on nixbuild.net
        run: |
          export DRV_SEED="$RANDOM$RANDOM"
          nix build github:nixbuild/nixbench#write-one-file \
            --impure \
            --eval-store auto \
            --store ssh-ng://eu.nixbuild.net \
            --print-build-logs

The important pieces are the id-token: write permission (allows OIDC ID Tokens to be fetched from GitHub), the oidc: true setting for nixbuild/nixbuild-action (enables the mechanism that fetches and passes OIDC ID Tokens on to nixbuild.net) and the nixbuild_token secret (should be set to the attenuated Biscuit token).

The above workflow is all it takes to integrate GitHub’s OIDC with nixbuild.net. If you are using some other OIDC Provider, or if you’re not using nixbuild-action you should pass in your OIDC ID Token to nixbuild.net by setting the SSH environment variable (or HTTP header, if you’re using the HTTP API) named NIXBUILDNET_OIDC_ID_TOKEN. The OIDC ID Token should have the audience nixbuild.net.

This blog post will try to summarize some of what we have been working on behind the scene the last couple of years. Read on to hear about nixbuild.net going enterprise, a brand new web UI and lots of new functionality!

Self-Hosting nixbuild.net

A big thing that has kept us busy the last year is packaging nixbuild.net for self-hosting inside enterprise setups. We’ve had a number of companies reaching out to us inquiring about such setups. The reason for wanting to self-host nixbuild.net are several: internal legal requirements, network bottlenecks, cost control and the ability to use hardware that is tricky to offer in the public service.

One of our enterprise customers self-hosts nixbuild.net in production running 10K+ builds daily. A very welcome side-effect of adapting nixbuild.net for self-hosting is that many scenarios that are seldom or never triggered by users of our public service are uncovered by enterprise usage. This has made nixbuild.net even more reliable and capable under high load.

We are in a stage where we are happy to work closely with companies, figuring out how to best deploy and integrate nixbuild.net into their environments. Our goal is to eventually be able to offer self-hosted nixbuild.net as an off-the-shelf product. Actually, if you are on AWS, the deployment of nixbuild.net is already very simple. We will publish a blog post with more details on this in the near future, but the short rundown is this:

• We provide an EC2 AMI that runs NixOS and all necessary nixbuild.net services.
• The user deploys this AMI using her preferred method of EC2 provisioning.
• The user provides nixbuild.net configuration through cloud-init or by simply putting configuration files in place on the EC2 instance.
• The nixbuild.net configuration includes details on what types of EC2 instances to use for running the builds, and how many builder instances that your nixbuild.net deployment should be allowed to provision.

You can then simply use your nixbuild.net EC2 server as remote Nix builder, and get all the nixbuild.net niceties of automatic builder provisioning, scale-to-zero, automatic CPU and memory allocation, strict build isolation etc.

If your company is interested in self-hosting nixbuild.net, please reach out to us!

Finally, a Web UI

We are proud to say that we now have a brand-new web UI available to all nixbuild.net users! This includes a much simplified sign-up flow, where the Hanko service is used to provide and secure users’ login details. It is now easier than ever getting started with nixbuild.net!

The web UI is a complement to the tried-and-true SSH-based admin shell. We are gradually building out the web functionality to cover everything you can do in the shell today, and more. For the moment you can get by using the web UI to get started with nixbuild.net, adding SSH keys or creating access tokens. For configuring settings or setting up billing you will still need to use the nixbuild.net shell.

Our plan ahead is to keep the SSH-based shell which is very appreciated by many users and offers an efficient interface for managing your nixbuild.net account. The web UI will be expanded with full ability to manage your account, as well as pages to present and visualize builds and Nix store contents in ways that the terminal shell cannot.

Support for More Types of Caches and Substituters

During the last year, nixbuild.net has gained new support for S3 and improved support for Cachix.

nixbuild.net can now use the following sources when substituting (downloading) cached store paths:

• Public HTTP caches, like cache.nixos.org
• Public Cachix caches
• Authenticated Cachix caches
• S3 buckets (AWS or compatible providers)

For more information, see the documentation for the substituters setting. nixbuild.net can use multiple substituters of different types at the same time. It tries to do as much as possible in parallel, for example, all substituters are queried concurrently about available narinfo files. The substituter that answers first is used.

nixbuild.net can also populate binary caches, not only with build outputs but also their inputs. By activating this functionality, you don’t have to do any post-build cache uploads from your Nix client machine, it will instead be performed automatically by nixbuild.net as part of the build. Also in this case, nixbuild.net will try to parallelize as much as possible. Build inputs will for example start uploading at the same time the build starts.

nixbuild.net can now upload store paths to the following types of targets:

• Cachix caches
• S3 buckets (AWS or any compatible provider)

For more information, see the documentation for the caches setting.

More Knobs to Tweak

We have exposed more settings to end users lately. Some of these settings existed internally already, and some are newly implemented.

These new settings correspond to the identically named Nix settings:

• hashed-mirrors
• impure-env
• max-silent-time
• timeout

These new settings lets you control resource allocation:

• default-cpu
• default-mem-per-cpu
• max-cpu
• max-mem
• min-cpu
• min-mem

For a complete list of settings, check out the documentation.

Support for Custom Nix Store Directories

nixbuild.net supports using Nix store directories other than the default /nix/store one. You don’t have do anything special in nixbuild.net to enable this, it just works. If your Nix client is building something using a custom Nix store directory, nixbuild.net will automatically setup the build sandbox with the custom directory and make all build inputs available as usual.

Self-Managed Signing Keys

By default, nixbuild.net creates a unique Nix signing key for every account. This key is used to sign all build outputs and all paths that you upload to nixbuild.net. In practice this is not something you have to care that much about. Nix doesn’t require remote builders to sign builds, and you don’t have to setup your Nix client with the public keys that corresponds to your account’s signing key. This is because Nix has roughly the same level of trust for remote builds as for local builds.

However, inside nixbuild.net, Nix signatures are central, because the set of store paths available in a session depends on which signatures are trusted in that session. To control which signatures to trust, you can use the trusted-public-keys setting. Like all settings, it can be specified on the SSH key level or even on the environment of individual SSH sessions.

This flexibility is perhaps not useful to all nixbuild.net users, but it can be valuable for power users that have embraced Nix signatures as a way to track build provenance and control store access. For example, you could have your deployment pipeline verify that only store paths signed by a specific key are deployed to your production environment.

We have now added support for letting you control the signing keys that should be used, instead of relying on keys auto-created by nixbuild.net. You can even setup your configuration so that your secret signing keys are not persistently stored in nixbuild.net, but only accessible by nixbuild.net during builds or uploads.

To read more about Nix signing key in general and more details on how nixbuild.net uses signing keys, check out our newly added documentation. There you’ll also learn about how to setup access-tokens to lock down sub-users of your nixbuild.net account to specific trusted public keys.

When the API was launched, it was using Biscuit auth tokens to handle authentication and authorization. In the initial implementation, we didn’t really take much advantage of Biscuit, and basically used them as plain API keys. This week, however, we’ve finally expanded our Biscuit usage to properly take advantage of the flexibility offered.

Read on to find out how this allows us to now support advanced access policies, token-based SSH auth and signed web links that gives easy access to build logs and fancy build reports.

Signed Web Links

Let’s start out with the feature we’re really happy to launch: signed web links. To show how it works, I grep for a broken package in nixpkgs and find one called textpieces. Then I try building it on my laptop which has been setup to use nixbuild.net as a remote builder (we need NIXPKGS_ALLOW_BROKEN=1 and --impure in order to force Nix to evaluate the broken package):

$ NIXPKGS_ALLOW_BROKEN=1 nix build --impure nixpkgs#textpieces

error: build of '/nix/store/s7ns32ymdf9iq44hpa62dkdbwlhvc7nx-textpieces-3.4.1.drv' on 'ssh://eu.nixbuild.net' failed: builder for '/nix/store/s7ns32ymdf9iq44hpa62dkdbwlhvc7nx-textpieces-3.4.1.drv' failed with exit code 1
error: builder for '/nix/store/s7ns32ymdf9iq44hpa62dkdbwlhvc7nx-textpieces-3.4.1.drv' failed with exit code 1;
last 10 log lines:
> upgrade: Use the '$' extern syntax introduced in blueprint 0.8.0
> at ../resources/ui/CustomToolPage.blp line 43 column 3:
>   43 |  .TextPiecesToolSettings tool_settings {
>      |  ^
> error: Cannot convert 2.5 to integer
> at ../resources/ui/Editor.blp line 45 column 26:
>   45 |          margin-bottom: 2.5;
>      |                         ^
> ninja: build stopped: subcommand failed.
> [nixbuild.net] See this link for build details and logs: https://nixbuild.net/builds/1868020?t=EtcBCm0KBWJ1aWxkCgpidWlsZDpyZWFkGAMiCAoGCAcSAhABIg0KCwgEEgc6BQoDGIEIMiYKJAoCCBsSBggFEgIIBRoWCgQKAggFCggKBiCZzeO0BgoEGgIIADIVChMKAggbEg0IAhIDGIAIEgQQ9IFyEiQIABIgKYLCEYA2IZ9hybF7HQni7n68uipmGCexG7F4x0KMH2oaQOVIWFexkOoOpgNYWNXNv9GY6I5bO5T1n6RBQvU6D25_XTJm41Dmj4_fKPmzc9aZrt4rhxGBZafirifzKecFPgkiIgogIsQ6_jdmw1s5IhuXrrPabnGXJuoGoKrpfIUHz1uUU2k=
       For full logs, run 'nix log /nix/store/s7ns32ymdf9iq44hpa62dkdbwlhvc7nx-textpieces-3.4.1.drv'.

At the very end of the build we see the message [nixbuild.net] See this link for build details and logs, pointing us to https://nixbuild.net/builds/1868020?t=[…].

Go ahead and try that link now! What you’ll find is a page with details of the build we just ran, along with its full build logs. This also works for builds that don’t fail. Every build you run on nixbuild.net will get this handy summary link printed in the end of the build. You can share those links as you please, just like I did here. No accounts or logins on nixbuild.net are needed to access the links.

You can explicitly create a build link for any build you’ve run using the builds url command in the nixbuild.net adminstration shell, or by using the new HTTP API endpoint.

Biscuit Auth Tokens

So how do these build links work? The t= argument in the links is a Biscuit auth token, and we can use the biscuit-cli tool to inspect it:

$ echo >token "EtcBCm0KBWJ1aWxkCgpidWlsZDpyZWFkGAMiCAoGCAcSAhABIg0KCwgEEgc6BQoDGIEIMiYKJAoCCBsSBggFEgIIBRoWCgQKAggFCggKBiCZzeO0BgoEGgIIADIVChMKAggbEg0IAhIDGIAIEgQQ9IFyEiQIABIgKYLCEYA2IZ9hybF7HQni7n68uipmGCexG7F4x0KMH2oaQOVIWFexkOoOpgNYWNXNv9GY6I5bO5T1n6RBQvU6D25_XTJm41Dmj4_fKPmzc9aZrt4rhxGBZafirifzKecFPgkiIgogIsQ6_jdmw1s5IhuXrrPabnGXJuoGoKrpfIUHz1uUU2k="

$ biscuit inspect ./token

Authority block:
== Datalog ==
owner(1);
right(["build:read"]);
check if time($time), $time < 2024-07-18T09:55:37Z;
check if resource("build", 1868020);

== Revocation id ==
e5485857b190ea0ea6035858d5cdbfd198e88e5b3b94f59fa44142f53a0f6e7f5d3266e350e68f8fdf28f9b373d699aede2b87118165a7e2ae27f329e7053e09

==========

🙈 Public key check skipped 🔑
🙈 Datalog check skipped 🛡️

What you see above is Biscuit Policy Language. It states that the owner of the token is 1, which is an internal id of my own account on nixbuild.net. Then it says the token has the build:read right. That means that you can only use this token for reading information about builds. You can’t use it to run builds, or fetch store paths or anything like that. Furthermore, there are two check statements in the token. One of them sets an expiration time of the token, and the other one restricts the token to requests concerning a specific resource, namely the build with id 1868020.

Now, the really interesting thing about Biscuit tokens is that you can yourself restrict the tokens further, without involving the nixbuild.net service at all. This process is called attenuation. We can for example set a new expiration time on the token like this:

$ biscuit attenuate --block 'check if time($time), $time < 2023-10-01T00:00:00Z;' ./token

EtcBCm0KBWJ1aWxkCgpidWlsZDpyZWFkGAMiCAoGCAcSAhABIg0KCwgEEgc6BQoDGIEIMiYKJAoCCBsSBggFEgIIBRoWCgQKAggFCggKBiCZzeO0BgoEGgIIADIVChMKAggbEg0IAhIDGIAIEgQQ9IFyEiQIABIgKYLCEYA2IZ9hybF7HQni7n68uipmGCexG7F4x0KMH2oaQOVIWFexkOoOpgNYWNXNv9GY6I5bO5T1n6RBQvU6D25_XTJm41Dmj4_fKPmzc9aZrt4rhxGBZafirifzKecFPgkalAEKKhgDMiYKJAoCCBsSBggFEgIIBRoWCgQKAggFCggKBiCA7eKoBgoEGgIIABIkCAASICCqMLKzBGTXvvVsPHikBh0QIN9-BTcnPR0y3IBrSz_ZGkD_REi0xNW8VOi2-MDXPivZUgJdCv4A6cxcK4UuGgb_F9DBuizVuMv1GXkRxawlcMU6irjAUJH8KCLRxN7aUjACIiIKIE2HZlmwEYZZ4v2Rf4qjVmh9ldCUeYYTl3xP13bemrWY

Now, we got a new auth token with our additional restriction added to it. The interested reader can try to use the new token as t= parameter. This should work fine as long as the current date is before 2023-10-01.

To learn more about Biscuit tokens and policies I recommend reading through our new Access Control documentation, and of course also the Biscuit documentation.

Using Auth Tokens for Everything

As part of the launch of the signed web links, we have implemented support for using auth tokens in all parts of nixbuild.net. You can use them for accessing the HTTP API or when using the Nix client to run builds or interact with your Nix store in nixbuild.net.

You can create tokens using the tokens create command in the administration shell of nixbuild.net. When doing this, you select what permissions you want to embed in the token, and set an expiration time. With the permission system it is possible to create tokens that can only be used for specific operations in nixbuild.net.

When you have created a token, you can create new, more restricted tokens from it, using the attenuation process demonstrated above.

You can still use SSH keys for auth, if you like. In fact, the newly introduced permissions are also possible to assign to specific SSH keys, using the default-permissions setting. This way, you can gain some more granularity in your authorization setup, without going all in on Biscuit tokens.

The Future

For now, the access policies you are able to define in your auth tokens are somewhat limited since the policy context is still something we are expanding. The context is represented as a set of facts that are made available by nixbuild.net during the authorization phase. We are working on adding more facts, and also implementing more example policies that you can gain inspiration from. As always, if you have any feedback, questions or issues, just ping us on support@nixbuild.net.

For now, ARM builds are priced in the exact same way as x86-64 builds. This might change in the future.

If you’re already a nixbuild.net user, head over to the documentation to learn about how to configure Nix to run aarch64-linux builds on nixbuild.net. If you’re not a user, register today and get started! Every new account gets free build hours that can be spent on both ARM and x86-64 builds.

By switching over our own CI setup (running on GitHub Actions) to this build mode we reduced the best-case build time of our slowest CI job from 20 minutes down to just 20 seconds, an amazing 60x speedup!

No extra setup is needed, everything is already built into Nix and the feature is available to all nixbuild.net users right now.

Read on to find out how to try this out on your own CI builds!

Remote Builders vs Stores

Usually, when Nix distributes builds, it sends them to one or more remote builder. These remote builders are just normal Nix machines, accessed over SSH by the Nix client. Since nixbuild.net tries hard to act just like a normal Nix machine from the outside, you can use nixbuild.net as a drop-in replacement for a remote builder. This is how nixbuild.net has worked from the very launch.

Here is a brief iteration of what happens when Nix uses a remote builder to perform a build:

1. Nix checks if the build result already is available in the local store or can be fetched from any configured substituter (binary cache).

2. If Nix needs to run the build, it will first make sure that all dependencies (inputs), and their transitive closures, are in place in the local Nix store. This might mean it has to build the inputs, if there are no existing builds to be found.

3. Once all inputs are locally available, Nix will select a remote builder machine that can perform the build, and then check if the builder has all build inputs in place in its store. If there are any missing closures, they will either be downloaded from a substituter directly to the remote builder, or uploaded from the local store to the builder.

4. Now, Nix asks the remote builder to perform the build. Once it is done Nix will download the build result from the builder to its local store.

We have just added a chapter in the nixbuild.net documentation that describes this process in greater detail.

If you use a service such as GitHub Actions, each run will start out with a completely empty environment. This is pretty bad when using Nix, because it means all build inputs have to be fetched on each run. Even for changes that only trigger quick builds, the total size of the build inputs might be considerable. Not only does it take time to fetch the closures from binary caches, it also takes time for Nix to unpack the closure archives (compressed nar-files) to the local file system.

If you have configured a binary cache together with your CI, where all new build results are stored, you might sometimes be lucky and commit code that doesn’t trigger any new builds at all. In this case, no build inputs needs to be fetched. However, the build output and its transitive closure will still be fetched from your binary cache to your CI. This might seem unavoidable and even desireable, but in fact you are often not interested in the build output itself when it comes to CI builds. What you really want to see is if your builds and tests pass. If you are using a remote builder like nixbuild.net, actually fetching the build output to the ephemeral CI runner is just a waste of time.

For some builds, the time spent on transporting Nix closures can add up to a considerable amount. In our own CI for nixbuild.net, we had one particularly slow build. It builds our development shell, including NixOS containers for a complete development deployment of nixbuild.net itself. It contains all software that the nixbuild.net service directly or indirectly uses. The total closure size is just shy of 10 GB. Running this build in GitHub Actions (hooked up to nixbuild.net, of course) took about 20 minutes, even when minimal actual building was required.

Today, the same build takes just 20 seconds!

Nix has a lesser-known way of distributing builds, using remote stores. It is conceptually the same thing as SSHing to a remote Nix machine and running your build on it instead of on your local machine. But Nix makes the process a bit more convenient, allowing the Nix evaluation to happen locally (where you have your sources). To use it with nixbuild.net, all you have to do is to run this:

nix build \
  --eval-store auto \
  --store ssh-ng://eu.nixbuild.net \
  ...

Now, the build process will look roughly like this instead:

1. Nix evaluates your build as usual. During this phase, derivation files (.drv-files) will be written to your local store, and any sources that are needed for evaluation will be fetched.

2. Nix will copy over all .drv-files for the build and its inputs to the remote Nix machine (eu.nixbuild.net in this case). Any sources used during evaluation will also be copied over (if they don’t already exist on remote machine).

3. Now, Nix will tell the remote machine to build the top-level .drv file of the build you requested. During the build, logs will be sent over to your local console so it looks just like an ordinary build, but everything is running remotely.

4. When the build is done, the build results will be available on the remote machine, but Nix will not fetch them. If you need to, you can fetch them explicitly, with nix copy.

As you can see, all copying of Nix closures is gone, improving performance tremendously for many types of builds. We have created a public demonstration repository where you can see more comparisons between remote store building and traditional building, and practical examples of GitHub Actions. We plan on keeping the repository updated and adding more sample projects.

It is not only CI runners that benefit from using nixbuild.net as a remote store. If you use Nix on underpowered (in terms of network, storage or CPU) machines, you might want to offload as much work as possible from them. Using a remote store for building guarantees that the local machine will only be used for Nix evaluation.

Nix itself has had support for remote store building for some time, but the support for --eval-store (which makes the whole thing practically useful for actual builds) was recently added. We were able to add support for remote store building to nixbuild.net by implementing more pieces of the nix-daemon protocol (than we had already implemented). With this new feature nixbuild.net fits into even more Nix use cases, and as we add support for even more pieces of the nix-daemon protocol we hope to find many more.

If you want to try this new feature out, you need to use Nix 2.4 or newer.

Lastly, we should add that the remote store building feature on nixbuild.net should be considered beta for the time being. There is some polishing needed in the UX (the output you see when running Nix in your shell), some known limitations, and we are also working on further performance improvements. If you try it out, your feedback is very much welcome, either through support@nixbuild.net or our issue tracker.

KVM support is something that almost no public CI/CD provider offers, so we are very happy to be able to make this available to our users.

For NixOS users and developers, this is especially valuable since it makes it possible to run integration tests based on the powerful testing framework found in nixpkgs/NixOS.

KVM builds are currently in an Early Access phase. If you want to try it out, contact us at support@nixbuild.net. Anyone is free to request access, but there might be waiting time depending on interest. During Early Access, KVM builds are priced and handled exactly as standard builds. The final pricing and price model for builds that require KVM has not been settled yet.

So far KVM builds are only supported on x86_64-linux, not on ARM.

Setup

Once you’ve gained access to running KVM builds, configuring your Nix client is simple. You just need to mark your eu.nixbuild.net remote builder with the kvm system feature. And if you want to run NixOS integration tests you’ll need the nixos-test feature too. For more detailed configuration descriptions see the documentation.

KVM builds in GitHub Actions

The nixbuild.net GitHub Action has been updated to support KVM builds too. If you’ve been granted access to KVM builds, all you have to do is to make sure you use the latest release of nixbuild-action. No extra configuration is needed to run KVM builds, or NixOS integration tests, in GitHub Actions.

For the moment, ARM builds are in an Early Access phase. If you want to try it out, just drop us a line at support@nixbuild.net. Anyone is free to request access, but there might be waiting time depending on interest. During Early Access, ARM builds are priced and handled exactly as the standard x86_64-linux builds. The final pricing and price model for ARM builds has not been settled yet.

Setup

Once you’ve gained access to running ARM builds, configuring your Nix client is just as straightforward as for x86_64-linux builds. Just swap the system of your remote builder to aarch64-linux. You can use nixbuild.net for both x86_64 and ARM builds at the same time. For more detailed configuration descriptions see the documentation.

Zero-config aarch64-linux builds in GitHub Actions

Together with the new support for ARM builds in nixbuild.net, there’s also a new release of the nixbuild.net GitHub Action. The release (v6) enables support for aarch64-linux for your GitHub builds. If you’ve been granted access to aarch64-linux builds, all you have to do is to add --system aarch64-linux or similar to your nix invocations inside your GitHub actions.

It is of course also simple to run aarch64-linux builds from GitLab, Hydra or any other CI setup using Nix. Just add nixbuild.net as an aarch64-linux remote builder as described in the documentation.

The idea is that you have a task you want to perform with varying input parameters. If the task takes multiple parameters, and you’d like to try it out with multiple values for each parameter, it is easy to end up with a combinatorial explosion.

This blog post gives a practical demonstration showing how Nix is a perfect companion for managing parameter sweeps and build matrices, and how nixbuild.net can be used to supercharge your workflow.

My hope is that this text can interest readers that don’t know anything about Nix as well as experienced Nix users.

Use Cases

In scientific computing, it is common to run simulations of physical processes. The list of things simulated is endless: weather forecasting, molecular dynamics, celestial movements, FEM analysis, particle physics etc. A simulation is usually implemented directly as a computer program or as a description for a higher level simulation framework. A simulation generally has a set of input parameters that can be defined. These parameters can describe initial states, environmental aspects or tweak the behavior of the simulation algorithm itself. Scientists are interested in comparing simulation results for a range of different parameter values, and the process of doing so is referred to as a parameter sweep.

Parameter sweeping is often built into simulation frameworks. For simulations implemented directly as specialized programs, scientists will simply run the program over and over again with different parameters, collecting and comparing the results. When supercomputers are used for running the simulations, the job scheduler usually has some support for launching multiple simulation instances with varying parameters.

In the software industry, the term build matrix is used to mean basically the same thing as parameter sweeping. Regularly, build matrices are used to build different variants of the same deliverable. In the simplest case, a programmer builds and packages a program for set of different targets (Windows, MacOS, Android etc). But more complex build matrices with (much) higher dimensionality is of course also used.

Benchmarking is another area where build matrices are utilized, and combinatorial explosions are common. The demo below will show how a compression benchmark can be implemented with Nix and nixbuild.net.

Embarassing Parallelism

Parameter sweeping can be classified as an embarrassingly parallel problem. As long as we have enough CPUs, all simulations or builds can be executed in parallel, since there is (usually) no dependencies between them. This is a perfect workload for nixbuild.net, which is built to be very scalable.

At the same time, it is also easy to get into troubles managing all possible combinations of parameter values. Adding new parameters or parameter values can increase the number of runs exponentially, and the work of managing the runs and their results becomes overwhelming. The next section will show how Nix can help out with this.

How Nix Helps

One of the aspects of Nix that I find most empowering is that it helps you with the boring and stress-inducing task of managing files. Let me see if I can explain.

Assume you have a program called simulation that takes a parameter file as its only argument. The parameter file contains simulation parameters in a simple INI-format like this:

param1=3
param2=92
param3=0

The program outputs a simulation result on its standard output, in CSV format.

Now we want to run the simulation for some different combinations of input parameters. We could manually author the needed parameter files, or we can write a simple shell script for it:

round=1

for param1 in 3 4 5; do
  for param2 in 10 33 50 92; do
    for param3 in 0 1; do
      echo "param1=$param1" >> "round$round.ini"
      echo "param2=$param2" >> "round$round.ini"
      echo "param3=$param3" >> "round$round.ini"
      round=$((round+1))
    done
  done
done

We now got 24 different parameter files with all possible combinations of the values we are interested in:

$ ls -v
round1.ini  round6.ini   round11.ini  round16.ini  round21.ini
round2.ini  round7.ini   round12.ini  round17.ini  round22.ini
round3.ini  round8.ini   round13.ini  round18.ini  round23.ini
round4.ini  round9.ini   round14.ini  round19.ini  round24.ini
round5.ini  round10.ini  round15.ini  round20.ini

$ cat round20.ini
param1=5
param2=33
param3=1

To run the simulations, we simply loop through all parameter files:

for round in round*.ini; do
  simulation $round >> results.csv
done

So far, so good. At this point, we might want to tweak our parameter generation a bit. Maybe there are more parameter values we want to explore, different sweeps to do. So we change our parameter generator script and re-run a few times. We then realise we want to make changes to our simulation program itself. So we do that, and recompile it. Now we want to re-run all the different parameter sweeps we’ve done. Luckily, we saved all different versions of our parameter generation script, so we can just run the updated simulator with the previous simulator.

At the end of our productive simulation session we have the following mess (with all intermediate parameter files removed):

gen-params.sh         results-1.csv   results-12.csv  simulation-3
gen-params-1.sh       results-2.csv   results-13.csv  simulation-O2-1
gen-params-2.sh       results-3.csv   results-14.csv  simulation-O2-2
gen-params-3.sh       results-4.csv   results-15.csv  simulation-O3
gen-params-3v2.sh     results-5.csv   results-16.csv  simulation-debug
gen-params-4.sh       results-6.csv   results-17.csv  simulation-debug2
gen-params-test.sh    results-7.csv   simulation      simulation-wrong
gen-params-test-2.sh  results-8.csv   simulation-1
results.csv           results-10.csv  simulation-2

Admittedly, this is how things usually end up for me when I’m doing any kind of “exploratory” work. I’m sure you all are much more organized. To my rescue comes Nix. It allows me to stop caring entirely about generated files, and only care about how stuff is generated. Additionally, it gives me tools to abstract, parameterize and reuse generators.

Let’s make an attempt at recreating our workflow above with Nix:

{ pkgs ? import <nixpkgs> {} }:

let

  inherit (pkgs) lib callPackage runCommand writeText;

  # Compiles the 'simulation' program, and allows us to provide
  # build-time arguments
  simulation = callPackage ./simulation.nix;

  # Executes the given simulation program with the given parameters
  runSimulation = buildArgs: parameters:
    runCommand "result.csv" {
      buildInputs = [ (simulation buildArgs) ];
      parametersFile = writeText "params.ini" (
        lib.generators.toKeyValue {} parameters
      );
    } ''
      simulation $parametersFile > $out
    '';

  # Merges multiple CSV files into a single one
  mergeResults = results: runCommand "results.csv" {
    inherit results;
  } ''
    cat $results > $out
  '';

in {

  sim_O3_std_sweep = mergeResults (
    lib.forEach (lib.cartesianProductOfSets {
      param1 = [3 4 5];
      param2 = [10 33 50 92];
      param3 = [0 1];
    }) (
      runSimulation {
        optimizationLevel = 3;
      }
    )
  );

  sim_O2_small_sweep = mergeResults (
    lib.forEach (lib.cartesianProductOfSets {
      param1 = [1 3];
      param2 = [20 60 92];
      param3 = [0 1];
    }) (
      runSimulation {
        optimizationLevel = 2;
      }
    )
  );

}

The key function above is perhaps cartesianProductOfSets, from the library functions in nixpkgs. It will create all possible combinations of input parameters, if we list the possible value for each parameter. Our build function is then mapped over all these combinations using forEach.

We can build one of our parameter sweeps like this:

nix-build -A sim_O3_std_sweep

When all 24 simulation runs are done, Nix will create a single result symlink in the current directory, pointing to a results.csv file containing all simulation results. We can add new sweeps to our Nix file and re-run the build at any time. We never have to care about any generated files, since everything needed to re-generate results exists in the Nix file. The Nix file itself can be version-controlled like any source file.

In addition to the demonstrated ability to parameterize builds, Nix provides us with two more things, for free.

No unnecessary re-builds

In the example above, the sim_O3_std_sweep and sim_O2_small_sweep builds have some overlapping parameter sets. If you build both, Nix will only run the overlapping simulations once, and use the same result.csv files to create the two different results.csv files. This happens without any extra effort from the user. The same is true if you make changes that only affect part of your build. Nix also has support for external caches which makes it easy to share and reuse build results between computers (or you can simply use nixbuild.net to get build sharing without any extra configuration).

Automatic parallelization

When Nix evaluates an expression, it constructs a build graph that tracks all dependencies between builds. In the example above, the results.csv file depends on the list of result.csv files, which in turn depend on specific builds of simulation. All of these dependencies are implicit; you don’t have to do anything other than simply refer to the things you need to perform your build.

The build graph allows Nix to execute the actual builds with maximum parallelization. However, running as many builds as possible in parallel is often not optimal, if your compute resources (usually: your local computer) are limited. Nix has a simplistic build scheduler, which is just a user-configurable limit of the maximum number of concurrent builds. This works in many cases, but quickly gets non-optimal when you have lots of builds that could run in parallel, or when the builds themselves have varying compute requirements.

This is where nixbuild.net can step in. It is able to run an “infinite” number of concurrent Nix builds for you, while keeping all builds perfectly isolated from each other (security- and resource-wise). It also selects compute resources intelligently for each individual build.

From the perspective of scientific computing, you can say that Nix provides a generic framework for parallel workloads, and nixbuild.net acts somewhat as a supercomputer, minus the effort of writing submission scripts and explicitly managing compute results.

Demo: Compression Benchmark

I’m now going to show you an example that is similar to the example in the previous section, but instead of an imaginary simulation we will run an actual benchmark this time. The benchmark will compare the compression ratio of a number of different lossless compression implementations. Since this article is about parameter sweeping, we will vary the following parameters during the benchmark:

• Compression implementation: brotli, bzip2, gzip, lz4, xz and zstd.

• Two different versions of each compression implementation. We’ll use the versions packaged in nixpkgs 16.03 and 20.09, respectively.

• Compression level: 1-9.

• Corpus type: text, binaries and jpeg files.

• Corpus size: small, medium and large.

We’ll try out the Cartesian product of the above parameters, resulting in 972 different builds. There is no particular thought behind the parameter selection, they are just picked to demonstrate the abilities of Nix and nixbuild.net. If you were to design a proper benchmark you’d likely come up with different parameters, but the concept would be the same.

Here is the complete Nix expression implementing the benchmark outlined above. The expression is parameterized over package sets from different releases of nixpkgs. There are different ways of actually importing those package sets, but that is out of the scope of this example.

{ pkgs, pkgs_2009, pkgs_1603 }:

let

  inherit (pkgs)
    stdenv fetchurl lib writers runCommand unzip gnutar
    referencesByPopularity uclibc hello zig;

  compressionCommand = pkgs: program: level: {

    brotli = writers.writeBash "brotli-compress" ''
      if [ -x ${pkgs.brotli}/bin/brotli ]; then
        ${pkgs.brotli}/bin/brotli --stdout -${toString level}
      else
        ${pkgs.brotli}/bin/bro --quality ${toString level}
      fi
    '';

    bzip2 = "${pkgs.bzip2}/bin/bzip2 --stdout -${toString level}";

    gzip = "${pkgs.gzip}/bin/gzip --stdout -${toString level}";

    lz4 = "${pkgs.lz4}/bin/lz4 --stdout -${toString level}";

    xz = "${pkgs.xz}/bin/xz --stdout -${toString level}";

    zstd = "${pkgs.zstd}/bin/zstd --stdout -${toString level}";

  }.${program};

  corpus = rec {
    txt.small = calgary-text.small;
    txt.medium = calgary-text;
    txt.large = runCommand "enwik8" {
      buildInputs = [ unzip ];
      src = fetchurl {
        url = "http://mattmahoney.net/dc/enwik8.zip";
        sha256 = "1g1l4n9x8crxghapq956j7i4z89qkycm5ml0hcld3ghfk3cr8yal";
      };
    } ''
      unzip "$src"
      mv enwik8 "$out"
    '';

    pkg.small = closure-tar "uclibc-closure.tar" uclibc;
    pkg.medium = closure-tar "hello-closure.tar" hello;
    pkg.large = closure-tar "zig-closure.tar" zig;

    jpg.small = fetchurl {
      url = "https://people.sc.fsu.edu/~jburkardt/data/jpg/charlie.jpg";
      sha256 = "0cmd8wwm0vaqxsbvb3lxk2f7w2lliz8p361s6pg4nw0vzya6lzrg";
    };
    jpg.medium = fetchurl {
      url = "https://cdn.hasselblad.com/samples/x1d-II-50c/x1d-II-sample-02.jpg";
      sha256 = "15pz84f5d34jmp0ljz61wx3inx8442sgf9n8adbgb8m4v88vifk2";
    };
    jpg.large = fetchurl {
      url = "https://cdn.hasselblad.com/samples/Cam_1_Borna_AOS-H5.jpg";
      sha256 = "0rdcxlxcxanlgfnlxs9ffd3s36a05g8g3ca9khkfsgbyd5spk343";
    };

    calgary-text = stdenv.mkDerivation {
      name = "calgary-corpus-text";
      src = fetchurl {
        url = "http://corpus.canterbury.ac.nz/resources/calgary.tar.gz";
        sha256 = "1dwk417ql549l0sa4jzqab67ffmyli4nmgaq7i9ywp4wq6yyw2g1";
      };
      sourceRoot = ".";
      outputs = [ "out" "small" ];
      installPhase = ''
        cat bib book2 news paper* prog* > "$out"
        cat paper1 > "$small"
      '';
    };

    closure-tar = name: pkg: runCommand name {
      buildInputs = [ gnutar ];
      closure = referencesByPopularity pkg;
    } ''
      tar -c --files-from="$closure" > "$out"
    '';
  };

  benchmark = { release, program, level, corpusType, corpusSize }:
    runCommand (lib.concatStringsSep "-" [
      "zbench" program "l${toString level}" corpusType corpusSize release.rel
    ]) rec {
      corpusFile = corpus.${corpusType}.${corpusSize};
      run = compressionCommand release.pkgs program level;
      version = lib.getVersion release.pkgs.${program};
      tags = lib.concatStringsSep "," [
        program version (toString level) corpusType corpusSize
      ];
    } ''
      orig_size="$(stat -c %s "$corpusFile")"
      result_size="$($run < "$corpusFile" | wc -c)"
      percent="$((100*result_size / orig_size))"
      echo >"$out" "$tags,$orig_size,$result_size,$percent"
    '';

in runCommand "compression-benchmarks" {
  results = map benchmark (lib.cartesianProductOfSets {
    program = [
      "brotli"
      "bzip2"
      "gzip"
      "lz4"
      "xz"
      "zstd"
    ];
    release = [
      { pkgs = pkgs_1603; rel = "1603"; }
      { pkgs = pkgs_2009; rel = "2009"; }
    ];
    level = lib.range 1 9;
    corpusType = [ "txt" "pkg" "jpg" ];
    corpusSize = [ "small" "medium" "large" ];
  });
} ''
  echo program,version,level,corpus,class,orig_size,result_size,ratio > $out
  cat $results >> $out
''

Above, compressionCommand defines the command used for each compression program to compress stdin to stdout with a given level.

The corpus attribute set defines txt, pkg and jpg datasets. For text and jpeg we simply fetch suitable sets, and for the binary (pkg) sets we use Nix itself to create a tar file out of the transistive closure of some different packages. The corpus sizes varies between around 50 kB and 300 MB.

benchmark runs a single compression command for one combination of input parameters.

Finally, we again use the cartesianProductOfSets function to create builds of all possible combinations of parameters, and then simply concatenate all individual results into a big CSV file.

Building the complete benchmark takes about 25 minutes on my somewhat old 8-core workstation, with Nix configured to run at most 8 builds concurrently. If I use nixbuild.net instead, time is cut down to 10 minutes due to the parallelization gains possible when running 972 independent Nix builds.

In the end we get a CSV-file with values for each parameter combination. The first ten lines of the file looks like this:

program,version,level,corpus,class,orig_size,result_size,ratio
brotli,0.3.0,1,txt,small,53161,19634,36
brotli,1.0.9,1,txt,small,53161,21162,39
bzip2,1.0.6,1,txt,small,53161,16558,31
bzip2,1.0.6.0.1,1,txt,small,53161,16558,31
gzip,1.6,1,txt,small,53161,21605,40
gzip,1.10,1,txt,small,53161,21605,40
lz4,131,1,txt,small,53161,27936,52
lz4,1.9.2,1,txt,small,53161,28952,54
xz,5.2.2,1,txt,small,53161,18416,34

To quickly get some sort of visualization of the benchmark data, I dumped the CSV contents into rawgraphs.io and produced the following graph:

From this visualization, we can draw a few conclusions:

• There’s little point in compressing (already compressed) JPG data.

• xz is the clear winner when it comes to producing small archives of binary data.

• bzip2 produces almost the same compression ratio for all level settings. It even looks like level 9 can produce slightly worse compression than level 8.

• lz4 makes a very big jump in compression ratio between level 2 and 3.

To further refine our workflow, we could also produce data visualizations directly in our Nix expression, by creating builds that would feed the CSV data into some visualization software.

Remember, this blog post is not about benchmarking compression, but about how you can use Nix and nixbuild.net for such workflows. Hopefully you’ve gained some insights into how Nix can be used in scientific computing and data science workflows. Let’s wrap up with a summary of why I find Nix useful in these situations:

• The Nix programming language and standard library provide tools for managing combinatorial problems, and allows us to quickly come up with high level abstractions giving us sensible knobs to turn when exploring parameter sweeps and build matrices.

• We don’t have to think about parallelization, Nix takes care of it for us.

• Nix makes it very easy to build specific variants of packages. This is helpful if you want make comparisons between different software versions or patches. nixpkgs is a huge repository of pre-packaged software available to anyone.

• nixbuild.net gives you extreme scalability with no adaptation or configuration needed. In the example above we saw build times cut to less than half by sending our Nix builds to nixbuild.net.

• Reproducibility and build reuse is first-rate in Nix.

Thank you for reading this rather lengthy blog post! If have any comments or questions about the content or about nixbuild.net in general, don’t hesitate to contact me.

Nix is particularly suited for working on reproducibility, since it by design isolates builds and comes with tools for finding non-determinism. The Nix community also works on related projects, like Trustix and the content-addressed store.

This blog post summarises how nixbuild.net can be useful for finding non-deterministic builds, and announces a new feature related to reproducibility!

Repeated Builds

The way to find non-reproducible builds is to run the same build multiple times and check for any difference in results, when compared bit-for-bit. Since Nix guarantees that all inputs will be identical between the runs, just finding differing output results is enough to conclude that a build is non-deterministic. Of course, we can never prove that a build is deterministic this way, but if we run the build many times, we gain a certain confidence in it.

To run a Nix build multiple times, simply add the –repeat option to your build command. It will run your build the number of extra times you specify.

Suppose we have the following Nix expression in deterministic.nix:

let
  inherit (import <nixpkgs> {}) runCommand;
in {
  stable = runCommand "stable" {} ''
    touch $out
  '';

  unstable = runCommand "unstable" {} ''
    echo $RANDOM > $out
  '';
}

We can run repeated builds like this (note that the --builders "" option is there to force a local build, to not use nixbuild.net):

$ nix-build deterministic.nix --builders "" -A stable --repeat 1
these derivations will be built:
  /nix/store/0fj164aqyhsciy7x97s1baswygxn8lzf-stable.drv
building '/nix/store/0fj164aqyhsciy7x97s1baswygxn8lzf-stable.drv' (round 1/2)...
building '/nix/store/0fj164aqyhsciy7x97s1baswygxn8lzf-stable.drv' (round 2/2)...
/nix/store/6502c5490rap0c8dhvfwm5rhi22i9clz-stable

$ nix-build deterministic.nix --builders "" -A unstable --repeat 1
these derivations will be built:
  /nix/store/psmn1s3bb97989w5a5b1gmjmprqcmf0k-unstable.drv
building '/nix/store/psmn1s3bb97989w5a5b1gmjmprqcmf0k-unstable.drv' (round 1/2)...
building '/nix/store/psmn1s3bb97989w5a5b1gmjmprqcmf0k-unstable.drv' (round 2/2)...
output '/nix/store/g7a5sf7iwdxs7q12ksrzlvjvz69yfq3l-unstable' of '/nix/store/psmn1s3bb97989w5a5b1gmjmprqcmf0k-unstable.drv' differs from previous round
error: build of '/nix/store/psmn1s3bb97989w5a5b1gmjmprqcmf0k-unstable.drv' failed

Running repeated builds on nixbuild.net works exactly the same way:

$ nix-build deterministic.nix -A stable --repeat 1
these derivations will be built:
  /nix/store/wnd5y30jp3xwpw1bhs4bmqsg5q60vc8i-stable.drv
building '/nix/store/wnd5y30jp3xwpw1bhs4bmqsg5q60vc8i-stable.drv' (round 1/2) on 'ssh://eu.nixbuild.net'...
copying 1 paths...
copying path '/nix/store/z3wlpwgz66ningdbggakqpvl0jp8bp36-stable' from 'ssh://eu.nixbuild.net'...
/nix/store/z3wlpwgz66ningdbggakqpvl0jp8bp36-stable

$ nix-build deterministic.nix -A unstable --repeat 1
these derivations will be built:
  /nix/store/6im1drv4pklqn8ziywbn44vq8am977vm-unstable.drv
building '/nix/store/6im1drv4pklqn8ziywbn44vq8am977vm-unstable.drv' (round 1/2) on 'ssh://eu.nixbuild.net'...
[nixbuild.net] output '/nix/store/srch6l8pyl7z93c7gp1xzf6mq6rwqbaq-unstable' of '/nix/store/6im1drv4pklqn8ziywbn44vq8am977vm-unstable.drv' differs from previous round
error: build of '/nix/store/6im1drv4pklqn8ziywbn44vq8am977vm-unstable.drv' on 'ssh://eu.nixbuild.net' failed: build was non-deterministic
builder for '/nix/store/6im1drv4pklqn8ziywbn44vq8am977vm-unstable.drv' failed with exit code 1
error: build of '/nix/store/6im1drv4pklqn8ziywbn44vq8am977vm-unstable.drv' failed

As you can see, the log output differs slightly between the local and the remote builds. This is because when Nix submits a remote build, it will not do the determinism check itself, instead it will leave it up to the builder (nixbuild.net in our case). This is actually a good thing, because it allows nixbuild.net to perform some optimizations for repeated builds. The following sections will enumerate those optimizations.

Finding Non-determinism in Past Builds

If you locally try to rebuild a something that has failed due to non-determinism, Nix will build it again at least two times (due to --repeat) and fail it due to non-determinism again, since it keeps no record of the previous build failure (other than the build log).

However, nixbuild.net keeps a record of every build performed, also for repeated builds. So when you try to build the same derivation again, nixbuild.net is smart enough to look at its past build and figure out that the derivation is non-deterministic without having to rebuild it. We can demonstrate this by re-running the last build from the example above:

$ nix-build deterministic.nix -A unstable --repeat 1
these derivations will be built:
  /nix/store/6im1drv4pklqn8ziywbn44vq8am977vm-unstable.drv
building '/nix/store/6im1drv4pklqn8ziywbn44vq8am977vm-unstable.drv' (round 1/2) on 'ssh://eu.nixbuild.net'...
[nixbuild.net] output '/nix/store/srch6l8pyl7z93c7gp1xzf6mq6rwqbaq-unstable' of '/nix/store/6im1drv4pklqn8ziywbn44vq8am977vm-unstable.drv' differs from previous round
error: build of '/nix/store/6im1drv4pklqn8ziywbn44vq8am977vm-unstable.drv' on 'ssh://eu.nixbuild.net' failed: a previous build of the derivation was non-deterministic
builder for '/nix/store/6im1drv4pklqn8ziywbn44vq8am977vm-unstable.drv' failed with exit code 1
error: build of '/nix/store/6im1drv4pklqn8ziywbn44vq8am977vm-unstable.drv' failed

As you can see, the exact same derivation fails again, but now the build status message says: a previous build of the derivation was non-deterministic. This means nixbuild.net didn’t have to run the build, it just checked its past outputs for the derivation and noticed they differed.

When nixbuild.net looks at past builds it considers all outputs that have been signed by a key that the account trusts. That means that it can even compare outputs that have been fetched by substitution.

Scaling Out Repeated Builds

When you use --repeat, nixbuild.net will create multiple copies of the build and schedule all of them like any other build would have been scheduled. This means that every repeated build will run in parallel, saving time for the user. As soon as nixbuild.net has found proof of non-determinism, any repeated build still running will be cancelled.

Provoking Non-determinism through Filesystem Randomness

As promised in the beginning of this blog post, we have new a feature to announce! nixbuild.net is now able to inject randomness into the filesystem that the builds see when they run. This can be used to provoke builds to uncover non-deterministic behavior.

The idea is not new, it is in fact the exact same concept as have been implemented in the disorderfs project by reproducible-builds.org. However, we’re happy to make it easily accessible to nixbuild.net users. The feature is disabled by default, but can be enabled through a new user setting.

For the moment, the implementation will return directory entries in a random order when enabled. In the future we might inject more metadata randomness.

To demonstrate this feature, let’s use this build:

let
  inherit (import <nixpkgs> {}) runCommand;
in rec {
  files = runCommand "files" {} ''
    mkdir $out
    touch $out/{1..10}
  '';

  list = runCommand "list" {} ''
    ls -f ${files} > $out
  '';
}

The files build just creates ten empty files as its output, and the list build lists those file with ls. The -f option of ls disables sorting entirely, so the file names will be printed in the order the filesystem returns them. This means that the build output will depend on how the underlying filesystem is implemented, which could be considered a non-deterministic behavior.

First, we build it locally with --repeat:

$ nix-build non-deterministic-fs.nix --builders "" -A list --repeat 1
these derivations will be built:
  /nix/store/153s3ir379cy27wpndd94qlfhz0wj71v-list.drv
building '/nix/store/153s3ir379cy27wpndd94qlfhz0wj71v-list.drv' (round 1/2)...
building '/nix/store/153s3ir379cy27wpndd94qlfhz0wj71v-list.drv' (round 2/2)...
/nix/store/h1591y02qff8vls5v41khgjz2zpdr2mg-list

As you can see, the build succeeded. Then we delete the result from our Nix store so we can run the build again:

rm result
nix-store --delete /nix/store/h1591y02qff8vls5v41khgjz2zpdr2mg-list

We enable the inject-fs-randomness feature through the nixbuild.net shell:

nixbuild.net> set inject-fs-randomness true

Then we run the build (with --repeat) on nixbuild.net:

$ nix-build non-deterministic-fs.nix -A list --repeat 1
these derivations will be built:
  /nix/store/153s3ir379cy27wpndd94qlfhz0wj71v-list.drv
building '/nix/store/153s3ir379cy27wpndd94qlfhz0wj71v-list.drv' (round 1/2) on 'ssh://eu.nixbuild.net'...
copying 1 paths...
copying path '/nix/store/vl13q40hqp4q8x6xjvx0by06s1v9g3jy-files' to 'ssh://eu.nixbuild.net'...
[nixbuild.net] output '/nix/store/h1591y02qff8vls5v41khgjz2zpdr2mg-list' of '/nix/store/153s3ir379cy27wpndd94qlfhz0wj71v-list.drv' differs from previous round
error: build of '/nix/store/153s3ir379cy27wpndd94qlfhz0wj71v-list.drv' on 'ssh://eu.nixbuild.net' failed: build was non-deterministic
builder for '/nix/store/153s3ir379cy27wpndd94qlfhz0wj71v-list.drv' failed with exit code 1
error: build of '/nix/store/153s3ir379cy27wpndd94qlfhz0wj71v-list.drv' failed

Now, nixbuild.net found the non-determinism! We can double check that the directory entries are in a random order by running without --repeat:

$ nix-build non-deterministic-fs.nix -A list
these derivations will be built:
  /nix/store/153s3ir379cy27wpndd94qlfhz0wj71v-list.drv
building '/nix/store/153s3ir379cy27wpndd94qlfhz0wj71v-list.drv' on 'ssh://eu.nixbuild.net'...
copying 1 paths...
copying path '/nix/store/h1591y02qff8vls5v41khgjz2zpdr2mg-list' from 'ssh://eu.nixbuild.net'...
/nix/store/h1591y02qff8vls5v41khgjz2zpdr2mg-list

$ cat /nix/store/h1591y02qff8vls5v41khgjz2zpdr2mg-list
6
1
2
5
10
7
8
..
9
4
3
.

Future Work

There are lots of possibilities to improve the utility of nixbuild.net when it comes to reproducible builds. Your feedback and ideas are very welcome to support@nixbuild.net.

Here are some of the things that could be done:

• Make it possible to trigger repeated builds for any previous build, without submitting a new build with Nix. For example, there could be a command in the nixbuild.net shell allowing a user to trigger a repeated build and report back any non-determinism issues.

• Implement functionality similar to diffoscope to be able to find out exactly what differs between builds. This could be available as a shell command or through an API.

• Make it possible to download specific build outputs. The way Nix downloads outputs (and stores them locally) doesn’t allow for having multiple variants of the same output, but nixbuild.net could provide this functionality through the shell or an API.

• Inject more randomness inside the sandbox. Since we have complete control over the sandbox environment we can introduce more differences between repeated builds to provoke non-determinism. For example, we can schedule builds on different hardware or use different kernels between repeated builds.

• Add support for listing known non-deterministic derivations.

This blog post will try to summarize how nixbuild.net has evolved since GA four months ago, and give a glimpse of the future for the service.

Stability and Performance

Thousands of Nix builds have been built by nixbuild.net so far, and every build helps in making the service more reliable by uncovering possible edge cases in the build environment.

These are some of the stability-related improvements and fixes that have been deployed since GA:

• Better detection and handling of builds that time out or hang.

• Improved retry logic should our backend storage not deliver Nix closures as expected.

• Fixes to the virtual file system inside the KVM sandbox.

• Better handling of builds that have binary data in their log output.

• Changes to the virtual sandbox environment so it looks even more like a “standard” Linux environment.

• Application of the Nix sandbox inside our KVM sandbox. This basically guarantees that the Nix environment provided through nixbuild.net is identical to the Nix environment for local builds.

• Support for following HTTP redirects from binary caches.

Even Better Build Reuse

One of the fundamental ideas in nixbuild.net is to try as hard as possible to not build your builds, if an existing build result can be reused instead. We can trivially reuse an account’s own builds since they are implicitly trusted by the user, but also untrusted builds can be reused under certain circumstances. This has been described in detail in an earlier blog post

Since GA we’ve introduced a number of new ways build results can be reused.

Reuse of Build Failures

Build failures are now also reused. This means that if someone tries to build a build that is identical (in the sense that the derivation and its transitive input closure is bit-by-bit identical) to a previously failed build, nixbuild.net will immediately serve back the failed result instead of re-running the build. You will even get the build log replayed.

Build failures can be reused since we are confident that our sandbox is pure, meaning that it will behave exactly the same as long as the build is exactly the same. Only non-transient failures will be reused. So if the builder misbehaves in some way that is out of control for Nix, that failure will not be reused. This can happen if the builder machine breaks down or something similar. In such cases we will automatically re-run the build anyway.

When we fix bugs or make major changes in our sandbox it can happen that we alter the behavior in terms of which builds succeed or fail. For example, we could find a build that fail just because we have missed implementing some specific detail in the sandbox. Once that is fixed, we don’t want to reuse such failures. To avoid that, all existing build failures will be “invalidated” on each major update of the sandbox.

If a user really wants to re-run a failed build on nixbuild.net, failure reuse can be turned off using the new user settings (see below).

Reuse of Build Timeouts

In a similar vein to reused build failures, we can also reuse build timeouts. This is not enabled by default, since users can select different timeout limits. A user can activate reuse of build timeouts through the user settings.

The reuse of timed out builds works like this: Each time a new build is submitted, we check if we have any previous build results of the exact same build. If no successful results or plain failures are found, we look for builds that have timed out. We then check if any of the existing timed out builds ran for longer than the user-specified timeout for the new build. If we can find such a result, it will be served back to the user instead of re-running the build.

This feature can be very useful if you want to avoid re-running builds that timeout over and over again (which can be a very time-consuming excercise). For example, say that you have your build timeout set to two hours, and some input needed for a build takes longer than that to build. The first time that input is needed you have to wait two hours to detect that the build will fail. If you then try building something else that happens to depend on the very same input you will save two hours by directly being served the build failure from nixbuild.net!

Wait for Running Builds

When a new build is submitted, nixbuild.net will now check if there is any identical build currently running (after checking for previous build results or failures). If there is, the new build will simply hold until the running build has finished. After that, the result of the running build will likely be served back as the result of the new build (as long as the running build wasn’t terminated in a transient way, in which case the new build will have to run from scratch). The identical running builds are checked and reused across accounts.

Before this change, nixbuild.net would simply start another build in parallel even if the builds were identical.

New Features

User Settings

A completely new feature has been launched since GA: User Settings. This allows end users to tweak the behavior of nixbuild.net. For example, the build reuse described above can be controlled by user settings. Other settings includes controlling the maximum used build time per month, and the possibility to lock down specific SSH keys which is useful in CI setups.

The user settings can be set in various way; through the nixbuild.net shell, the SSH client environment and even through the Nix derivations themselves.

Even if many users probably never need to change any settings, it can be helpful to read through the documentation to get a feeling for what is possible. If you need to differentiate permissions in any way (different settings for account administrators, developers, CI etc) you should definitely look into the various user settings.

GitHub CI Action

A GitHub Action has been published. This action makes it very easy to use nixbuild.net as a remote Nix builder in your GitHub Actions workflows. Instead of running you Nix builds on the two vCPUs provided by GitHub you can now enjoy scale-out Nix builds on nixbuild.net with minimal setup required.

The nixbuild.net GitHub Action is developed by the nixbuild.net team and there are plans on adding more functionality that nixbuild.net can offer users, like automatically generated cost and performance reports for your Nix builds.

Shell Improvements

Various minor improvements have been made to the nixbuild.net shell. It is for example now much easier to get an overview on how large your next invoice will be, through the usage command.

The Future

After one year of real world usage, we are very happy with the progress of nixbuild.net. It has been well received in the Nix community, proved both reliable and scalable, and it has delivered on our initial vision of a simple service that can integrate into any setup using Nix.

We feel that we can go anywhere from here, but we also realize that we must be guided by our users’ needs. We have compiled a small and informal roadmap below. The items on this list are things that we, based on the feedback we’ve received throughout the year, think are natural next steps for nixbuild.net.

The roadmap has no dates and no prioritization, and should be seen as merely a hint about which direction the development is heading. Any question or comment concerning this list (or what’s missing from the list) is very welcome to support@nixbuild.net.

Support aarch64-linux Builds

Work is already underway to add support for aarch64-linux builds to nixbuild.net, and so far it is looking good. With the current surge in performant ARM hardware (Apple M1, Ampere Altra etc), we think having aarch64 support in nixbuild.net is an obvious feature. It is also something that has been requested by our users.

We don’t know yet how the pricing of aarch64 builds will look, or what scalability promises we can make. If you are interested in evaluating aarch64 builds on nixbuild.net in an early access setting, just send us an email to support@nixbuild.net.

Provide an API over SSH and HTTP

Currently the nixbuild.net shell is the administrative tool we offer end users. We will keep developing the shell and make it more intuitive for interactive use. But will also add an alternative, more scriptable variant of the shell.

This alternative version will provide roughly the same functionality as the original shell, only more adapted to scripting instead of interactive use. The reason for providing such an SSH-based API is to make it easy to integrate nixbuild.net more tightly into CI and similar scenarios.

There is in fact already a tiny version of this API deployed. You can run the following command to try it out:

$ ssh eu.nixbuild.net api show public-signing-key
{"keyName":"nixbuild.net/bob-1","publicKey":"PmUhzAc4Ug6sf1uG8aobbqMdalxW41SHWH7FE0ie1BY="}

The above API command is in use by the nixbuild-action for GitHub. So far, this is the only API command implemented, and it should be seen as a very first proof of concept. Nothing has been decided on how the API should look and work in the future.

The API will also be offered over HTTP in addition to SSH.

Upload builds to binary caches

Adding custom binary caches that nixbuild.net can fetch dependencies from is supported today, although such requests are still handled manually through support.

We also want to support uploading to custom binary caches. That way users could gain performance by not having to first download build results from nixbuild.net and then upload them somewhere else. This could be very useful for CI setups that can spend a considerable amount of their time just uploading closures.

Provide an HTTP-based binary cache

Using nixbuild.net as a binary cache is handy since you don’t have to wait for any uploads after a build has finished. Instead, the closures will be immediately available in the binary cache, backed by nixbuild.net.

It is actually possible to use nixbuild.net as a binary cache today, by configuring an SSH-based cache (ssh://eu.nixbuild.net). This works out of the box right now. You can even use nix-copy-closure to upload paths to nixbuild.net. We just don’t yet give any guarantees on how long store paths are kept.

However, there are benfits to providing an HTTP-based cache. It would most probably have better performance (serving nar files over HTTP instead of using the nix-store protocol over SSH), but more importantly it would let us use a CDN for serving cache contents. This could help mitigate the fact that nixbuild.net is only deployed in Europe so far.

Support builds that use KVM

The primary motivation for this is to be able to run NixOS tests (with good performance) on nixbuild.net.

Thank You!

Finally we’d like to thank all our users. We look forward to an exciting new year with lots of Nix builds!