A Batch of nixbuild.net Updates

We have not written about nixbuild.net for over a year now, but that doesn’t mean nothing has happened. Quite the opposite - we’ve been really busy!

This blog post will try to summarize some of what we have been working on behind the scene the last couple of years. Read on to hear about nixbuild.net going enterprise, a brand new web UI and lots of new functionality!

Self-Hosting nixbuild.net

A big thing that has kept us busy the last year is packaging nixbuild.net for self-hosting inside enterprise setups. We’ve had a number of companies reaching out to us inquiring about such setups. The reason for wanting to self-host nixbuild.net are several: internal legal requirements, network bottlenecks, cost control and the ability to use hardware that is tricky to offer in the public service.

One of our enterprise customers self-hosts nixbuild.net in production running 10K+ builds daily. A very welcome side-effect of adapting nixbuild.net for self-hosting is that many scenarios that are seldom or never triggered by users of our public service are uncovered by enterprise usage. This has made nixbuild.net even more reliable and capable under high load.

We are in a stage where we are happy to work closely with companies, figuring out how to best deploy and integrate nixbuild.net into their environments. Our goal is to eventually be able to offer self-hosted nixbuild.net as an off-the-shelf product. Actually, if you are on AWS, the deployment of nixbuild.net is already very simple. We will publish a blog post with more details on this in the near future, but the short rundown is this:

• We provide an EC2 AMI that runs NixOS and all necessary nixbuild.net services.
• The user deploys this AMI using her preferred method of EC2 provisioning.
• The user provides nixbuild.net configuration through cloud-init or by simply putting configuration files in place on the EC2 instance.
• The nixbuild.net configuration includes details on what types of EC2 instances to use for running the builds, and how many builder instances that your nixbuild.net deployment should be allowed to provision.

You can then simply use your nixbuild.net EC2 server as remote Nix builder, and get all the nixbuild.net niceties of automatic builder provisioning, scale-to-zero, automatic CPU and memory allocation, strict build isolation etc.

If your company is interested in self-hosting nixbuild.net, please reach out to us!

Finally, a Web UI

We are proud to say that we now have a brand-new web UI available to all nixbuild.net users! This includes a much simplified sign-up flow, where the Hanko service is used to provide and secure users’ login details. It is now easier than ever getting started with nixbuild.net!

The web UI is a complement to the tried-and-true SSH-based admin shell. We are gradually building out the web functionality to cover everything you can do in the shell today, and more. For the moment you can get by using the web UI to get started with nixbuild.net, adding SSH keys or creating access tokens. For configuring settings or setting up billing you will still need to use the nixbuild.net shell.

Our plan ahead is to keep the SSH-based shell which is very appreciated by many users and offers an efficient interface for managing your nixbuild.net account. The web UI will be expanded with full ability to manage your account, as well as pages to present and visualize builds and Nix store contents in ways that the terminal shell cannot.

Support for More Types of Caches and Substituters

During the last year, nixbuild.net has gained new support for S3 and improved support for Cachix.

nixbuild.net can now use the following sources when substituting (downloading) cached store paths:

• Public HTTP caches, like cache.nixos.org
• Public Cachix caches
• Authenticated Cachix caches
• S3 buckets (AWS or compatible providers)

For more information, see the documentation for the substituters setting. nixbuild.net can use multiple substituters of different types at the same time. It tries to do as much as possible in parallel, for example, all substituters are queried concurrently about available narinfo files. The substituter that answers first is used.

nixbuild.net can also populate binary caches, not only with build outputs but also their inputs. By activating this functionality, you don’t have to do any post-build cache uploads from your Nix client machine, it will instead be performed automatically by nixbuild.net as part of the build. Also in this case, nixbuild.net will try to parallelize as much as possible. Build inputs will for example start uploading at the same time the build starts.

nixbuild.net can now upload store paths to the following types of targets:

• Cachix caches
• S3 buckets (AWS or any compatible provider)

For more information, see the documentation for the caches setting.

More Knobs to Tweak

We have exposed more settings to end users lately. Some of these settings existed internally already, and some are newly implemented.

These new settings correspond to the identically named Nix settings:

• hashed-mirrors
• impure-env
• max-silent-time
• timeout

These new settings lets you control resource allocation:

• default-cpu
• default-mem-per-cpu
• max-cpu
• max-mem
• min-cpu
• min-mem

For a complete list of settings, check out the documentation.

Support for Custom Nix Store Directories

nixbuild.net supports using Nix store directories other than the default /nix/store one. You don’t have do anything special in nixbuild.net to enable this, it just works. If your Nix client is building something using a custom Nix store directory, nixbuild.net will automatically setup the build sandbox with the custom directory and make all build inputs available as usual.

Self-Managed Signing Keys

By default, nixbuild.net creates a unique Nix signing key for every account. This key is used to sign all build outputs and all paths that you upload to nixbuild.net. In practice this is not something you have to care that much about. Nix doesn’t require remote builders to sign builds, and you don’t have to setup your Nix client with the public keys that corresponds to your account’s signing key. This is because Nix has roughly the same level of trust for remote builds as for local builds.

However, inside nixbuild.net, Nix signatures are central, because the set of store paths available in a session depends on which signatures are trusted in that session. To control which signatures to trust, you can use the trusted-public-keys setting. Like all settings, it can be specified on the SSH key level or even on the environment of individual SSH sessions.

This flexibility is perhaps not useful to all nixbuild.net users, but it can be valuable for power users that have embraced Nix signatures as a way to track build provenance and control store access. For example, you could have your deployment pipeline verify that only store paths signed by a specific key are deployed to your production environment.

We have now added support for letting you control the signing keys that should be used, instead of relying on keys auto-created by nixbuild.net. You can even setup your configuration so that your secret signing keys are not persistently stored in nixbuild.net, but only accessible by nixbuild.net during builds or uploads.

To read more about Nix signing key in general and more details on how nixbuild.net uses signing keys, check out our newly added documentation. There you’ll also learn about how to setup access-tokens to lock down sub-users of your nixbuild.net account to specific trusted public keys.