---
title: "Run a Nix remote builder and binary cache on exe.dev"
description: "Send Nix builds from your Mac, Linux machines, and CI to a big exe.dev VM over SSH, and serve its store as a signed, private binary cache."
url: https://exe-staging.dev/deploy/nix-builder
category: "Developer tools"
updated: 2026-11-04
---

# Run a Nix remote builder and binary cache on exe.dev

One big VM that builds for every machine and caches what it built.

- **Software:** Nix remote builder (https://nix.dev/manual/nix/latest/advanced-topics/distributed-builds)
- **Source:** https://github.com/nix-community/harmonia
- **License:** LGPL-2.1 (Nix), MIT (Harmonia)
- **Time:** 15 minutes
- **Example VM name:** `my-nix-builder` (yours will differ; on the VM, `hostname` prints it)
- **Machine-readable:** https://exe-staging.dev/deploy/nix-builder/manifest.json

## What you get

- Nix with nix-daemon, `exedev` as a trusted user, building on every core, reachable over `ssh-ng://` through exe.dev's SSH
- A signing key made on the VM, its public key printed for clients
- Harmonia serving the store as a signed binary cache on port 8000, private behind the HTTPS proxy
- A nightly harmonia-gc that keeps 25% of the disk free and everything from the last 14 days, and Nix's own collection when a build runs low
- A test build on the VM whose output the cache serves with the VM's signature

Running on the VM:

- **Nix 2.35**: the official multi-user install; nix-daemon builds as exedev's trusted proxy, on every core, data in `/nix/store, /nix/var/nix/db`
- **Harmonia 3.3**: serves /nix/store as a binary cache, signing every path; the harmonia systemd service, port 8000, data in `/etc/harmonia/harmonia.toml`
- **Signing key**: my-nix-builder-1, made on the VM; clients trust its public half, data in `/etc/nix/cache-key.sec, /etc/nix/cache-key.pub`
- **Garbage collection**: harmonia-gc nightly, plus Nix's own min-free during builds

How it is reached:

- ssh-ng://my-nix-builder.exe.xyz (private): remote builds, for SSH keys you add with ssh-key add --vm
- https://my-nix-builder.exe.xyz/ (private): the binary cache, for your exe.dev login or a VM token

## Deploy with Shelley

Open https://exe-staging.dev/new?prompt=Set%20up%20Nix%20remote%20builder%20on%20this%20VM%20by%20following%20the%20exe.dev%20recipe%20at%20https%3A%2F%2Fexe-staging.dev%2Fdeploy%2Fnix-builder.md%20from%20top%20to%20bottom.%0A%0AThe%20recipe%27s%20example%20VM%20name%20is%20my-nix-builder.%20This%20VM%27s%20name%20is%20the%20output%20of%20%60hostname%60%3A%20use%20it%20wherever%20the%20recipe%20writes%20my-nix-builder.%20Skip%20the%20step%20that%20creates%20the%20VM%3B%20you%20are%20on%20it.%0A%0AThis%20recipe%20needs%20a%20200GB%20disk%2C%20and%20a%20new%20VM%27s%20may%20be%20smaller.%20Before%20you%20install%20anything%2C%20run%20%60df%20-h%20%2F%60.%20If%20its%20size%20is%20under%20180G%2C%20stop%20and%20ask%20me%20to%20grow%20the%20disk%2C%20either%20with%20this%20link%3A%20https%3A%2F%2Fexe.dev%2Fsuggest%3Fcommand%3Dresize%2520my-nix-builder%2520--disk%253D200GB%20or%2C%20on%20my%20computer%2C%20with%20%60ssh%20exe.dev%20resize%20my-nix-builder%20--disk%3D200GB%60.%20When%20I%20say%20it%27s%20done%2C%20run%20%60sudo%20resize2fs%20%22%24%28findmnt%20-no%20SOURCE%20%2F%29%22%60%2C%20check%20%60df%20-h%20%2F%60%20again%2C%20and%20carry%20on.%0A%0AYou%20are%20done%20when%20all%20of%20these%20are%20true.%20Check%20each%20one%20yourself%3A%0A-%20Nix%20with%20nix-daemon%2C%20%60exedev%60%20as%20a%20trusted%20user%2C%20building%20on%20every%20core%2C%20reachable%20over%20%60ssh-ng%3A%2F%2F%60%20through%20exe.dev%27s%20SSH%0A-%20A%20signing%20key%20made%20on%20the%20VM%2C%20its%20public%20key%20printed%20for%20clients%0A-%20Harmonia%20serving%20the%20store%20as%20a%20signed%20binary%20cache%20on%20port%208000%2C%20private%20behind%20the%20HTTPS%20proxy%0A-%20A%20nightly%20harmonia-gc%20that%20keeps%2025%25%20of%20the%20disk%20free%20and%20everything%20from%20the%20last%2014%20days%2C%20and%20Nix%27s%20own%20collection%20when%20a%20build%20runs%20low%0A-%20A%20test%20build%20on%20the%20VM%20whose%20output%20the%20cache%20serves%20with%20the%20VM%27s%20signature%0A%0ARun%20%22Set%20up%20the%20builder%22%20on%20this%20VM%2C%20skipping%20VM%20creation.%20Everything%0Ain%20%22Connect%20a%20Linux%20machine%22%2C%20%22Connect%20a%20Mac%22%2C%20%22Use%20it%20from%20CI%22%2C%20and%0Athe%20ssh-key%20and%20generate-api-key%20commands%20run%20on%20the%20owner%27s%20computers%2C%0Anot%20here.%20Don%27t%20add%20SSH%20keys%20or%20tokens%20to%20the%20owner%27s%20account%20yourself.%0AUse%20the%20official%20installer%20from%20nixos.org%20%28multi-user%2C%20--daemon%29%3B%20don%27t%0Ainstall%20Determinate%20Nix%20here.%20Install%20Harmonia%20from%20nixpkgs%20as%20written.%0ANever%20print%20%2Fetc%2Fnix%2Fcache-key.sec.%20Verify%3A%20%60nix%20store%20info%20--store%0Adaemon%60%20works%20as%20exedev%3B%20%60sudo%20-i%20nix%20config%20show%20trusted-users%60%0Aincludes%20exedev%3B%20%60curl%20-s%20localhost%3A8000%2Fnix-cache-info%60%20answers%3B%20the%0Atest%20build%27s%20narinfo%20has%20a%20%60Sig%3A%60%20line%20with%20this%20VM%27s%20key%20name%3B%0Aharmonia-gc.timer%20is%20enabled%3B%20if%20system-features%20has%20kvm%2C%20the%0Anix-daemon%20kvm.conf%20drop-in%20is%20in%20place%20and%20%2Fdev%2Fkvm%20is%20mode%200666%0Aafter%20%60sudo%20systemctl%20restart%20nix-daemon%60.%20Finish%20with%20this%20VM%27s%20CPUs%2C%20memory%2C%20and%0Afree%20disk%2C%20the%20contents%20of%20%2Fetc%2Fnix%2Fcache-key.pub%2C%20the%20builder%27s%0Asystem-features%20%28%60sudo%20-i%20nix%20config%20show%20system-features%60%29%2C%20and%20the%0Aowner%27s%20next%20steps%20from%20%22Connect%20a%20Linux%20machine%22%20and%20%22Connect%20a%20Mac%22%0Awith%20this%20VM%27s%20name%20and%20public%20key%20filled%20in.%0A%0AFinish%20with%20the%20URLs%20I%20should%20open.%0A to create a VM whose agent, Shelley, follows this recipe. From a terminal:

```sh
curl -fsSL https://exe-staging.dev/deploy/nix-builder/prompt.txt | ssh exe.dev new --disk=200GB --prompt=/dev/stdin
```

It needs a 200 GB disk. The terminal command makes the VM that size; after the button, Shelley checks and, if the disk is smaller, gives you a link to grow it.

Or follow the steps below by hand.

Nix can send builds to another machine over SSH and fetch finished
packages from an HTTP binary cache. This page makes one exe.dev VM both.
Your Mac, your Linux machines, and your CI send it the derivations they
can't or shouldn't build themselves; it builds them on all its cores,
keeps the results in its store, and serves that store as a signed
binary cache, so the next machine that needs the same package downloads
it instead of building it.

exe.dev does the plumbing. The builder is reached through exe.dev's SSH
with keys you scope to this one VM; the cache is behind the
[HTTPS proxy](/docs/proxy), private until you share it, with tokens for
machines that can't log in. There's no port to open and no certificate to
manage.

## How it works

- **Nix** on `my-nix-builder`, installed the official way, with
  `exedev` as a [trusted user](https://nix.dev/manual/nix/latest/command-ref/conf-file#conf-trusted-users).
  Clients connect with `ssh-ng://my-nix-builder.exe.xyz`, the Nix
  daemon protocol over SSH. With `builders-use-substitutes`, the builder
  downloads dependencies from cache.nixos.org itself, instead of your
  laptop uploading them.
- **[Harmonia](https://github.com/nix-community/harmonia)** serves
  `/nix/store` on port 8000, so `https://my-nix-builder.exe.xyz/` is a
  binary cache of everything the builder has built or downloaded. It
  signs each path with the VM's key; clients trust that key and nothing
  else from it.
- **Garbage collection** keeps the disk from filling: Nix collects when
  a build runs low on space, and `harmonia-gc` runs nightly, keeping
  anything from the last 14 days.

A machine can use the builder, the cache, or both. A laptop usually
wants both; a CI job that only needs what was already built wants the
cache alone, with a read-only token.

## Set up the builder

<details class="dp-details" id="create-the-vm" data-manual-setup>
<summary>Create the VM and install Nix and Harmonia</summary>

### Create the VM

On your computer, create a VM with room to build. VM names are global,
so pick any free name and use it wherever this page says
`my-nix-builder`:

```bash
ssh exe.dev new --name=my-nix-builder --cpu=8 --memory=16GB --disk=200GB
ssh my-nix-builder.exe.xyz
```

Use as many vCPUs as your [plan](/docs/billing/personal) has (`--cpu`
up to your capacity, 2 GB of memory per vCPU). The rest of this section
runs **on the VM**. Check what you got:

```bash
nproc; free -h; df -h /
```

If `df` shows less than about 180 GB, grow the disk from your computer
(`ssh exe.dev resize my-nix-builder --disk=200GB`), then on the VM
`sudo resize2fs $(findmnt -no SOURCE /)`.

### Install Nix

The [official installer](https://nixos.org/download/), multi-user, as
nixos.org recommends for Linux with systemd:

```bash
curl --proto '=https' --tlsv1.2 -sSfL https://nixos.org/nix/install | sh -s -- --daemon --yes
```

It ends with `Nix won't work in active shell sessions until you restart
them`. Clients connect with commands that don't start a login shell, so
put the Nix commands on the default `PATH`, then configure the daemon:

```bash
for p in nix nix-daemon nix-store; do
  sudo ln -sf /nix/var/nix/profiles/default/bin/$p /usr/local/bin/$p
done
sudo tee -a /etc/nix/nix.conf >/dev/null <<'EOF'
experimental-features = nix-command flakes
trusted-users = root exedev
max-jobs = auto
cores = 0
secret-key-files = /etc/nix/cache-key.sec
min-free = 20G
max-free = 60G
EOF
```

- `trusted-users`: clients log in as `exedev`, and Nix only builds
  derivations sent over SSH for a trusted user. A trusted user can do
  anything the daemon can, which makes any key that reaches this VM root
  on it (as `exedev`'s `sudo` already does).
- `max-jobs = auto` runs one build per vCPU at a time; `cores = 0` lets
  each one use every core for its own `make -j`.
- `secret-key-files` signs everything built here.
- `min-free` and `max-free`: when a build finds less than 20 GB free, Nix
  collects garbage until 60 GB is.

Make the signing key. Its name is the VM's, with `-1` so you can
rotate to `-2` later:

```bash
sudo -i sh -c "umask 077; nix key generate-secret --key-name $(hostname)-1 > /etc/nix/cache-key.sec"
sudo -i sh -c 'nix key convert-secret-to-public < /etc/nix/cache-key.sec > /etc/nix/cache-key.pub'
sudo systemctl restart nix-daemon
cat /etc/nix/cache-key.pub; echo
```

The last line is the public key, something like
`my-nix-builder-1:KISg07I/HaZWK0r8bXNquX6lfQ79MAVlLnk6NVoeHgY=`; clients
need it. Check the daemon:

```bash
nix store info --store daemon
sudo -i nix config show system-features
```

The first says `Trusted: 1`. The second lists what this builder can
do; you'll copy it into each client's builder line. It includes `kvm`
(needed for NixOS VM tests) when the VM has `/dev/kvm`, which on
exe.dev depends on your plan ([Enterprise](/docs/enterprise) has
nested virtualization). On Ubuntu, `/dev/kvm` is only for the `kvm`
group, and a sandboxed build can't use it, so a NixOS test silently
falls back to emulation: in our test, 167 seconds instead of 35. If the
list has `kvm`, open it to builds whenever the daemon starts:

```bash
sudo mkdir -p /etc/systemd/system/nix-daemon.service.d
printf '[Service]\nExecStartPre=-/bin/chmod 0666 /dev/kvm\n' \
  | sudo tee /etc/systemd/system/nix-daemon.service.d/kvm.conf
sudo systemctl daemon-reload && sudo systemctl restart nix-daemon
```

### Serve the store with Harmonia

Install Harmonia from nixpkgs into a profile of its own, and run its
`harmonia-cache` on port 8000 as a systemd service. systemd hands it the
signing key; Harmonia itself runs as a throwaway user that can read the
store and nothing else:

```bash
sudo -i nix profile add --profile /nix/var/nix/profiles/harmonia nixpkgs#harmonia
sudo ln -sf /nix/var/nix/profiles/harmonia/bin/harmonia-gc /usr/local/bin/harmonia-gc
sudo mkdir -p /etc/harmonia
sudo tee /etc/harmonia/harmonia.toml >/dev/null <<'EOF'
bind = "[::]:8000"
priority = 50
EOF
sudo tee /etc/systemd/system/harmonia.service >/dev/null <<'EOF'
[Unit]
Description=Harmonia Nix binary cache
After=network-online.target nix-daemon.socket
Wants=network-online.target

[Service]
Type=notify
ExecStart=/nix/var/nix/profiles/harmonia/bin/harmonia-cache
Environment=CONFIG_FILE=/etc/harmonia/harmonia.toml
Environment=SIGN_KEY_PATHS=%d/sign-key
LoadCredential=sign-key:/etc/nix/cache-key.sec
DynamicUser=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
NoNewPrivileges=yes
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now harmonia
curl -s localhost:8000/nix-cache-info
```

The last command prints `StoreDir: /nix/store`, `WantMassQuery: 1`, and
`Priority: 50`. Priority 50 makes clients ask cache.nixos.org (40)
first for what it has, and this cache for the rest.

### Collect garbage nightly

[`harmonia-gc`](https://github.com/nix-community/harmonia/releases/tag/harmonia-v3.3.0)
is a faster `nix-collect-garbage` that can stop once enough is free and
spare recent paths. Run it every night:

```bash
sudo tee /etc/systemd/system/harmonia-gc.service >/dev/null <<'EOF'
[Unit]
Description=Nix garbage collection (harmonia-gc)

[Service]
Type=oneshot
Environment=PATH=/nix/var/nix/profiles/default/bin:/usr/bin:/bin
ExecStart=/nix/var/nix/profiles/harmonia/bin/harmonia-gc --ensure-free 25%% --keep-recent 14d
EOF
sudo tee /etc/systemd/system/harmonia-gc.timer >/dev/null <<'EOF'
[Unit]
Description=Nightly Nix garbage collection

[Timer]
OnCalendar=03:15
RandomizedDelaySec=30m
Persistent=true

[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now harmonia-gc.timer
sudo systemctl start harmonia-gc.service
journalctl -u harmonia-gc -n 3 --no-pager -o cat
```

On a fresh disk it finds nothing to do. See
[Keep the disk from filling](#keep-the-disk-from-filling) for what it
keeps and why.

### Check it end to end

Build something on the VM and fetch its signed entry from the cache:

```bash
out=$(nix build --no-link --print-out-paths --impure --expr \
  'with import (builtins.getFlake "nixpkgs") {}; runCommand "cache-check" {} "date > $out"')
curl -s localhost:8000/$(basename $out | cut -c1-32).narinfo | grep -E '^(StorePath|Sig):'
```

You should see the store path and a `Sig:` line that starts with
`my-nix-builder-1:`.

</details>

After Deploy, Shelley has done all of the above and shows you the
builder's public key. The rest happens on the machines that will use it.

## Connect a Linux machine

This is for any Linux machine with Nix's multi-user install (a
workstation, a server, another exe.dev VM). The Nix daemon, which runs
as root, does the SSH and the downloads, so the key and the token are
root's.

**On the machine**, make an SSH key for the daemon and show it:

```bash
sudo install -d -m 700 /root/.ssh
sudo ssh-keygen -q -t ed25519 -N '' -C "nix-daemon@$(hostname)" -f /root/.ssh/nix-builder
sudo cat /root/.ssh/nix-builder.pub
```

**On your computer**, let that key into the builder and nowhere else,
and make a token for the cache (paste the public key between the
quotes):

```bash
ssh exe.dev ssh-key add --vm=my-nix-builder 'ssh-ed25519 AAAA... nix-daemon@workstation'
ssh exe.dev ssh-key generate-api-key --vm=my-nix-builder --label=nix-cache-workstation --exp=1y
```

The second command prints a token starting with `exe1.`. It gets
through the HTTPS proxy to this VM only, and Harmonia only serves
reads, so it can't change anything.

**Back on the machine**, connect once as root, which records the host
key and proves the key works, then tell Nix about the builder, the
cache, and the token. Put in your builder's public key, the token, and
the `system-features` from setup (drop `kvm` if the builder lacks it):

```bash
sudo ssh -i /root/.ssh/nix-builder -o StrictHostKeyChecking=accept-new my-nix-builder.exe.xyz nix --version
echo 'ssh-ng://my-nix-builder.exe.xyz x86_64-linux /root/.ssh/nix-builder 8 1 nixos-test,benchmark,big-parallel,kvm - -' \
  | sudo tee /etc/nix/machines
printf 'Cache token: '; read -rs T; echo
printf 'machine my-nix-builder.exe.xyz\nlogin nix\npassword %s\n' "$T" | sudo install -m 600 /dev/stdin /etc/nix/netrc
unset T
sudo tee -a /etc/nix/nix.conf >/dev/null <<'EOF'
extra-experimental-features = nix-command flakes
builders-use-substitutes = true
extra-substituters = https://my-nix-builder.exe.xyz
extra-trusted-public-keys = my-nix-builder-1:KISg07I/HaZWK0r8bXNquX6lfQ79MAVlLnk6NVoeHgY=
netrc-file = /etc/nix/netrc
EOF
sudo systemctl restart nix-daemon
```

The `nix --version` prints the builder's Nix version. In the
[machines line](https://nix.dev/manual/nix/latest/command-ref/conf-file#conf-builders),
`8` is how many builds to send at once (the builder's vCPUs) and `1` its
speed factor; `/etc/nix/machines` is where Nix looks by default. Check
both halves:

```bash
sudo -i nix store info --store 'ssh-ng://my-nix-builder.exe.xyz?ssh-key=/root/.ssh/nix-builder'
curl -s --netrc-file <(sudo cat /etc/nix/netrc) https://my-nix-builder.exe.xyz/nix-cache-info
```

The first says `Trusted: 1`; the second prints the cache info from
setup. Now build something that isn't in any cache, on the builder only:

```bash
nix build --max-jobs 0 -L --impure --expr \
  'with import (builtins.getFlake "nixpkgs") {}; hello.overrideAttrs { NONCE = "'$RANDOM'"; }'
```

The log says `building '/nix/store/…-hello-2.12.3.drv' on
'ssh-ng://my-nix-builder.exe.xyz'`. `--max-jobs 0` means "never build
here"; without it, Nix builds locally while it has free slots and sends
the rest. Put `max-jobs = 0` in `nix.conf` to make that the default on
a laptop.

### NixOS and nix-darwin

On NixOS, the same settings go in your configuration (nix-darwin takes
the same options); the key and `netrc` files stay where you made them:

```nix
{
  nix.distributedBuilds = true;
  nix.buildMachines = [{
    hostName = "my-nix-builder.exe.xyz";
    protocol = "ssh-ng";
    systems = [ "x86_64-linux" ];
    sshKey = "/root/.ssh/nix-builder";   # /var/root/.ssh/nix-builder on a Mac
    maxJobs = 8;
    supportedFeatures = [ "nixos-test" "benchmark" "big-parallel" "kvm" ];
  }];
  nix.settings = {
    builders-use-substitutes = true;
    extra-substituters = [ "https://my-nix-builder.exe.xyz" ];
    extra-trusted-public-keys = [ "my-nix-builder-1:KISg07I/HaZWK0r8bXNquX6lfQ79MAVlLnk6NVoeHgY=" ];
    netrc-file = "/etc/nix/netrc";
  };
}
```

## Connect a Mac

A Mac builds its own `aarch64-darwin` (or `x86_64-darwin`) packages and
sends `x86_64-linux` ones to the builder: NixOS systems, Docker images
from `dockerTools`, anything your servers run. The daemon runs as root,
whose home on macOS is `/var/root`.

On the Mac, make the daemon's key and show it:

```bash
sudo install -d -m 700 /var/root/.ssh
sudo ssh-keygen -q -t ed25519 -N '' -C "nix-daemon@$(hostname -s)" -f /var/root/.ssh/nix-builder
sudo cat /var/root/.ssh/nix-builder.pub
```

Add it and make a token exactly as in
[Connect a Linux machine](#connect-a-linux-machine) (`ssh exe.dev ssh-key
add --vm=my-nix-builder …` and `generate-api-key`), then:

```bash
sudo ssh -i /var/root/.ssh/nix-builder -o StrictHostKeyChecking=accept-new my-nix-builder.exe.xyz nix --version
echo 'ssh-ng://my-nix-builder.exe.xyz x86_64-linux /var/root/.ssh/nix-builder 8 1 nixos-test,benchmark,big-parallel,kvm - -' \
  | sudo tee /etc/nix/machines
printf 'Cache token: '; read -rs T; echo
printf 'machine my-nix-builder.exe.xyz\nlogin nix\npassword %s\n' "$T" | sudo tee /etc/nix/netrc >/dev/null
sudo chmod 600 /etc/nix/netrc; unset T
```

Then add the same five lines as on Linux (from
`extra-experimental-features` to `netrc-file`) to Nix's configuration and restart the daemon:

- **Nix from nixos.org:** append them to `/etc/nix/nix.conf`, then
  `sudo launchctl kickstart -k system/org.nixos.nix-daemon`.
- **Determinate Nix:** append them to `/etc/nix/nix.custom.conf` (its
  `nix.conf` is managed), then
  `sudo launchctl kickstart -k system/systems.determinate.nix-daemon`.
- **nix-darwin:** use the [module settings](#nixos-and-nix-darwin) and
  `darwin-rebuild switch`.

Try it with a Linux build:

```bash
nix build -L --impure --expr \
  'with import (builtins.getFlake "nixpkgs") { system = "x86_64-linux"; }; runCommand "where" {} "uname -sm > $out"'
cat result
```

`Linux x86_64`: it was built on the VM and copied back.

## Use the cache without the builder

A machine that only needs what the builder already has (a deploy
target, a teammate's laptop, a CI job for pull requests) needs no SSH
key: give it a token and the same `nix.conf` lines without the machines
file. Anything it builds itself stays on that machine.

For exe.dev VMs, skip the token: a
[VM-to-VM integration](/docs/integrations-vm-to-vm) reaches the cache
with a key that exe.dev holds and injects, so it's on no VM. On your
computer:

```bash
ssh exe.dev integrations add http-proxy --name nix-cache \
  --target https://my-nix-builder.exe.xyz/ --peer --attach tag:nix
ssh exe.dev tag my-app nix
```

On every VM tagged `nix`, the cache is `https://nix-cache.int.exe.xyz`:

```
extra-substituters = https://nix-cache.int.exe.xyz
extra-trusted-public-keys = my-nix-builder-1:KISg07I/HaZWK0r8bXNquX6lfQ79MAVlLnk6NVoeHgY=
```

Making the cache public (`ssh exe.dev share set-public my-nix-builder`)
opens the cache to everyone, and never the builder.

## Use it from CI

CI is where a builder and cache earn their keep: every job starts with
an empty store, and without help it rebuilds whatever cache.nixos.org
doesn't have. The same key, token and settings work from any CI that
runs Nix on Linux.

### GitHub-hosted runners

Make a key and a token for CI on your computer, and store both as
repository secrets (this uses the [GitHub CLI](https://cli.github.com/);
**Settings → Secrets and variables → Actions** works too):

```bash
ssh-keygen -q -t ed25519 -N '' -C nix-ci -f nix-ci
ssh exe.dev ssh-key add --vm=my-nix-builder < nix-ci.pub
gh secret set NIX_BUILDER_SSH_KEY < nix-ci
ssh exe.dev ssh-key generate-api-key --vm=my-nix-builder --label=nix-cache-ci --exp=1y --json \
  | jq -r .token | gh secret set NIX_CACHE_TOKEN
rm nix-ci nix-ci.pub
```

Then, in a workflow:

```yaml
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: cachix/install-nix-action@v31
        with:
          extra_nix_config: |
            builders = ssh-ng://my-nix-builder.exe.xyz x86_64-linux /root/.ssh/nix-builder 8 1 nixos-test,benchmark,big-parallel - -
            builders-use-substitutes = true
            extra-substituters = https://my-nix-builder.exe.xyz
            extra-trusted-public-keys = my-nix-builder-1:KISg07I/HaZWK0r8bXNquX6lfQ79MAVlLnk6NVoeHgY=
            netrc-file = /etc/nix/netrc
      - name: Connect to the Nix builder
        env:
          SSH_KEY: ${{ secrets.NIX_BUILDER_SSH_KEY }}
          CACHE_TOKEN: ${{ secrets.NIX_CACHE_TOKEN }}
        run: |
          sudo install -d -m 700 /root/.ssh
          printf '%s\n' "$SSH_KEY" | sudo install -m 600 /dev/stdin /root/.ssh/nix-builder
          sudo ssh -i /root/.ssh/nix-builder -o StrictHostKeyChecking=accept-new my-nix-builder.exe.xyz true
          printf 'machine my-nix-builder.exe.xyz\nlogin nix\npassword %s\n' "$CACHE_TOKEN" \
            | sudo install -m 600 /dev/stdin /etc/nix/netrc
      - run: nix build --max-jobs 0 -L .#default
```

Everything the job builds is built on `my-nix-builder` and stays in its
cache, so the next job, on any branch, downloads it.

The builder key is root on the builder, so don't hand it to code you
don't trust. For pull requests from forks, which can't read secrets
anyway, leave the builder out; for pull requests from people with
write access you trust, it's the same as giving them `ssh` to the VM.
A job that should only read the cache uses `NIX_CACHE_TOKEN` alone.

### Jobs on exe.dev sandboxes

With [GitHub Actions on exe.dev](/deploy/github-actions), each job runs
on a fresh copy of a base VM. Install Nix on the base, with the
cache lines from [Use the cache without the builder](#use-the-cache-without-the-builder)
pointing at the `nix-cache` integration, and attach that integration by
the `ci` tag the job VMs carry (`--attach tag:ci`). Then:

- Every job VM starts with the base's `/nix/store`, so it begins warm,
  and fetches the rest from the cache with no token on any VM.
- For builds the base doesn't have, add the builder as on GitHub's
  runners, with the key from a GitHub secret. Don't put the key on the
  base: every job VM would have it.

Refresh the base's store now and then
(`ssh my-ci-base.exe.xyz 'nix build .#default && sync'` from a checkout),
as with any other warm cache.

## Build times

Measured on 2026-11-04 with Nix 2.35.4 and nixpkgs `0bb2eed`, between a
2 vCPU / 4 GB exe.dev VM as the client (about a small laptop's worth)
and an 8 vCPU / 32 GB builder. Each package was rebuilt from source
with every dependency already in cache.nixos.org, by adding an unused
environment variable to it. Times are `nix build` wall-clock, after
evaluation.

| Package | On the 2 vCPU client | On the 8 vCPU builder | From the cache |
|---|---|---|---|
| `hello` | 27 s | 23 s | under 1 s |
| `ripgrep` | 2 min 13 s | 57 s | under 1 s |
| `python3` | 10 min 0 s | 4 min 42 s | 2.7 s (137 MB) |
| `cmake` | 38 min 24 s | 11 min 58 s | 1.1 s (71 MB) |

The builder column includes copying the result back to the client.
Small packages gain little: `hello` spends its time in a `configure`
script that runs on one core either way. The bigger the build, the more
of it runs in parallel and the more the builder's cores pay off, but no
package uses them all the whole time (Python's test suite, cmake's
bootstrap), so 4× the cores gave 2 to 3× the speed. Several builds at
once, as when a NixOS system or a CI matrix needs many packages, use
them better.

"From the cache" is the second machine to want the same output: one
download from the builder instead of any build, timed here between two
exe.dev VMs. From your own network, it goes at your connection's speed.

The first remote build of a package also downloads its build
dependencies to the builder (the Rust toolchain for `ripgrep`, say),
once: 81 s instead of 57 s. A NixOS VM test
([`runNixOSTest`](https://nixos.org/manual/nixos/stable/#sec-call-nixos-test-outside-nixos))
that boots one machine took 35 s on the builder with KVM.

## Keep the disk from filling

The store only grows: every build adds its outputs and the
dependencies it downloaded. Two things trim it.

- **During builds**, Nix itself: when a build finds less than `min-free`
  (20 GB) free, it deletes garbage until `max-free` (60 GB) is free.
- **Every night**, `harmonia-gc --ensure-free 25% --keep-recent 14d`:
  if less than a quarter of the disk is free, it deletes garbage until a
  quarter is, but never a path added in the last 14 days.

"Garbage" is anything no GC root refers to, and on a builder that is
almost everything: the outputs it built for your clients went back to
them, and nothing on the builder keeps them. That's fine for a cache,
as long as the cache is mostly recent work, which `--keep-recent`
ensures. Two things to know:

- Collection goes by when a path arrived in the store, not when it was
  last downloaded. A package built a month ago and fetched every day can
  still go; the next client that needs it builds it again, and it's back.
- To keep something no matter what (a release, the current production
  system), add a GC root for it on the builder:
  `sudo -i nix build --out-link /nix/var/nix/gcroots/releases/app-1.4 'github:you/app/v1.4'`.
  Delete the link to let it go.

See what's there and what would go, on the VM:

```bash
df -h /
sudo harmonia-gc --dry-run | tail -1
```

If the disk is often full, grow it from your computer with
`ssh exe.dev resize my-nix-builder --disk=400GB`, then
`sudo resize2fs $(findmnt -no SOURCE /)` on the VM, or keep fewer days.
exe.dev bills disk by what's used, not its size.

## What it costs

The builder is an ordinary VM on your plan. On
[Personal](/docs/billing/personal), all your VMs share one capacity, so
an 8 vCPU builder on the $75-a-month plan uses all of it while it
builds and none of it while idle, and the plan's 400 GB of disk covers
the store. Cache downloads leave exe.dev and count toward the plan's
200 GB of outbound bandwidth a month; a team pulling big closures every
day should keep an eye on `ssh exe.dev billing usage`.

For builds that only happen sometimes, a
[sandbox](/deploy/sandbox) is billed by the second: a 16 vCPU / 32 GiB
one is $0.84 an hour. Its store is gone when you delete it, though, so
keep the cache on a regular VM.

## Back up and restore

The one thing to keep is the signing key. Every client trusts
`my-nix-builder-1`, and a new key means editing every client. On your
computer:

```bash
ssh my-nix-builder.exe.xyz 'sudo cat /etc/nix/cache-key.sec' > my-nix-builder-1.sec
chmod 600 my-nix-builder-1.sec
```

Store it as you'd store any secret. The store itself is a cache: what
you lose with it, the builder rebuilds or downloads again. To move the
builder, copy the whole VM (`ssh exe.dev cp my-nix-builder
my-nix-builder-2`), or set up a new one and restore the key to
`/etc/nix/cache-key.sec` (mode 600, owned by root) before starting
Harmonia.

If the key leaks, make `my-nix-builder-2` as in setup, replace the
public key on every client, and restart Harmonia. Paths signed only by
the old key stop being trusted, which is the point.

## Update

On the VM:

- **Nix:** `sudo -i nix upgrade-nix`, then
  `sudo systemctl restart nix-daemon`. Clients and builder don't need
  the same version, but keep them within a few releases.
- **Harmonia:** `sudo -i nix profile upgrade --profile /nix/var/nix/profiles/harmonia --all`,
  then `sudo systemctl restart harmonia`.
- **Tokens and keys:** the cache tokens above expire in a year. Make a
  new one with `generate-api-key` and replace `/etc/nix/netrc` (or the
  CI secret). `ssh exe.dev ssh-key list` shows every key and token;
  `ssh exe.dev ssh-key remove <name>` revokes one, such as a lost
  laptop's `nix-daemon@…` key.

## Troubleshooting

On the client, add `-L -v` to `nix build` to see which machine it picks
and why. On the builder, `journalctl -u nix-daemon -f` and
`journalctl -u harmonia -f`.

- **`Failed to find a machine for remote build!`** The derivation needs
  a system or feature the machines line doesn't list. A Linux build from
  a Mac needs `x86_64-linux`; a NixOS test needs `kvm` and `nixos-test`,
  which only a builder with `/dev/kvm` can offer.
- **`cannot build on 'ssh-ng://…': … Permission denied (publickey)`.**
  The daemon's key isn't added, or you added your own instead. Test as
  root: `sudo ssh -i /root/.ssh/nix-builder my-nix-builder.exe.xyz true`.
  `ssh exe.dev ssh-key list` shows the keys scoped to the VM.
- **`Host key verification failed`.** Root never connected before; run
  the `sudo ssh … -o StrictHostKeyChecking=accept-new` line once.
- **`nix-daemon: command not found`.** The symlinks in
  `/usr/local/bin` on the builder are missing; recreate them.
- **`… is not trusted` or `cannot add path … lacks a signature`.**
  `exedev` isn't in `trusted-users` on the builder, or the daemon wasn't
  restarted after the change.
- **The cache is never used** (`nix build` builds everything again).
  Check the token: `curl -s --netrc-file <(sudo cat /etc/nix/netrc)
  https://my-nix-builder.exe.xyz/nix-cache-info` must print the cache
  info, not an HTML login page (missing or expired token). Then check
  that `extra-trusted-public-keys` matches `/etc/nix/cache-key.pub`
  exactly; a wrong key shows as `ignoring substitute … because it's not
  signed by any of the keys in 'trusted-public-keys'`.
- **A path you just built isn't found in the cache** (`there is no
  substituter that can build it`), though `curl` finds its `.narinfo`.
  The client asked for it before it existed and remembers the miss for
  an hour. Add `--option narinfo-cache-negative-ttl 0` to that one
  `nix build`.
- **A NixOS test is very slow**, and its log (`nix log` on the builder)
  says `failed to initialize kvm: Permission denied` and `falling back
  to tcg`. `/dev/kvm` isn't open to builds; see the `kvm.conf` step in
  [Install Nix](#install-nix).
- **`No space left on device`** during a build. Run
  `sudo harmonia-gc --ensure-free 25%` on the builder, then see
  [Keep the disk from filling](#keep-the-disk-from-filling).

## Frequently asked questions

### Do I need NixOS?

No. The builder is the default Ubuntu image with Nix installed, and clients can be NixOS, nix-darwin, plain Nix on macOS or Linux, or a CI runner. What they share is the Nix daemon protocol over SSH and the binary cache format over HTTPS.

### Can it build for my Apple Silicon Mac, or for aarch64-linux?

No. exe.dev VMs are x86-64 Linux, so the builder builds `x86_64-linux` (and `i686-linux`). Your Mac keeps building its own `aarch64-darwin` packages; what it hands off is Linux: NixOS systems, container images, packages for your servers and CI. `aarch64-linux` would need QEMU emulation, which is slow enough that a native arm64 builder is the better answer.

### How is this different from Cachix or nixbuild.net?

Those are hosted services: Cachix stores and serves your build outputs, nixbuild.net runs your builds, and both charge by use. Here one VM does both, its disk is the cache, its CPUs are the builder, and the cost is the VM. You run the upkeep (updates, garbage collection), and there's one machine rather than a fleet.

### Why Harmonia, and not nix-serve-ng or Attic?

Harmonia serves the builder's own `/nix/store` directly, signs on the fly, compresses with zstd, and ships `harmonia-gc`, so a builder that is also the cache needs nothing else. [nix-serve-ng](https://github.com/aristanetworks/nix-serve-ng) does the same job and works with the same steps, minus the GC. [Attic](https://github.com/zhaofengli/attic) is a different shape: a separate, deduplicating store that clients push to with tokens, for many teams or object storage. Use it when the cache must outlive the builder.

### Who can use the cache and the builder?

Only you, until you say otherwise. The cache is behind the [HTTPS proxy](/docs/proxy), which lets in your exe.dev login and tokens made with `ssh-key generate-api-key --vm`. The builder accepts SSH keys you add with `ssh-key add --vm`, which reach this VM and no other. A builder key is root on the builder, so give it only to machines and workflows you trust; a cache token can only read.

### How big should the VM be?

As many vCPUs as your plan allows, 2 GB of memory per vCPU, and a disk with room for a few generations of everything you build: 200 GB is a good start for a team, and disks can grow later. On Personal, the builder shares your plan's CPUs with your other VMs and can use all of them when they're idle.


## Related

- [Run GitHub Actions jobs on exe.dev, a fresh VM per job](https://exe-staging.dev/deploy/github-actions.md): Every job on a fresh copy of a warm VM, billed by the second.
- [Use exe.dev as a remote Docker host](https://exe-staging.dev/deploy/docker-host.md): Keep the docker CLI. Send the builds, the containers, and the fan noise to a VM.
- [Sandboxes on exe.dev](https://exe-staging.dev/deploy/sandbox.md): A VM with its own CPU and memory, billed by the second while it runs.
- [Forgejo](https://exe-staging.dev/deploy/forgejo.md): Self-hosted Git forge with exe.dev SSO
- [Gitea](https://exe-staging.dev/deploy/gitea.md): Self-hosted Git forge with exe.dev SSO
- [Host your Tangled repos on your own knot](https://exe-staging.dev/deploy/tangled.md): Your git repos on your own VM, with tangled.org on top.