# Containerised build at a canonical path

`make container-build` compiles capOS inside a container with the worktree
bind-mounted at `/build`. The host build path is unchanged and remains the
default.

## Worker build-mode decision

As of `2026-07-29 05:02 UTC`, the autonomous worker remains host-mode by
default. A repeated comparison now covers the complete subset of compiling
universal gates measured in both modes: `capos-rt-check`,
`capos-service-check`, and `shell`. `workflow-check` is now also admitted, but
has not been added to that historical timing bundle. With target trees drained
before every run and all 121 cacheable requests hitting in both arms, three
container samples had a 31.45-second median versus 28.80 seconds on the host.
The container was 2.65 seconds (9.2%) slower at the median. The result is
deliberately limited to this gate bundle; it is not a measurement of universal
gates the container cannot yet run or of task-specific QEMU proofs.

The matching cold-target prime inventory establishes the composition of the
observed hit delta: all 19 host misses were first-party `capos-*` units, while
every registry dependency and build-std unit hit. The container recorded three
more hits over the identical cacheable-request multiset, so those additional
hits came from the first-party portion rather than dependency reuse. Because
the two live caches differed in size and population, this prime does not
separate canonical-path reuse from cache population or eviction. The
controlled cross-worktree measurements above establish the canonical-path
mechanism separately. On the fully warm bundle, that mechanism did not recover
the container wrapper overhead.

The route design below covers every compiling universal gate and every
applicable host-side QEMU artifact without changing the current wiring. It also
shows why a switch is not ready: eight universal targets compile, only four
are currently admitted, four compiling universal gates remain unrouted, and
only `run-smoke` has a prebuilt-artifact path. Until the route is implemented
and rehearsed as one fail-closed worker gate set, `container-build` remains an
explicit compilation and packaging optimization and the host remains the
canonical worker build and gate mode.

## Why a container

sccache hashes the compiler process's current working directory
unconditionally for every Rust compilation, so two worktrees at different host
paths never share a first-party cache entry. This is upstream-tracked and
unfixed: [mozilla/sccache#2652](https://github.com/mozilla/sccache/issues/2652)
(the Rust hash key ignores `SCCACHE_BASEDIRS`) and
[#2595](https://github.com/mozilla/sccache/issues/2595) (this exact worktree
case). `SCCACHE_BASEDIRS`, `--remap-path-prefix` and `-Zremap-cwd-prefix` were
each measured at zero additional hits; the first does not apply to Rust at all
despite its README, and the latter two rewrite the artifact rather than the key.

Making the directory genuinely identical is the only mechanism that works, and
it is the only one that stays honest: the artifacts really do carry `/build`,
rather than carrying whichever worktree happened to compile them first.

## What must match for a cache hit

| Must be identical | Need not be identical |
|---|---|
| The container mount path (`/build`) | The host worktree path |
| `CARGO_HOME` (`/cargo`) | The container uid |
| The toolchain mount path (`/rust`) | `HOME` |
| Every other `CARGO_*` variable, name and value | Any other variable no compilation reads |
| Any other variable a compilation reads | The base image |
| The rustc binary's *content* | |

The toolchain mount path belongs in the left column for the same reason the
worktree path does: cargo runs each unit's rustc with the working directory set
to that package's own root, and a `-Zbuild-std=core,alloc` unit's package root is
under `<toolchain>/lib/rustlib/src/rust/library/`. Every userspace build alias in
`.cargo/config.toml` uses `-Zbuild-std`, so parametrizing the toolchain mount
would silently lose every `core` and `alloc` hit. `capos-container-build.sh`
hardcodes `/rust`, which is why this is a latent trap rather than a live defect,
and the `ubuntu:24.04` versus `python:3.12-slim` spike is not counter-evidence:
it held the mount at `/rust` in both arms.

The environment rule has two halves, and only the first is unconditional:

- **Every `CARGO_*` variable enters the key regardless of whether anything reads
  it**, because `env!()` could. An unrelated `CARGO_UNRELATED=1` is enough to
  zero the hit rate. `capos-container-build.sh` therefore starts the container
  from an empty environment and sets the `CARGO_*` set explicitly, so no host
  variable can leak in.
- **Any other variable enters the key when the compilation actually reads it.**
  rustc records such a read as an `env-dep` in dep-info, and sccache tracks it.
  Measured with a build script emitting `cargo:rustc-env` and a library doing
  `env!()`: changing a non-`CARGO_` variable missed, changing it back hit, and
  each rlib carried the value it was built with.

Reading the first half alone invites the conclusion that a non-`CARGO_`
variable is invisible to the key. It is not, and the difference matters here:
`kernel/build.rs` emits `CAPOS_BUILD_COMMIT` / `CAPOS_BUILD_TIMESTAMP` /
`CAPOS_BUILD_VERSION` as `cargo:rustc-env`, and `kernel/src/cap/system_info.rs`
reads them with `env!()`. Treating them as invisible would be a
wrong-build-identity bug: a kernel serving another commit's identity. The
observed behaviour is the safe one -- see the reuse limits below.

## Measured reuse

Two worktrees at the same commit, at different host paths, building `make
kernel` (init plus kernel, so both the `x86_64-unknown-capos` userspace target
and the `x86_64-unknown-none` kernel target, including the kernel link). Each
arm starts from an empty cache and clean target directories, builds the first
worktree to populate, then builds the second.

| Arm | Populate | Second worktree | Cross-worktree hits |
|---|---|---|---|
| Host | 0 hits / 91 misses, 87 s | 79 hits / 12 misses, 64 s | 86.8% |
| Container at `/build` | 0 hits / 91 misses, 91 s | 91 hits / 0 misses, 41 s | 100% |

This table is **not reproduced**: it was taken once per cell, on a host running
concurrent builds, and the run recorded neither the cache-isolation procedure nor
the stats capture. Treat the hit and miss counts as indicative and the wall-clock
figures as unreliable; the reuse *ratio* is the durable part, and the artifact
identity below is separately reproducible. Re-measuring needs two worktrees at
one commit, a dedicated `SCCACHE_DIR` per arm so the live host cache is neither
read nor disturbed, drained target directories, and
`CAPOS_CONTAINER_SCCACHE_STATS=1` for the container arm (the server dies with the
container, so its counters can only be read from inside).

The headline host rate is high because this goal is dominated by registry
dependencies, and those already share: cargo runs each unit's rustc with the
working directory set to that package's own root, so a crate unpacked under
`CARGO_HOME` -- or a `-Zbuild-std` unit compiled from the rustup `rust-src`
component -- is worktree-independent already. The indicative run recorded 12
host misses but did not preserve a per-unit miss inventory. The earlier
inventory here incorrectly counted `capos-kernel`, which sccache declines as
non-cacheable rather than recording as a miss. Treat the 12 only as the
unreproduced aggregate in the table, not as a durable first-party unit count.

So the aggregate rate is workload- and source-delta-dependent. A higher
first-party fraction does not by itself predict a low host baseline, because
the aggregate also contains registry and build-std units whose working
directories are already stable. The canonical-path mechanism applies more
narrowly: an unchanged cacheable first-party compilation whose host cwd differs
between working trees can reuse the container entry because its cwd is `/build`
in both. It does not imply a 0%-to-100% change over a goal's aggregate cacheable
set, especially when the second tree changes a shared first-party dependency.
The measurements below do not retain the per-unit inventory needed to separate
that subset from already-path-independent units.

### The different-commit case, measured -- and `capos-kernel` is not a miss at all

The equal-commit table above is the best case. The autopilot case is two
worktrees on different branch tips, and it was left unmeasured. Measured
`2026-07-26 14:34 UTC`, the result corrects the explanation this document
previously gave.

Procedure, against the live shared container cache (so the numbers are the ones a
real worker sees, not an isolated ideal):

Create two detached worktrees at the same mainline commit. In the second, commit
one added line in a kernel source file, so its tip, its sources and its
`CAPOS_BUILD_COMMIT` all differ from the first's. Then, in each worktree in turn:

```sh
CAPOS_CONTAINER_SCCACHE_STATS=1 make container-build CONTAINER_GOALS=kernel
```

The stats variable is required: the container gets its own network namespace, so
its sccache server dies with the container and its counters can only be read from
inside.

| Worktree | Requests | Executed | Hits | Misses | Non-cacheable |
|---|---|---|---|---|---|
| First worktree | 123 | 91 | 91 | 0 | 29 |
| Second worktree, +1 kernel commit | 123 | 91 | 91 | 0 | 29 |

**First-party cross-worktree reuse at a different commit is 100% of the cacheable
set: 91 of 91, zero misses.** Both worktrees are at different host paths, and the
second differs from the first in a kernel source file, its commit and its build
identity.

The zero-miss column is the part that needed explaining, because a changed kernel
source cannot possibly hit. It does not: **`capos-kernel` is never offered to the
cache.** Isolated directly -- touch one kernel source in the second
worktree and rebuild, so the kernel is the only unit that recompiles:

```
Compile requests            3
Compile requests executed   0
Cache hits                  0
Cache misses                0
Non-cacheable calls         1     reason: crate-type
Non-compilation calls       2
```

So the earlier claim here -- that `CAPOS_BUILD_COMMIT` in the unit's key makes
`capos-kernel` *miss* across branch tips -- described the wrong mechanism. sccache
declines the kernel's `rustc` invocation outright on `crate-type`, in every build,
at every commit, in both host and container mode. The environment rule is real
(see above) but has no bearing on the kernel, because the kernel's compile never
reaches key computation. The consequence is a **ceiling, not a regression**: the
container path cannot accelerate the kernel compile-and-link, and neither can the
host path. No kernel cache hit is lost by choosing container mode for an explicit
build. Of the 123 requests in this goal, 29 are non-cacheable (`crate-type` 15,
`missing input` 9, stdin 5) and 3 are non-compilation probes; the reuse claim
covers the other 91.

Also directly observed: two container builds of the same tree at different commits
produce a byte-identical `capos-init` and a *different* kernel. With the mechanism
above, that difference is simply the kernel being compiled fresh both times, not a
cache decision.

### Compile-heavy goals across a source-changing commit

Recorded at `2026-07-29 01:26 UTC` with sccache 0.16.0 and container image
`sha256:21bafca13a2e0b732618c831c8a473172e28358ad90b2b57b4af26557ec862cf`.
The comparison used two detached working trees at different host paths. The
first was commit `8a97f0db3a8482f5c334b4096d944176a94ac3f8`; the second started
there, changed `capos-abi/src/lib.rs` from
`PROCESS_CAP_SLOT_LIMIT = 256` to the equivalent
`PROCESS_CAP_SLOT_LIMIT = 1 << 8`, and committed the change. This gives the
second tree a different tip and source while deliberately preserving runtime
behavior. Because all three goals depend on `capos-abi`, the comparison is not
the easier case where only unrelated metadata changed.

Each goal and arm had a separate initially empty cache. Before populating one,
both working trees were drained. `make clean` currently omits `shell/target` and
removes the tracked `target/.gitkeep`, so the exact drain also ran:

```sh
cargo clean --manifest-path shell/Cargo.toml
git restore target/.gitkeep
```

A recursive check then required every directory named `target` to contain no
entry other than `target/.gitkeep`. Without the explicit shell clean, the
container build is correctly refused because `shell/target` still holds host
output.

For a host pair, `$CACHE` was unique to that goal and arm, and `$PORT` was a
unique sccache server port so neither the ordinary host server nor its cache was
read or stopped:

```sh
SCCACHE_DIR="$CACHE" SCCACHE_SERVER_PORT="$PORT" sccache --zero-stats
/usr/bin/time -f 'wall_seconds=%e' \
  env SCCACHE_DIR="$CACHE" SCCACHE_SERVER_PORT="$PORT" \
  SCCACHE_CACHE_SIZE=10G make -C "$BASE" "$GOAL"
SCCACHE_DIR="$CACHE" SCCACHE_SERVER_PORT="$PORT" sccache --show-stats

SCCACHE_DIR="$CACHE" SCCACHE_SERVER_PORT="$PORT" sccache --zero-stats
/usr/bin/time -f 'wall_seconds=%e' \
  env SCCACHE_DIR="$CACHE" SCCACHE_SERVER_PORT="$PORT" \
  SCCACHE_CACHE_SIZE=10G make -C "$CHANGED" "$GOAL"
SCCACHE_DIR="$CACHE" SCCACHE_SERVER_PORT="$PORT" sccache --show-stats
SCCACHE_DIR="$CACHE" SCCACHE_SERVER_PORT="$PORT" sccache --stop-server
```

For the matching container pair, the wrapper's server is new for each
container, so its counters are already per-build:

```sh
/usr/bin/time -f 'wall_seconds=%e' \
  env CAPOS_CONTAINER_SCCACHE_DIR="$CACHE" \
  CAPOS_CONTAINER_SCCACHE_STATS=1 \
  make -C "$BASE" container-build CONTAINER_GOALS="$GOAL"

/usr/bin/time -f 'wall_seconds=%e' \
  env CAPOS_CONTAINER_SCCACHE_DIR="$CACHE" \
  CAPOS_CONTAINER_SCCACHE_STATS=1 \
  make -C "$CHANGED" container-build CONTAINER_GOALS="$GOAL"
```

`Populate` is the base tree with the empty cache. `Changed` is the second tree
against that populated cache. “Executed” is sccache's `Compile requests
executed`; every row also had two non-compilation calls, zero compilation
failures, and zero cache errors.

The command-wall column records the requested end-to-end observation, but the
arms are deliberately not compared. The host timer covers `make "$GOAL"` and
excludes its later stats read. The container timer covers image inspection,
reservation handling, host-side `capnp-ensure cue-ensure limine-ensure`,
container start and stop, the inner build, and the in-container stats dump. The
Cargo column comes from Cargo's own `Finished ... in ...` line and is the common
compile scope; Cargo rounds the two `demos` values shown in minutes to whole
seconds.

| Goal | Arm | Tree | Requests | Executed | Hits | Misses | Non-cacheable | Indicative command wall, unmatched scope | Cargo elapsed |
|---|---|---|---:|---:|---:|---:|---:|---:|---:|
| `demos` | Host | Populate | 500 | 126 | 0 | 126 | 372 | 108.52 s | 88 s |
| `demos` | Host | Changed | 500 | 126 | 82 | 44 | 372 | 65.39 s | 65 s |
| `demos` | Container | Populate | 500 | 126 | 0 | 126 | 372 | 86.58 s | 84 s |
| `demos` | Container | Changed | 500 | 126 | 91 | 35 | 372 | 63.45 s | 62 s |
| `shell` | Host | Populate | 40 | 25 | 0 | 25 | 13 | 54.41 s | 54.25 s |
| `shell` | Host | Changed | 40 | 25 | 21 | 4 | 13 | 35.85 s | 35.70 s |
| `shell` | Container | Populate | 40 | 25 | 0 | 25 | 13 | 55.38 s | 54.10 s |
| `shell` | Container | Changed | 40 | 25 | 22 | 3 | 13 | 37.59 s | 36.42 s |
| `capos-service-check` | Host | Populate | 37 | 24 | 0 | 24 | 11 | 52.71 s | 52.56 s |
| `capos-service-check` | Host | Changed | 37 | 24 | 19 | 5 | 11 | 34.96 s | 34.80 s |
| `capos-service-check` | Container | Populate | 37 | 24 | 0 | 24 | 11 | 54.38 s | 53.23 s |
| `capos-service-check` | Container | Changed | 37 | 24 | 20 | 4 | 11 | 36.08 s | 34.94 s |

The non-cacheable reason counts were identical between the two arms and both
trees:

| Goal | `crate-type` | `missing input` | `-` | `missing emit` |
|---|---:|---:|---:|---:|
| `demos` | 363 | 5 | 3 | 1 |
| `shell` | 6 | 4 | 3 | 0 |
| `capos-service-check` | 4 | 4 | 3 | 0 |

The container won aggregate cache reuse for every goal: 91 versus 82 hits for
`demos`, 22 versus 21 for `shell`, and 20 versus 19 for
`capos-service-check`. Across the three changed-tree builds that is 133 of 175
cacheable requests (76.0%) versus the host's 122 of 175 (69.7%). Because no
per-unit inventory was retained, this result does not say how many of the 11
additional hits were first-party units made reusable by `/build`, rather than a
different aggregate effect.

There is no valid winner from the command-wall column because its arm scopes
differ. Within Cargo's common compile scope, the changed-tree observations were
62 versus 65 seconds for container versus host on `demos`, 36.42 versus 35.70
seconds on `shell`, and 34.94 versus 34.80 seconds on
`capos-service-check`. Thus the container had the smaller Cargo time only for
`demos`; the host had the smaller time for the other two. The latter gaps were
0.72 and 0.14 seconds, and all values are single samples from a live
development host, so they are not evidence of a repeatable directional speedup.
The result supports neither a general container speedup nor a worker-default
switch.

### Repeated worker-gate wall time and unit inventory

Recorded from `2026-07-29 04:53 UTC` through `05:02 UTC` at commit
`061a98d6abee7c98623802781193ba614a2daff4`. The host was Ubuntu 24.04.4 LTS
on Linux 6.17.0, with eight Intel Xeon 2.80 GHz vCPUs and 7.7 GiB RAM. Docker
client and server were 29.5.3, sccache was 0.16.0, and the container image was
`sha256:21bafca13a2e0b732618c831c8a473172e28358ad90b2b57b4af26557ec862cf`.

The measured bundle was the same work in both arms and preserved runtime order:

```sh
make capos-rt-check capos-service-check shell
```

The timed command shapes were:

```sh
/usr/bin/time -f 'wall_seconds=%e' \
  env CAPOS_CONTAINER_SCCACHE_STATS=1 \
  make container-build \
    CONTAINER_GOALS='capos-rt-check capos-service-check shell'

/usr/bin/time -f 'wall_seconds=%e' \
  env SCCACHE_SERVER_PORT="$PORT" sh -c '
    sccache --zero-stats &&
    make capos-rt-check capos-service-check shell
    rc=$?
    sccache --show-stats
    sccache --stop-server >/dev/null
    exit "$rc"
  '
```

The container arm invoked those three goals in one `container-build` call. Its
timer covered wrapper preflight, pinned-tool verification, container start,
the inner Make invocation, in-container stats, and container stop. The host
timer covered a fresh sccache server, stats reset, the same Make invocation,
stats read, and server stop. Each arm first ran one prime against its existing
mode-specific cache. Before every timed sample, all target trees and the mode
marker were moved to a retained snapshot outside the worktree, leaving the
tracked `target/.gitkeep`; no output from an earlier sample remained visible.

The three timed samples per arm therefore had cold target trees and warm caches.
Every timed sample had the same sccache result: 190 requests, 121 executed,
121 hits, zero misses, 59 non-cacheable calls, and 10 non-compilation calls.
The non-cacheable reasons were also identical: 26 `crate-type`, 18 `missing
input`, and 15 `-`.

| Arm | Sample 1 | Sample 2 | Sample 3 | Mean | Median |
|---|---:|---:|---:|---:|---:|
| Host | 29.31 s | 28.59 s | 28.80 s | 28.90 s | 28.80 s |
| Container | 33.43 s | 29.87 s | 31.45 s | 31.58 s | 31.45 s |

The container median was 2.65 seconds (9.2%) slower; its mean was 2.68 seconds
(9.3%) slower. These are end-to-end command-wall measurements for the stated
bundle, not Cargo-only timers. They are three samples on one live host, so they
do not establish a portable performance constant. They do establish that the
canonical container path does not win this fully warm gate bundle on the
measured worker.

The primes were not used in that comparison because their pre-existing cache
contents differed. They remain useful for the requested unit inventory:

| Arm | Wall | Hits | Misses | Cache state after prime |
|---|---:|---:|---:|---|
| Host | 187.21 s | 102 | 19 | Existing 10 GiB cache at its bound |
| Container | 215.97 s | 105 | 16 | Existing cache, 772 MiB after the run |

The host prime ran sccache at debug level and retained every cache result. Its
19 misses were all first-party units:

| Unit | Misses | Hits in the same bundle |
|---|---:|---:|
| `capos` | 2 | 0 |
| `capos-abi` | 3 | 2 |
| `capos-capnp-build` | 2 | 3 |
| `capos-config` | 5 | 0 |
| `capos-posix` | 1 | 0 |
| `capos-rt` | 5 | 0 |
| `capos-service` | 1 | 0 |
| Registry dependencies and build-std | 0 | 97 |

The two arms executed the same 121 cacheable compilation requests. The host had
no dependency or build-std miss that the container could turn into an
additional dependency hit. The container's three additional hits therefore
came from first-party units, not dependencies. That identifies the unit class,
not the cause: the mode-specific live caches were differently sized and
populated, and the full host cache could have evicted first-party entries the
container cache retained. The prime timings likewise must not be compared as a
speed result.

Subsequent build measurements run their commands through
`tools/container-build/run-measurement.sh --run-id <id> -- <command>`. The
runner exposes its generated directory as `CAPOS_MEASUREMENT_SCRATCH`, places
it under the user's cache rather than the task-worktree parent, and removes it
after a successful command. A failing command retains the directory with a
`MEASUREMENT_FAILED` marker containing the run ID and exit status. Evidence
worth retaining must be copied to its durable evidence location before the
command returns success; the build trees themselves are disposable.

### Complete fail-closed worker gate route

The route is a design prerequisite, not current behavior. The runtime gate list
in `docs/workflow/gates.toml` is the source of truth and currently contains ten
universal gates. Eight compile or run Rust tests:

| Universal gate | Required container route |
|---|---|
| `workflow-check` | Admitted. It runs in the container, including the host-triple Rust test in `workflow-gates-test`; the image supplies Python 3 and PCRE2-capable ripgrep for the remaining checks. When the host-side runtime-policy lookup finds `.vibe-loop.toml`, the wrapper bind-mounts that same file read-only at its absolute path so `check-runtime-config --if-present` validates the policy instead of treating it as absent. |
| `webui-login-peer-logic-test` | Admit the complete host-test target and run it in the container. |
| `generated-code-check` | Admit the complete generation comparison, including `init` and its host-side generators, rather than prebuilding only `init`. |
| `capos-rt-check` | Already admitted. |
| `capos-rt-test` | Admit and run the host-test alias in the container. |
| `capos-tls-test` | Admit both the default-feature compile and enabled-feature test aliases. |
| `capos-service-check` | Already admitted. |
| `shell` | Already admitted. |

`fmt-check` invokes Cargo formatting but compiles nothing, and
`webui-secret-gating-check` is a source-shaped shell check. They may remain
host-side because they write no target output. That exception must remain
mechanically checked; a later compile added to either target makes it a
container-routed gate.

QEMU remains host-side. A complete route separates every applicable proof into
an immutable artifact build and a no-compile boot/assert phase:

1. Each QEMU target declares a named artifact target and an artifact contract:
   ISO, raw disk, or bundle; manifest; Cargo and demo feature sets; embedded
   payload goals; firmware assumptions; UART map; and immutable sidecars.
   Feature-dependent proofs get distinct named artifact targets. Raw Make
   variable assignments are not accepted as container goals.
2. `container-build` admits every declared artifact target and its complete
   prerequisite closure. This includes C and wasm payload builders when a
   selected proof needs them; the current blanket exclusion must be replaced by
   verified toolchain routes before those proofs are applicable.
3. The host proof takes a generalized `CAPOS_PREBUILT_ARTIFACT=1` path. It
   verifies the artifact digest, producing mode and image, manifest, feature
   contract, UART map, and every member of a bundle before starting QEMU.
   Mutable disk proofs copy an immutable container-built base into per-run
   scratch rather than modifying the recorded artifact.
4. The prebuilt branch removes the artifact's entire compile/package closure.
   Host-only drivers, browser helpers, manifest generation, C/wasm compilation,
   ISO mastering, and Cargo probes must either be prebuilt members of the
   verified bundle or be proven non-compiling harness work.
5. A generated checker derives the QEMU family from the Make rule database and
   fails unless every applicable target has exactly one declared artifact
   contract, a container-admitted artifact target, and a prebuilt branch.
   Under a container-mode marker, its dry run rejects any prebuilt route that
   reaches Cargo, rustc, clang, wasm builders, manifest generation, ISO/disk
   packaging, or the host build-mode guard. It also rejects stale declarations
   and artifact targets whose closure escapes the container allowlist.

`run-smoke` and `CAPOS_PREBUILT_ISO=1` are the current prototype for the
ISO/sidecar check, but they are not the complete route: no other QEMU target has
the branch, the sidecar does not identify a general proof-feature contract, and
disk or multi-artifact proofs do not fit an ISO-only variable.

The switch gate is one full rehearsal from a drained worktree: run all ten
universal gates in their declared mode, build every selected task-specific
artifact in the container, run each corresponding host prebuilt proof without
a mode change, run `container-build-mode-test` plus the new route-coverage
checker, and verify artifact provenance. Any unmapped compile or artifact
contract fails the rehearsal. Only that integrated result can support changing
the worker default; the design and wall-time measurement alone do not.

## The kernel linker script is not an obstacle

Worth recording, because it is easy to assume otherwise. The kernel linker
script is passed as a **relative** path from `.cargo/config.toml`
(`-C link-arg=-Tkernel/linker-x86_64.ld`), resolved against the compile working
directory, which is the workspace root. It does not travel through
`CARGO_ENCODED_RUSTFLAGS` -- the Makefile only ever *clears* that variable, for
the standalone wasm payload builds. So the kernel-target cache key carries no
absolute path from the link step, and the kernel ELF links inside the container
unchanged.

## Layout

- `tools/container-build/Dockerfile` -- distribution packages only: `gcc` plus
  glibc headers, `make`, `git`, Python 3, and ripgrep. No Rust toolchain, no
  capOS tool cache. Pinned by base-image digest.
- `tools/container-build/capos-container-build.sh` -- resolves the host paths
  and runs `make <goal>` inside the container. Its pre-flight refusals all come
  before any provisioning work: an unsupported goal, a missing container runtime,
  a missing `rustc`, a missing `sccache`, a missing toolchain, an image that is
  not present locally, a bind-mount path containing `,` or `=` (the `--mount`
  option string has no escaping for either), a container build of the same
  worktree already running, an active host build reservation, and a target
  directory in the other build mode.
  Ordering matters because the capnp build stamp depends on the `Makefile`, so any
  `Makefile` edit can otherwise make a from-source capnp build the price of being
  told `sccache` is missing. The image check matters because `docker run` treats a
  missing local image as a registry reference and reports a Docker Hub pull
  failure instead of "run `make container-image` first". The
  already-running check matters because `--rm` only removes a container after it
  exits, so a `SIGKILL` to the CLI leaves one running; the container is named
  from the worktree so concurrent builds of *different* worktrees stay allowed,
  which is the normal parallel case.
- `tools/container-build/capos-build-mode.sh` -- the build-mode guard and its
  self-test.
- `tools/container-build/container-goal-route-test.sh` -- asserts that
  `workflow-check` is admitted with its image tools while the named unrouted
  universal gates and representative QEMU routes still fail closed.
- `tools/container-build/run-measurement.sh` -- lifecycle wrapper for
  build-measurement scratch. It keeps scratch outside the worktrees root,
  removes it on success, and records the run identity when preserving a failed
  run. `run-measurement-test.sh` exercises both terminal paths.

The container is not a sandbox and `make container-build` is not a hardening
measure. `$HOME/.cargo` is bind-mounted read-write of necessity, since cargo
unpacks registry sources into it; that directory holds `credentials.toml` and a
`bin/` that is on the host `PATH`, so a build script running in the container can
read a crates.io token or overwrite a host binary. This is not a regression -- a
host build already runs the same build scripts with the same user's rights -- and
nothing in this path claims isolation. It is recorded so the container is not
later read as a trust boundary. `--network none` narrows what a build can reach,
but it is there to make an unexpected fetch fail loudly, not to contain a hostile
build script.

Everything version-pinned stays on the host and is bind-mounted:

- **Rust toolchain** -- `rust-toolchain.toml` is already the pin, and the
  measured cache key depends on the compiler's content rather than its path.
  Baking a toolchain into the image would add a second pin that can drift from
  the first, and would make image reproducibility a precondition for cache-key
  reproducibility. Mounted read-only at `/rust`.
- **capnp** -- `capos-config`'s build script executes it, so it must be
  reachable inside the container, and `tools/capnp-build` asserts its exact path
  under `CAPOS_TOOLS_ROOT`. The Makefile's existing SHA-256-pinned provisioning
  is unchanged; the tool cache is mounted read-only at `/capos-tools` and
  `capnp-ensure` inside the container only re-verifies it.
- **cue, limine** -- reachable inside the container through the same read-only
  `CAPOS_TOOLS_ROOT` mount, still SHA-256-pinned by the Makefile. Their
  *provisioning* stays on the host: the extraction recipes write into that mount,
  which fails inside the container as a read-only-filesystem error from within a
  `flock`, and pinned provisioning should not silently happen in a container
  anyway. So the host pre-flight runs `cue-ensure` and `limine-ensure` alongside
  `capnp-ensure`, and the container only consumes them.
- **xorriso** -- installed in the image, because unlike `cue` and `limine` it has
  no host provisioning to mount. It is the one image package that masters shipped
  artifact bytes, so it is recorded in `docs/trusted-build-inputs.md` rather than
  treated like `gcc` (whose output never reaches the shipped ELFs).
- **mdbook, typst** -- docs are not compilation. They stay on the host, outside
  this harness.

The git common directory is mounted at its own host path, read-only, because a
worktree's `.git` file names that absolute path and `kernel/build.rs` derives
the build identity (`CAPOS_BUILD_COMMIT`, `CAPOS_BUILD_TIMESTAMP`) from git.
Without it the kernel would stamp itself `unknown`. It is not a `CARGO_*`
variable and is not a compile working directory, so it cannot perturb the key.

## Cache directory

Container and host builds compute different keys for the same crate, so they
share nothing. Pointing both at one directory would only have them compete for
the same `SCCACHE_CACHE_SIZE` bound, so the container defaults to a separate
`$HOME/.cache/capos-container-sccache`. Override with
`CAPOS_CONTAINER_SCCACHE_DIR`, and collapse the two once the containerised path
is the only one in use. The consequence is that the host's cache bound and the
container's are independent, so the total budget is their sum; both are stated
explicitly rather than left to sccache's default.

Stating the container's bound explicitly is not optional tidiness. `HOME` inside
the container is a `tmpfs`, so no host `~/.config/sccache/config` is readable and
an operator's configured size (or any other option set there) does not apply.
Without an explicit `SCCACHE_CACHE_SIZE` the container silently falls back to the
built-in default, and this paragraph would be reasoning about a bound the
container never reads.

## Deliberately bounded surface

- **Running the QEMU gates inside the container is excluded.** `CONTAINER_GOALS` admits
  compilation and packaging, but not the gates themselves: they would need `qemu`
  in the image plus `--device /dev/kvm` and a supplementary `kvm` group, because
  `QEMU_KVM_FLAGS` falls back to TCG when `/dev/kvm` is unavailable -- correct, but
  far too slow for the smoke timeouts. Networking needs nothing extra: the
  harnesses use QEMU `user` networking, not a TAP device. The C and wasm payload
  goals are also out, because their recipes drive a C compiler and a second target
  triple that are not verified here. These are the current opt-in harness
  boundaries; the [complete fail-closed worker gate route](#complete-fail-closed-worker-gate-route)
  is authoritative for what must expand before a worker-default switch.
- **Booting a container-built ISO from a host QEMU target -- available, but only
  for `run-smoke` today.** `$(ISO)` depends on the compile targets, so a plain
  host QEMU target over a container-built tree is the mixed-mode hazard above and
  the guard refuses it. `CAPOS_PREBUILT_ISO=1` is the way through: it drops the ISO
  from the target's prerequisites and takes the UART map from the ISO's sidecar
  instead of recomputing it with a host `mkmanifest`, so the whole flagged path
  compiles nothing. Only `run-smoke` carries the flag today. Every other QEMU
  target intentionally reaches the ISO rule and is refused in a container-mode
  worktree -- including `test-default-boot`, which the
  [build and test guide](../build-run-test.md) names as *the* gate for the
  default `system.cue` boot and which the container can build (`all` produces
  `capos-default.iso`) but does not boot. The host-default worker decision above
  means this remains a deliberate boundary for the current opt-in harness. It
  is not sufficient for a worker-default switch; the route section above owns
  that expansion prerequisite.

## Packaging-mode provenance

The ISO sidecar binds its producing mode, container image, skipped verification
set, and `xorriso` identity to the final ISO digest. `make build-provenance`
validates that binding and fails closed rather than falling back to host mode.
For container packaging, the record therefore names the image-provided
`xorriso` that mastered the ISO and records the `ovmf-verify` skip; it does not
substitute the host's tool identity.

## One build mode per worktree, enforced

This is a correctness rule, not a performance note, and it is enforced rather
than documented. Cargo's freshness check for a workspace member does **not**
notice the switch: it tracks source files by a path relative to the package, and
the mount makes those relative paths and their mtimes identical. So a build in
one mode on top of the other mode's `target/` can mark a unit fresh and
**relink the rlib the other mode produced**, which carries that mode's build
root in its DWARF.

Both directions reproduce, and the host-over-container direction is the worse
one: the host path is the default that `make run`, the QEMU smokes and
`make build-provenance` all consume. Observed there: a plain `make kernel` over
a clean container-built target directory exited 0 with no warning and produced a
kernel in which one compilation unit (`capos-abi`) had
`DW_AT_comp_dir = /build`, a directory that does not exist on the host. Nothing
in the build output hinted at it. The artifact looked correct.

The extent is narrower than the mechanism suggests, which matters for triaging a
real occurrence: in that reproduction 92 of 93 units recompiled and exactly one
crate, `capos-abi`, survived stale.

The reason is not simply that `CARGO_HOME` differs between the modes. That
explains why registry units rebuild, but not why a first-party unit resolving no
registry path rebuilds -- `capos-lib` recompiled too. The operative rule is
cargo's dependency-output comparison: once a dependency's rlib is rewritten, every
transitive dependent is dirty regardless of its own dep-info. That predicts the
survivor set is exactly the **dependency-free first-party leaves**, which is a
property of the workspace graph rather than a constant. `capos-abi` is currently
the only workspace member with no dependencies and no build-dependencies, which
is why it was the only survivor. If it ever gains a dependency, nothing survives
a mode switch and the hazard disappears by accident; if a second dependency-free
crate is added, two survive.

The mechanism is inferred from cargo's fingerprint model rather than measured --
what was measured is the outcome, 92 of 93 -- but the graph premise is checkable
against the workspace manifests, and the practical consequence stands either way:
one mixed unit is enough to make the artifact wrong, and a mostly-reused build is
not what an occurrence looks like.

`tools/container-build/capos-build-mode.sh` closes this. It records the mode that
last wrote build output in `.capos-build-mode` (gitignored, repository root),
reserves the worktree through `.capos-build-mode-reservation`, and refuses a
build in the other mode while either that mode has an active reservation or any
target directory still holds output. `capos-container-build.sh` acquires its
explicit reservation before provisioning or `docker run`, reuses it in both
make invocations by passing the wrapper-issued reservation ID as a non-exported
make command-line variable, and releases it from its exit trap. `make
container-build-mode-test` is the self-test.

Ordinary host reservations are bound to the outermost `make` process by PID,
process start time, and boot identity. A later invocation reclaims one only
after that exact process identity is gone, avoiding PID-reuse guesses. The same
owner-generation rule applies to an explicitly named host-mode reservation;
only container-mode reservations remain unreclaimable from host process state
because their container may still be writing after the wrapper exits. A
replacement reservation records the validated new owner generation rather than
reusing fields read from the stale reservation.
Reservation inspection, stale-owner reclamation, replacement, and owned release
are serialized by the host's `flock`, so a contender cannot replace the
directory between another guard's identity check and rename. A persistent
rename failure refuses immediately instead of retrying while holding the lock.
The in-container re-entry must present the wrapper-issued ID, validates it
against the active reservation, and does not need `flock` inside the image.

The wrapper arms `EXIT`, `INT`, and `TERM` cleanup before attempting acquisition,
so an ordinary Ctrl-C or termination releases a reservation even during host
capnp provisioning or the remaining Docker preflight. Its explicit reservation
is released only by its matching identity. If the wrapper receives `SIGKILL`,
or the host loses power before its trap runs, the guard fails closed rather than
assuming the named container stopped writing.

Two independent make invocations in one worktree are refused even when both
request the same mode: host builds have distinct process-generation identities,
and container-mode re-entry without the wrapper-issued reservation ID fails.
The build-mode mix would be absent, but concurrent Cargo writes to one target
tree are already outside this harness's contract and can race independently of
provenance. Parallel builds in different worktrees use different reservation
and lock files and never serialize.
After verifying that no container for the worktree remains, an operator can
release such an orphan with the ID stored in
`.capos-build-mode-reservation/id`:

```sh
tools/container-build/capos-build-mode.sh --release-reservation \
  "$(sed -n 1p .capos-build-mode-reservation/id)"
```

Its reach into the Makefile is the assumption everything else here rests on, so
it is asserted rather than described. The guard is a prerequisite of
`capnp-ensure`, which covers every target that compiles capOS code for a capOS
target — but that is not the same as every cargo invocation, and the difference
was larger than a reading of the Makefile suggested.

Writing the exception list by hand produced six targets. Computing the
prerequisite closure from make's own rule database found **sixteen** cargo-invoking
rules that did not reach the guard. Three families mattered:

- `model-scheduler-nohz-loom`, a host-triple `capos-config` Loom test, and
  `wasi-env-negative-check`, a host-triple `mkmanifest` test, both write the
  **root `target/`** —
  the directory a container build fills. An unguarded write there can rewrite
  `target/.rustc_info.json` with the host sysroot inside a directory holding
  container output, which is the evidence the unmarked-worktree inference below
  depends on.
- Eleven `demos/wasi-*/target/wasm32-wasip1/release/*.wasm` artifact rules,
  reachable through their `wasi-*-build` parents. These compile `wasm32-wasip1`,
  which disproves the reasoning offered for the hand list — that every bypass was
  host-triple and therefore harmless. Their directories are already in the
  guard's discovery set, so the guard could refuse a build over output nothing
  had declared.
- `capos-tls-test` (deliberately capnp-free, since it builds only
  `-p capos-tls`), `webui-login-peer-logic-test`,
  `userspace-broker-policy-test`, `task-coordinator-logic-test`, `fuzz-build`
  and `fuzz-smoke`. The first two are `make check` members, so without the
  prerequisite `make check` in a container-mode worktree compiled into
  `demos/target/` before failing at `capos-rt-check`.

All of them now reach the guard; the eleven artifact rules take it as an
order-only prerequisite so a `.PHONY` dependency does not make every `.wasm`
perpetually out of date.

Only three cargo-invoking rules are deliberately unguarded, and
`tools/container-build/check-cargo-guard-coverage.py` asserts *set equality*
against exactly that allowlist: `fmt` and `fmt-check` compile nothing, and
`clean` is the remedy for a refusal, so guarding it would strand a worktree that
could then neither build in the recorded mode nor drain out of it. A newly
unguarded rule fails the gate, and so does an allowlisted rule that becomes
guarded. Widening the allowlist is a reviewed diff to that file.

Two checks run, deliberately: the closure computation above is complete but
parses `make -p` output, so it also asserts a floor on the number of cargo rules
it found — a degraded parse would otherwise report everything as guarded. Beside
it, `capos-build-mode.sh --check-guarded-targets` asks `make --dry-run` what a
sample of targets would actually run. One is complete, the other is
authoritative.

Three properties are worth stating because they are what make it fail closed:

- **Target directories are discovered, not listed.** Any directory named
  `target` at any depth that holds build output counts. The walk prunes `.git`
  and each target directory after finding it, so its cost does not scale with
  the contents of build output. This is the
  property that makes the guard cover the whole goal set rather than the default
  goal: nine standalone crate manifests build through `.cargo/config.toml`
  aliases -- `init/`, `demos/`, `shell/`, `capos-rt/`, `capos-service/`,
  `capos/`, `libcapos/`, `libcapos-posix/`, `capos-wasm/` -- each with its own
  `<crate>/target`, and the `demos/wasi-*` payloads are separate workspaces one
  level deeper. `CONTAINER_GOALS` is overridable, so `shell`, `demos`,
  `capos-rt-check` and friends are ordinary usage, and by the reuse argument
  above they are the goals with the *highest* first-party fraction -- the ones
  this path pays best on. A `target` plus `init/target` list would leave every one
  of them exposed. `make container-build-mode-test` asserts the whole set and the
  existing depth-four `tools/remote-session-client/src-tauri/target`: restricting
  discovery to the former depth bound or to the first two directories fails the
  gate.
- **An unmarked worktree is classified from evidence.** Cargo records the rustc
  sysroot in `<target-dir>/.rustc_info.json`, and the container's sysroot is the
  fixed mount path `/rust`, so an unmarked directory still declares which mode
  filled it. Without this, a worktree holding container output produced before
  the guard existed would be misread as host and permitted -- exactly the mix
  the guard is for. Every occupied directory is classified rather than the first
  one that answers, so a tree that is *already* mixed is reported as such and
  refuses both modes instead of being adopted as whichever mode was found first.
- **It refuses rather than wiping.** Discarding build output is the operator's
  call; the refusal names the directories and prints the command.

The reservation lock is acquired before the mode marker is checked or written.
This closes both empty-tree startup races: a host build cannot enter while the
container wrapper is starting Docker, and a container wrapper cannot enter
after a host guard passes but before that host build writes its first target
artifact. The self-test exercises both directions without pre-populating a
target directory, refuses a second independent same-mode owner, verifies that
the OS lock remains held while stale-owner reclaim is paused, and races
opposite-mode contenders over a stale host reservation to assert that exactly
one proceeds. It also proves that a persistent reclaim error returns after one
attempt, a stale explicit host reservation is reclaimed by owner generation,
the replacement retains the new owner's generation, an orphaned explicit
container reservation is never reclaimed, and an invalid owner reports its
actual preflight failure without waiting for a partial claim.

The cost is that the modes are exclusive for the *whole* worktree, not per goal.
After a full host ISO build, switching to container builds means discarding
every populated target directory, and vice versa. That is the same "pick one
mode per worktree" rule as before, now with a build-time signal instead of a
convention. Note `target/.gitkeep` is tracked, so `rm -rf target` needs it
restored afterwards.

Consequently a container-mode worktree cannot run the host gates
(`make check` and everything under it reaches `capnp-ensure`), and a host-mode
worktree cannot run `make container-build` without draining first. Drain a
worktree -- empty target directories and no marker -- to leave it free for
either mode. A completed host make may leave its process-bound reservation
record behind; the next guard reclaims it after verifying that owner generation
is no longer live.

## Artifacts

Two worktrees at different host paths, each built clean in a container, produce
a **byte-identical** kernel ELF (same SHA-256) and a byte-identical `capos-init`
ELF. The kernel contains no host path at all: every first-party compilation
unit's `DW_AT_comp_dir` is `/build`, and so are the embedded source-path strings.
A host build of the same tree carries the worktree path in both.

Measured on the `232e1791` container-built kernel:

```sh
K=target/x86_64-unknown-none/debug/capos-kernel
sha256sum "$K" init/target/x86_64-unknown-capos/release/capos-init
readelf --debug-dump=info "$K" | grep -c 'DW_AT_comp_dir.*: /build$'   # 49
readelf --debug-dump=info "$K" | grep DW_AT_comp_dir | sed 's/.*: //' \
    | sort -u | wc -l                                                  # 22
strings -a "$K" | grep -oE '^/build/[A-Za-z0-9._/-]+' | sort -u         # 5 paths
strings -a "$K" | grep -c '/home/'                                     # 0
```

The 49 are the first-party units; the 22 distinct `comp_dir` values are those
plus the `/cargo/registry/...` and `/rustc/<hash>` roots, which are
worktree-independent in either mode. The five embedded source-path strings are
`capos-rt/src/{panic,lib,entry}.rs`, `capos-config/src/validation.rs`, and the
generated `capos_capnp.rs` under the build-script `OUT_DIR`. All five counts are
commit-specific -- the `OUT_DIR` one embeds a build hash -- so re-measure rather
than quoting them.

That is what makes serving one worktree's cached object to another sound rather
than a fudge: the compiler really did run in `/build` both times. Key
normalisation alone would leave the artifact carrying whichever build root
compiled it first, which is unacceptable for a kernel that prints paths.

## Remaining caveats

- **`DW_AT_comp_dir` becomes `/build`.** Host-side debuggers resolving source
  from container-built artifacts need a path mapping (`set substitute-path
  /build <worktree>` in gdb).
- **Panic locations change for the standalone crates.** Workspace members are
  unaffected: cargo passes relative source paths for them, so their `file!()`
  strings are identical either way. `init/` and its outside-the-package path
  dependencies are not, because cargo passes those as absolute paths. The
  container-built kernel, which embeds `capos-init` via `include_bytes!`, carries
  `/build/capos-rt/src/panic.rs`, `/build/capos-rt/src/lib.rs`,
  `/build/capos-rt/src/entry.rs` and `/build/capos-config/src/validation.rs`
  where a host build of the same tree carries the worktree path. So init-side
  panic output names a directory that does not exist on the host.
- **The total sccache budget doubles.** The container defaults to its own cache
  directory (see above), and neither cache is a subset of the other, so the
  bound is the host's plus the container's -- currently 10G each. Both are set
  explicitly (`SCCACHE_CACHE_SIZE`, and `CAPOS_CONTAINER_SCCACHE_CACHE_SIZE` for
  the container) rather than inherited from sccache's default. A server that is
  already running keeps the size it started with.
- **Build mode is recorded in provenance, and the two are not comparable.**
  `make build-provenance` reports the artifact-bound `Build mode`, container
  image, skipped verification set, and producing `xorriso` identity from the ISO
  sidecar. A host artifact and a container artifact differ by construction, so a
  comparison across modes flags the coincident mode change while stating that
  the record pair alone cannot prove it caused artifact-hash drift; see the
  retention and comparison policy in `docs/trusted-build-inputs.md`.
- **The image is a build input.** It is pinned by digest and inventoried in
  `docs/trusted-build-inputs.md`, which also carries the refresh procedure.