Containerised build at a canonical path

make container-build compiles capOS inside a container with the worktree bind-mounted at /build. The host build path is unchanged and remains the default.

Worker build-mode decision

As of 2026-07-29 05:02 UTC, the autonomous worker remains host-mode by default. A repeated comparison now covers the complete subset of compiling universal gates measured in both modes: capos-rt-check, capos-service-check, and shell. workflow-check is now also admitted, but has not been added to that historical timing bundle. With target trees drained before every run and all 121 cacheable requests hitting in both arms, three container samples had a 31.45-second median versus 28.80 seconds on the host. The container was 2.65 seconds (9.2%) slower at the median. The result is deliberately limited to this gate bundle; it is not a measurement of universal gates the container cannot yet run or of task-specific QEMU proofs.

The matching cold-target prime inventory establishes the composition of the observed hit delta: all 19 host misses were first-party capos-* units, while every registry dependency and build-std unit hit. The container recorded three more hits over the identical cacheable-request multiset, so those additional hits came from the first-party portion rather than dependency reuse. Because the two live caches differed in size and population, this prime does not separate canonical-path reuse from cache population or eviction. The controlled cross-worktree measurements above establish the canonical-path mechanism separately. On the fully warm bundle, that mechanism did not recover the container wrapper overhead.

The route design below covers every compiling universal gate and every applicable host-side QEMU artifact without changing the current wiring. It also shows why a switch is not ready: eight universal targets compile, only four are currently admitted, four compiling universal gates remain unrouted, and only run-smoke has a prebuilt-artifact path. Until the route is implemented and rehearsed as one fail-closed worker gate set, container-build remains an explicit compilation and packaging optimization and the host remains the canonical worker build and gate mode.

Why a container

sccache hashes the compiler process’s current working directory unconditionally for every Rust compilation, so two worktrees at different host paths never share a first-party cache entry. This is upstream-tracked and unfixed: mozilla/sccache#2652 (the Rust hash key ignores SCCACHE_BASEDIRS) and #2595 (this exact worktree case). SCCACHE_BASEDIRS, --remap-path-prefix and -Zremap-cwd-prefix were each measured at zero additional hits; the first does not apply to Rust at all despite its README, and the latter two rewrite the artifact rather than the key.

Making the directory genuinely identical is the only mechanism that works, and it is the only one that stays honest: the artifacts really do carry /build, rather than carrying whichever worktree happened to compile them first.

What must match for a cache hit

The toolchain mount path belongs in the left column for the same reason the worktree path does: cargo runs each unit’s rustc with the working directory set to that package’s own root, and a -Zbuild-std=core,alloc unit’s package root is under <toolchain>/lib/rustlib/src/rust/library/. Every userspace build alias in .cargo/config.toml uses -Zbuild-std, so parametrizing the toolchain mount would silently lose every core and alloc hit. capos-container-build.sh hardcodes /rust, which is why this is a latent trap rather than a live defect, and the ubuntu:24.04 versus python:3.12-slim spike is not counter-evidence: it held the mount at /rust in both arms.

The environment rule has two halves, and only the first is unconditional:

• Every CARGO_* variable enters the key regardless of whether anything reads it, because env!() could. An unrelated CARGO_UNRELATED=1 is enough to zero the hit rate. capos-container-build.sh therefore starts the container from an empty environment and sets the CARGO_* set explicitly, so no host variable can leak in.
• Any other variable enters the key when the compilation actually reads it. rustc records such a read as an env-dep in dep-info, and sccache tracks it. Measured with a build script emitting cargo:rustc-env and a library doing env!(): changing a non-CARGO_ variable missed, changing it back hit, and each rlib carried the value it was built with.

Reading the first half alone invites the conclusion that a non-CARGO_ variable is invisible to the key. It is not, and the difference matters here: kernel/build.rs emits CAPOS_BUILD_COMMIT / CAPOS_BUILD_TIMESTAMP / CAPOS_BUILD_VERSION as cargo:rustc-env, and kernel/src/cap/system_info.rs reads them with env!(). Treating them as invisible would be a wrong-build-identity bug: a kernel serving another commit’s identity. The observed behaviour is the safe one – see the reuse limits below.

Measured reuse

Two worktrees at the same commit, at different host paths, building make kernel (init plus kernel, so both the x86_64-unknown-capos userspace target and the x86_64-unknown-none kernel target, including the kernel link). Each arm starts from an empty cache and clean target directories, builds the first worktree to populate, then builds the second.

This table is not reproduced: it was taken once per cell, on a host running concurrent builds, and the run recorded neither the cache-isolation procedure nor the stats capture. Treat the hit and miss counts as indicative and the wall-clock figures as unreliable; the reuse ratio is the durable part, and the artifact identity below is separately reproducible. Re-measuring needs two worktrees at one commit, a dedicated SCCACHE_DIR per arm so the live host cache is neither read nor disturbed, drained target directories, and CAPOS_CONTAINER_SCCACHE_STATS=1 for the container arm (the server dies with the container, so its counters can only be read from inside).

The headline host rate is high because this goal is dominated by registry dependencies, and those already share: cargo runs each unit’s rustc with the working directory set to that package’s own root, so a crate unpacked under CARGO_HOME – or a -Zbuild-std unit compiled from the rustup rust-src component – is worktree-independent already. The indicative run recorded 12 host misses but did not preserve a per-unit miss inventory. The earlier inventory here incorrectly counted capos-kernel, which sccache declines as non-cacheable rather than recording as a miss. Treat the 12 only as the unreproduced aggregate in the table, not as a durable first-party unit count.

So the aggregate rate is workload- and source-delta-dependent. A higher first-party fraction does not by itself predict a low host baseline, because the aggregate also contains registry and build-std units whose working directories are already stable. The canonical-path mechanism applies more narrowly: an unchanged cacheable first-party compilation whose host cwd differs between working trees can reuse the container entry because its cwd is /build in both. It does not imply a 0%-to-100% change over a goal’s aggregate cacheable set, especially when the second tree changes a shared first-party dependency. The measurements below do not retain the per-unit inventory needed to separate that subset from already-path-independent units.

The different-commit case, measured – and capos-kernel is not a miss at all

The equal-commit table above is the best case. The autopilot case is two worktrees on different branch tips, and it was left unmeasured. Measured 2026-07-26 14:34 UTC, the result corrects the explanation this document previously gave.

Procedure, against the live shared container cache (so the numbers are the ones a real worker sees, not an isolated ideal):

Create two detached worktrees at the same mainline commit. In the second, commit one added line in a kernel source file, so its tip, its sources and its CAPOS_BUILD_COMMIT all differ from the first’s. Then, in each worktree in turn:

CAPOS_CONTAINER_SCCACHE_STATS=1 make container-build CONTAINER_GOALS=kernel

The stats variable is required: the container gets its own network namespace, so its sccache server dies with the container and its counters can only be read from inside.

First-party cross-worktree reuse at a different commit is 100% of the cacheable set: 91 of 91, zero misses. Both worktrees are at different host paths, and the second differs from the first in a kernel source file, its commit and its build identity.

The zero-miss column is the part that needed explaining, because a changed kernel source cannot possibly hit. It does not: capos-kernel is never offered to the cache. Isolated directly – touch one kernel source in the second worktree and rebuild, so the kernel is the only unit that recompiles:

Compile requests            3
Compile requests executed   0
Cache hits                  0
Cache misses                0
Non-cacheable calls         1     reason: crate-type
Non-compilation calls       2

So the earlier claim here – that CAPOS_BUILD_COMMIT in the unit’s key makes capos-kernel miss across branch tips – described the wrong mechanism. sccache declines the kernel’s rustc invocation outright on crate-type, in every build, at every commit, in both host and container mode. The environment rule is real (see above) but has no bearing on the kernel, because the kernel’s compile never reaches key computation. The consequence is a ceiling, not a regression: the container path cannot accelerate the kernel compile-and-link, and neither can the host path. No kernel cache hit is lost by choosing container mode for an explicit build. Of the 123 requests in this goal, 29 are non-cacheable (crate-type 15, missing input 9, stdin 5) and 3 are non-compilation probes; the reuse claim covers the other 91.

Also directly observed: two container builds of the same tree at different commits produce a byte-identical capos-init and a different kernel. With the mechanism above, that difference is simply the kernel being compiled fresh both times, not a cache decision.

Compile-heavy goals across a source-changing commit

Recorded at 2026-07-29 01:26 UTC with sccache 0.16.0 and container image sha256:21bafca13a2e0b732618c831c8a473172e28358ad90b2b57b4af26557ec862cf. The comparison used two detached working trees at different host paths. The first was commit 8a97f0db3a8482f5c334b4096d944176a94ac3f8; the second started there, changed capos-abi/src/lib.rs from PROCESS_CAP_SLOT_LIMIT = 256 to the equivalent PROCESS_CAP_SLOT_LIMIT = 1 << 8, and committed the change. This gives the second tree a different tip and source while deliberately preserving runtime behavior. Because all three goals depend on capos-abi, the comparison is not the easier case where only unrelated metadata changed.

Each goal and arm had a separate initially empty cache. Before populating one, both working trees were drained. make clean currently omits shell/target and removes the tracked target/.gitkeep, so the exact drain also ran:

cargo clean --manifest-path shell/Cargo.toml
git restore target/.gitkeep

A recursive check then required every directory named target to contain no entry other than target/.gitkeep. Without the explicit shell clean, the container build is correctly refused because shell/target still holds host output.

For a host pair, $CACHE was unique to that goal and arm, and $PORT was a unique sccache server port so neither the ordinary host server nor its cache was read or stopped:

SCCACHE_DIR="$CACHE" SCCACHE_SERVER_PORT="$PORT" sccache --zero-stats
/usr/bin/time -f 'wall_seconds=%e' \
  env SCCACHE_DIR="$CACHE" SCCACHE_SERVER_PORT="$PORT" \
  SCCACHE_CACHE_SIZE=10G make -C "$BASE" "$GOAL"
SCCACHE_DIR="$CACHE" SCCACHE_SERVER_PORT="$PORT" sccache --show-stats

SCCACHE_DIR="$CACHE" SCCACHE_SERVER_PORT="$PORT" sccache --zero-stats
/usr/bin/time -f 'wall_seconds=%e' \
  env SCCACHE_DIR="$CACHE" SCCACHE_SERVER_PORT="$PORT" \
  SCCACHE_CACHE_SIZE=10G make -C "$CHANGED" "$GOAL"
SCCACHE_DIR="$CACHE" SCCACHE_SERVER_PORT="$PORT" sccache --show-stats
SCCACHE_DIR="$CACHE" SCCACHE_SERVER_PORT="$PORT" sccache --stop-server

For the matching container pair, the wrapper’s server is new for each container, so its counters are already per-build:

/usr/bin/time -f 'wall_seconds=%e' \
  env CAPOS_CONTAINER_SCCACHE_DIR="$CACHE" \
  CAPOS_CONTAINER_SCCACHE_STATS=1 \
  make -C "$BASE" container-build CONTAINER_GOALS="$GOAL"

/usr/bin/time -f 'wall_seconds=%e' \
  env CAPOS_CONTAINER_SCCACHE_DIR="$CACHE" \
  CAPOS_CONTAINER_SCCACHE_STATS=1 \
  make -C "$CHANGED" container-build CONTAINER_GOALS="$GOAL"

Populate is the base tree with the empty cache. Changed is the second tree against that populated cache. “Executed” is sccache’s Compile requests executed; every row also had two non-compilation calls, zero compilation failures, and zero cache errors.

The command-wall column records the requested end-to-end observation, but the arms are deliberately not compared. The host timer covers make "$GOAL" and excludes its later stats read. The container timer covers image inspection, reservation handling, host-side capnp-ensure cue-ensure limine-ensure, container start and stop, the inner build, and the in-container stats dump. The Cargo column comes from Cargo’s own Finished ... in ... line and is the common compile scope; Cargo rounds the two demos values shown in minutes to whole seconds.

The non-cacheable reason counts were identical between the two arms and both trees:

The container won aggregate cache reuse for every goal: 91 versus 82 hits for demos, 22 versus 21 for shell, and 20 versus 19 for capos-service-check. Across the three changed-tree builds that is 133 of 175 cacheable requests (76.0%) versus the host’s 122 of 175 (69.7%). Because no per-unit inventory was retained, this result does not say how many of the 11 additional hits were first-party units made reusable by /build, rather than a different aggregate effect.

There is no valid winner from the command-wall column because its arm scopes differ. Within Cargo’s common compile scope, the changed-tree observations were 62 versus 65 seconds for container versus host on demos, 36.42 versus 35.70 seconds on shell, and 34.94 versus 34.80 seconds on capos-service-check. Thus the container had the smaller Cargo time only for demos; the host had the smaller time for the other two. The latter gaps were 0.72 and 0.14 seconds, and all values are single samples from a live development host, so they are not evidence of a repeatable directional speedup. The result supports neither a general container speedup nor a worker-default switch.

Repeated worker-gate wall time and unit inventory

Recorded from 2026-07-29 04:53 UTC through 05:02 UTC at commit 061a98d6abee7c98623802781193ba614a2daff4. The host was Ubuntu 24.04.4 LTS on Linux 6.17.0, with eight Intel Xeon 2.80 GHz vCPUs and 7.7 GiB RAM. Docker client and server were 29.5.3, sccache was 0.16.0, and the container image was sha256:21bafca13a2e0b732618c831c8a473172e28358ad90b2b57b4af26557ec862cf.

The measured bundle was the same work in both arms and preserved runtime order:

make capos-rt-check capos-service-check shell

The timed command shapes were:

/usr/bin/time -f 'wall_seconds=%e' \
  env CAPOS_CONTAINER_SCCACHE_STATS=1 \
  make container-build \
    CONTAINER_GOALS='capos-rt-check capos-service-check shell'

/usr/bin/time -f 'wall_seconds=%e' \
  env SCCACHE_SERVER_PORT="$PORT" sh -c '
    sccache --zero-stats &&
    make capos-rt-check capos-service-check shell
    rc=$?
    sccache --show-stats
    sccache --stop-server >/dev/null
    exit "$rc"
  '

The container arm invoked those three goals in one container-build call. Its timer covered wrapper preflight, pinned-tool verification, container start, the inner Make invocation, in-container stats, and container stop. The host timer covered a fresh sccache server, stats reset, the same Make invocation, stats read, and server stop. Each arm first ran one prime against its existing mode-specific cache. Before every timed sample, all target trees and the mode marker were moved to a retained snapshot outside the worktree, leaving the tracked target/.gitkeep; no output from an earlier sample remained visible.

The three timed samples per arm therefore had cold target trees and warm caches. Every timed sample had the same sccache result: 190 requests, 121 executed, 121 hits, zero misses, 59 non-cacheable calls, and 10 non-compilation calls. The non-cacheable reasons were also identical: 26 crate-type, 18 missing input, and 15 -.

The container median was 2.65 seconds (9.2%) slower; its mean was 2.68 seconds (9.3%) slower. These are end-to-end command-wall measurements for the stated bundle, not Cargo-only timers. They are three samples on one live host, so they do not establish a portable performance constant. They do establish that the canonical container path does not win this fully warm gate bundle on the measured worker.

The primes were not used in that comparison because their pre-existing cache contents differed. They remain useful for the requested unit inventory:

The host prime ran sccache at debug level and retained every cache result. Its 19 misses were all first-party units:

The two arms executed the same 121 cacheable compilation requests. The host had no dependency or build-std miss that the container could turn into an additional dependency hit. The container’s three additional hits therefore came from first-party units, not dependencies. That identifies the unit class, not the cause: the mode-specific live caches were differently sized and populated, and the full host cache could have evicted first-party entries the container cache retained. The prime timings likewise must not be compared as a speed result.

Subsequent build measurements run their commands through tools/container-build/run-measurement.sh --run-id <id> -- <command>. The runner exposes its generated directory as CAPOS_MEASUREMENT_SCRATCH, places it under the user’s cache rather than the task-worktree parent, and removes it after a successful command. A failing command retains the directory with a MEASUREMENT_FAILED marker containing the run ID and exit status. Evidence worth retaining must be copied to its durable evidence location before the command returns success; the build trees themselves are disposable.

Complete fail-closed worker gate route

The route is a design prerequisite, not current behavior. The runtime gate list in docs/workflow/gates.toml is the source of truth and currently contains ten universal gates. Eight compile or run Rust tests:

fmt-check invokes Cargo formatting but compiles nothing, and webui-secret-gating-check is a source-shaped shell check. They may remain host-side because they write no target output. That exception must remain mechanically checked; a later compile added to either target makes it a container-routed gate.

QEMU remains host-side. A complete route separates every applicable proof into an immutable artifact build and a no-compile boot/assert phase:

1. Each QEMU target declares a named artifact target and an artifact contract: ISO, raw disk, or bundle; manifest; Cargo and demo feature sets; embedded payload goals; firmware assumptions; UART map; and immutable sidecars. Feature-dependent proofs get distinct named artifact targets. Raw Make variable assignments are not accepted as container goals.
2. container-build admits every declared artifact target and its complete prerequisite closure. This includes C and wasm payload builders when a selected proof needs them; the current blanket exclusion must be replaced by verified toolchain routes before those proofs are applicable.
3. The host proof takes a generalized CAPOS_PREBUILT_ARTIFACT=1 path. It verifies the artifact digest, producing mode and image, manifest, feature contract, UART map, and every member of a bundle before starting QEMU. Mutable disk proofs copy an immutable container-built base into per-run scratch rather than modifying the recorded artifact.
4. The prebuilt branch removes the artifact’s entire compile/package closure. Host-only drivers, browser helpers, manifest generation, C/wasm compilation, ISO mastering, and Cargo probes must either be prebuilt members of the verified bundle or be proven non-compiling harness work.
5. A generated checker derives the QEMU family from the Make rule database and fails unless every applicable target has exactly one declared artifact contract, a container-admitted artifact target, and a prebuilt branch. Under a container-mode marker, its dry run rejects any prebuilt route that reaches Cargo, rustc, clang, wasm builders, manifest generation, ISO/disk packaging, or the host build-mode guard. It also rejects stale declarations and artifact targets whose closure escapes the container allowlist.

run-smoke and CAPOS_PREBUILT_ISO=1 are the current prototype for the ISO/sidecar check, but they are not the complete route: no other QEMU target has the branch, the sidecar does not identify a general proof-feature contract, and disk or multi-artifact proofs do not fit an ISO-only variable.

The switch gate is one full rehearsal from a drained worktree: run all ten universal gates in their declared mode, build every selected task-specific artifact in the container, run each corresponding host prebuilt proof without a mode change, run container-build-mode-test plus the new route-coverage checker, and verify artifact provenance. Any unmapped compile or artifact contract fails the rehearsal. Only that integrated result can support changing the worker default; the design and wall-time measurement alone do not.

The kernel linker script is not an obstacle

Worth recording, because it is easy to assume otherwise. The kernel linker script is passed as a relative path from .cargo/config.toml (-C link-arg=-Tkernel/linker-x86_64.ld), resolved against the compile working directory, which is the workspace root. It does not travel through CARGO_ENCODED_RUSTFLAGS – the Makefile only ever clears that variable, for the standalone wasm payload builds. So the kernel-target cache key carries no absolute path from the link step, and the kernel ELF links inside the container unchanged.

Layout

• tools/container-build/Dockerfile – distribution packages only: gcc plus glibc headers, make, git, Python 3, and ripgrep. No Rust toolchain, no capOS tool cache. Pinned by base-image digest.
• tools/container-build/capos-container-build.sh – resolves the host paths and runs make <goal> inside the container. Its pre-flight refusals all come before any provisioning work: an unsupported goal, a missing container runtime, a missing rustc, a missing sccache, a missing toolchain, an image that is not present locally, a bind-mount path containing , or = (the --mount option string has no escaping for either), a container build of the same worktree already running, an active host build reservation, and a target directory in the other build mode. Ordering matters because the capnp build stamp depends on the Makefile, so any Makefile edit can otherwise make a from-source capnp build the price of being told sccache is missing. The image check matters because docker run treats a missing local image as a registry reference and reports a Docker Hub pull failure instead of “run make container-image first”. The already-running check matters because --rm only removes a container after it exits, so a SIGKILL to the CLI leaves one running; the container is named from the worktree so concurrent builds of different worktrees stay allowed, which is the normal parallel case.
• tools/container-build/capos-build-mode.sh – the build-mode guard and its self-test.
• tools/container-build/container-goal-route-test.sh – asserts that workflow-check is admitted with its image tools while the named unrouted universal gates and representative QEMU routes still fail closed.
• tools/container-build/run-measurement.sh – lifecycle wrapper for build-measurement scratch. It keeps scratch outside the worktrees root, removes it on success, and records the run identity when preserving a failed run. run-measurement-test.sh exercises both terminal paths.

The container is not a sandbox and make container-build is not a hardening measure. $HOME/.cargo is bind-mounted read-write of necessity, since cargo unpacks registry sources into it; that directory holds credentials.toml and a bin/ that is on the host PATH, so a build script running in the container can read a crates.io token or overwrite a host binary. This is not a regression – a host build already runs the same build scripts with the same user’s rights – and nothing in this path claims isolation. It is recorded so the container is not later read as a trust boundary. --network none narrows what a build can reach, but it is there to make an unexpected fetch fail loudly, not to contain a hostile build script.

Everything version-pinned stays on the host and is bind-mounted:

• Rust toolchain – rust-toolchain.toml is already the pin, and the measured cache key depends on the compiler’s content rather than its path. Baking a toolchain into the image would add a second pin that can drift from the first, and would make image reproducibility a precondition for cache-key reproducibility. Mounted read-only at /rust.
• capnp – capos-config’s build script executes it, so it must be reachable inside the container, and tools/capnp-build asserts its exact path under CAPOS_TOOLS_ROOT. The Makefile’s existing SHA-256-pinned provisioning is unchanged; the tool cache is mounted read-only at /capos-tools and capnp-ensure inside the container only re-verifies it.
• cue, limine – reachable inside the container through the same read-only CAPOS_TOOLS_ROOT mount, still SHA-256-pinned by the Makefile. Their provisioning stays on the host: the extraction recipes write into that mount, which fails inside the container as a read-only-filesystem error from within a flock, and pinned provisioning should not silently happen in a container anyway. So the host pre-flight runs cue-ensure and limine-ensure alongside capnp-ensure, and the container only consumes them.
• xorriso – installed in the image, because unlike cue and limine it has no host provisioning to mount. It is the one image package that masters shipped artifact bytes, so it is recorded in docs/trusted-build-inputs.md rather than treated like gcc (whose output never reaches the shipped ELFs).
• mdbook, typst – docs are not compilation. They stay on the host, outside this harness.

The git common directory is mounted at its own host path, read-only, because a worktree’s .git file names that absolute path and kernel/build.rs derives the build identity (CAPOS_BUILD_COMMIT, CAPOS_BUILD_TIMESTAMP) from git. Without it the kernel would stamp itself unknown. It is not a CARGO_* variable and is not a compile working directory, so it cannot perturb the key.

Cache directory

Container and host builds compute different keys for the same crate, so they share nothing. Pointing both at one directory would only have them compete for the same SCCACHE_CACHE_SIZE bound, so the container defaults to a separate $HOME/.cache/capos-container-sccache. Override with CAPOS_CONTAINER_SCCACHE_DIR, and collapse the two once the containerised path is the only one in use. The consequence is that the host’s cache bound and the container’s are independent, so the total budget is their sum; both are stated explicitly rather than left to sccache’s default.

Stating the container’s bound explicitly is not optional tidiness. HOME inside the container is a tmpfs, so no host ~/.config/sccache/config is readable and an operator’s configured size (or any other option set there) does not apply. Without an explicit SCCACHE_CACHE_SIZE the container silently falls back to the built-in default, and this paragraph would be reasoning about a bound the container never reads.

Deliberately bounded surface

• Running the QEMU gates inside the container is excluded. CONTAINER_GOALS admits compilation and packaging, but not the gates themselves: they would need qemu in the image plus --device /dev/kvm and a supplementary kvm group, because QEMU_KVM_FLAGS falls back to TCG when /dev/kvm is unavailable – correct, but far too slow for the smoke timeouts. Networking needs nothing extra: the harnesses use QEMU user networking, not a TAP device. The C and wasm payload goals are also out, because their recipes drive a C compiler and a second target triple that are not verified here. These are the current opt-in harness boundaries; the complete fail-closed worker gate route is authoritative for what must expand before a worker-default switch.
• Booting a container-built ISO from a host QEMU target – available, but only for run-smoke today. $(ISO) depends on the compile targets, so a plain host QEMU target over a container-built tree is the mixed-mode hazard above and the guard refuses it. CAPOS_PREBUILT_ISO=1 is the way through: it drops the ISO from the target’s prerequisites and takes the UART map from the ISO’s sidecar instead of recomputing it with a host mkmanifest, so the whole flagged path compiles nothing. Only run-smoke carries the flag today. Every other QEMU target intentionally reaches the ISO rule and is refused in a container-mode worktree – including test-default-boot, which the build and test guide names as the gate for the default system.cue boot and which the container can build (all produces capos-default.iso) but does not boot. The host-default worker decision above means this remains a deliberate boundary for the current opt-in harness. It is not sufficient for a worker-default switch; the route section above owns that expansion prerequisite.

Packaging-mode provenance

The ISO sidecar binds its producing mode, container image, skipped verification set, and xorriso identity to the final ISO digest. make build-provenance validates that binding and fails closed rather than falling back to host mode. For container packaging, the record therefore names the image-provided xorriso that mastered the ISO and records the ovmf-verify skip; it does not substitute the host’s tool identity.

One build mode per worktree, enforced

This is a correctness rule, not a performance note, and it is enforced rather than documented. Cargo’s freshness check for a workspace member does not notice the switch: it tracks source files by a path relative to the package, and the mount makes those relative paths and their mtimes identical. So a build in one mode on top of the other mode’s target/ can mark a unit fresh and relink the rlib the other mode produced, which carries that mode’s build root in its DWARF.

Both directions reproduce, and the host-over-container direction is the worse one: the host path is the default that make run, the QEMU smokes and make build-provenance all consume. Observed there: a plain make kernel over a clean container-built target directory exited 0 with no warning and produced a kernel in which one compilation unit (capos-abi) had DW_AT_comp_dir = /build, a directory that does not exist on the host. Nothing in the build output hinted at it. The artifact looked correct.

The extent is narrower than the mechanism suggests, which matters for triaging a real occurrence: in that reproduction 92 of 93 units recompiled and exactly one crate, capos-abi, survived stale.

The reason is not simply that CARGO_HOME differs between the modes. That explains why registry units rebuild, but not why a first-party unit resolving no registry path rebuilds – capos-lib recompiled too. The operative rule is cargo’s dependency-output comparison: once a dependency’s rlib is rewritten, every transitive dependent is dirty regardless of its own dep-info. That predicts the survivor set is exactly the dependency-free first-party leaves, which is a property of the workspace graph rather than a constant. capos-abi is currently the only workspace member with no dependencies and no build-dependencies, which is why it was the only survivor. If it ever gains a dependency, nothing survives a mode switch and the hazard disappears by accident; if a second dependency-free crate is added, two survive.

The mechanism is inferred from cargo’s fingerprint model rather than measured – what was measured is the outcome, 92 of 93 – but the graph premise is checkable against the workspace manifests, and the practical consequence stands either way: one mixed unit is enough to make the artifact wrong, and a mostly-reused build is not what an occurrence looks like.

tools/container-build/capos-build-mode.sh closes this. It records the mode that last wrote build output in .capos-build-mode (gitignored, repository root), reserves the worktree through .capos-build-mode-reservation, and refuses a build in the other mode while either that mode has an active reservation or any target directory still holds output. capos-container-build.sh acquires its explicit reservation before provisioning or docker run, reuses it in both make invocations by passing the wrapper-issued reservation ID as a non-exported make command-line variable, and releases it from its exit trap. make container-build-mode-test is the self-test.

Ordinary host reservations are bound to the outermost make process by PID, process start time, and boot identity. A later invocation reclaims one only after that exact process identity is gone, avoiding PID-reuse guesses. The same owner-generation rule applies to an explicitly named host-mode reservation; only container-mode reservations remain unreclaimable from host process state because their container may still be writing after the wrapper exits. A replacement reservation records the validated new owner generation rather than reusing fields read from the stale reservation. Reservation inspection, stale-owner reclamation, replacement, and owned release are serialized by the host’s flock, so a contender cannot replace the directory between another guard’s identity check and rename. A persistent rename failure refuses immediately instead of retrying while holding the lock. The in-container re-entry must present the wrapper-issued ID, validates it against the active reservation, and does not need flock inside the image.

The wrapper arms EXIT, INT, and TERM cleanup before attempting acquisition, so an ordinary Ctrl-C or termination releases a reservation even during host capnp provisioning or the remaining Docker preflight. Its explicit reservation is released only by its matching identity. If the wrapper receives SIGKILL, or the host loses power before its trap runs, the guard fails closed rather than assuming the named container stopped writing.

Two independent make invocations in one worktree are refused even when both request the same mode: host builds have distinct process-generation identities, and container-mode re-entry without the wrapper-issued reservation ID fails. The build-mode mix would be absent, but concurrent Cargo writes to one target tree are already outside this harness’s contract and can race independently of provenance. Parallel builds in different worktrees use different reservation and lock files and never serialize. After verifying that no container for the worktree remains, an operator can release such an orphan with the ID stored in .capos-build-mode-reservation/id:

tools/container-build/capos-build-mode.sh --release-reservation \
  "$(sed -n 1p .capos-build-mode-reservation/id)"

Its reach into the Makefile is the assumption everything else here rests on, so it is asserted rather than described. The guard is a prerequisite of capnp-ensure, which covers every target that compiles capOS code for a capOS target — but that is not the same as every cargo invocation, and the difference was larger than a reading of the Makefile suggested.

Writing the exception list by hand produced six targets. Computing the prerequisite closure from make’s own rule database found sixteen cargo-invoking rules that did not reach the guard. Three families mattered:

• model-scheduler-nohz-loom, a host-triple capos-config Loom test, and wasi-env-negative-check, a host-triple mkmanifest test, both write the root target/ — the directory a container build fills. An unguarded write there can rewrite target/.rustc_info.json with the host sysroot inside a directory holding container output, which is the evidence the unmarked-worktree inference below depends on.
• Eleven demos/wasi-*/target/wasm32-wasip1/release/*.wasm artifact rules, reachable through their wasi-*-build parents. These compile wasm32-wasip1, which disproves the reasoning offered for the hand list — that every bypass was host-triple and therefore harmless. Their directories are already in the guard’s discovery set, so the guard could refuse a build over output nothing had declared.
• capos-tls-test (deliberately capnp-free, since it builds only -p capos-tls), webui-login-peer-logic-test, userspace-broker-policy-test, task-coordinator-logic-test, fuzz-build and fuzz-smoke. The first two are make check members, so without the prerequisite make check in a container-mode worktree compiled into demos/target/ before failing at capos-rt-check.

All of them now reach the guard; the eleven artifact rules take it as an order-only prerequisite so a .PHONY dependency does not make every .wasm perpetually out of date.

Only three cargo-invoking rules are deliberately unguarded, and tools/container-build/check-cargo-guard-coverage.py asserts set equality against exactly that allowlist: fmt and fmt-check compile nothing, and clean is the remedy for a refusal, so guarding it would strand a worktree that could then neither build in the recorded mode nor drain out of it. A newly unguarded rule fails the gate, and so does an allowlisted rule that becomes guarded. Widening the allowlist is a reviewed diff to that file.

Two checks run, deliberately: the closure computation above is complete but parses make -p output, so it also asserts a floor on the number of cargo rules it found — a degraded parse would otherwise report everything as guarded. Beside it, capos-build-mode.sh --check-guarded-targets asks make --dry-run what a sample of targets would actually run. One is complete, the other is authoritative.

Three properties are worth stating because they are what make it fail closed:

• Target directories are discovered, not listed. Any directory named target at any depth that holds build output counts. The walk prunes .git and each target directory after finding it, so its cost does not scale with the contents of build output. This is the property that makes the guard cover the whole goal set rather than the default goal: nine standalone crate manifests build through .cargo/config.toml aliases – init/, demos/, shell/, capos-rt/, capos-service/, capos/, libcapos/, libcapos-posix/, capos-wasm/ – each with its own <crate>/target, and the demos/wasi-* payloads are separate workspaces one level deeper. CONTAINER_GOALS is overridable, so shell, demos, capos-rt-check and friends are ordinary usage, and by the reuse argument above they are the goals with the highest first-party fraction – the ones this path pays best on. A target plus init/target list would leave every one of them exposed. make container-build-mode-test asserts the whole set and the existing depth-four tools/remote-session-client/src-tauri/target: restricting discovery to the former depth bound or to the first two directories fails the gate.
• An unmarked worktree is classified from evidence. Cargo records the rustc sysroot in <target-dir>/.rustc_info.json, and the container’s sysroot is the fixed mount path /rust, so an unmarked directory still declares which mode filled it. Without this, a worktree holding container output produced before the guard existed would be misread as host and permitted – exactly the mix the guard is for. Every occupied directory is classified rather than the first one that answers, so a tree that is already mixed is reported as such and refuses both modes instead of being adopted as whichever mode was found first.
• It refuses rather than wiping. Discarding build output is the operator’s call; the refusal names the directories and prints the command.

The reservation lock is acquired before the mode marker is checked or written. This closes both empty-tree startup races: a host build cannot enter while the container wrapper is starting Docker, and a container wrapper cannot enter after a host guard passes but before that host build writes its first target artifact. The self-test exercises both directions without pre-populating a target directory, refuses a second independent same-mode owner, verifies that the OS lock remains held while stale-owner reclaim is paused, and races opposite-mode contenders over a stale host reservation to assert that exactly one proceeds. It also proves that a persistent reclaim error returns after one attempt, a stale explicit host reservation is reclaimed by owner generation, the replacement retains the new owner’s generation, an orphaned explicit container reservation is never reclaimed, and an invalid owner reports its actual preflight failure without waiting for a partial claim.

The cost is that the modes are exclusive for the whole worktree, not per goal. After a full host ISO build, switching to container builds means discarding every populated target directory, and vice versa. That is the same “pick one mode per worktree” rule as before, now with a build-time signal instead of a convention. Note target/.gitkeep is tracked, so rm -rf target needs it restored afterwards.

Consequently a container-mode worktree cannot run the host gates (make check and everything under it reaches capnp-ensure), and a host-mode worktree cannot run make container-build without draining first. Drain a worktree – empty target directories and no marker – to leave it free for either mode. A completed host make may leave its process-bound reservation record behind; the next guard reclaims it after verifying that owner generation is no longer live.

Artifacts

Two worktrees at different host paths, each built clean in a container, produce a byte-identical kernel ELF (same SHA-256) and a byte-identical capos-init ELF. The kernel contains no host path at all: every first-party compilation unit’s DW_AT_comp_dir is /build, and so are the embedded source-path strings. A host build of the same tree carries the worktree path in both.

Measured on the 232e1791 container-built kernel:

K=target/x86_64-unknown-none/debug/capos-kernel
sha256sum "$K" init/target/x86_64-unknown-capos/release/capos-init
readelf --debug-dump=info "$K" | grep -c 'DW_AT_comp_dir.*: /build$'   # 49
readelf --debug-dump=info "$K" | grep DW_AT_comp_dir | sed 's/.*: //' \
    | sort -u | wc -l                                                  # 22
strings -a "$K" | grep -oE '^/build/[A-Za-z0-9._/-]+' | sort -u         # 5 paths
strings -a "$K" | grep -c '/home/'                                     # 0

The 49 are the first-party units; the 22 distinct comp_dir values are those plus the /cargo/registry/... and /rustc/<hash> roots, which are worktree-independent in either mode. The five embedded source-path strings are capos-rt/src/{panic,lib,entry}.rs, capos-config/src/validation.rs, and the generated capos_capnp.rs under the build-script OUT_DIR. All five counts are commit-specific – the OUT_DIR one embeds a build hash – so re-measure rather than quoting them.

That is what makes serving one worktree’s cached object to another sound rather than a fudge: the compiler really did run in /build both times. Key normalisation alone would leave the artifact carrying whichever build root compiled it first, which is unacceptable for a kernel that prints paths.

Remaining caveats

• DW_AT_comp_dir becomes /build. Host-side debuggers resolving source from container-built artifacts need a path mapping (set substitute-path /build <worktree> in gdb).
• Panic locations change for the standalone crates. Workspace members are unaffected: cargo passes relative source paths for them, so their file!() strings are identical either way. init/ and its outside-the-package path dependencies are not, because cargo passes those as absolute paths. The container-built kernel, which embeds capos-init via include_bytes!, carries /build/capos-rt/src/panic.rs, /build/capos-rt/src/lib.rs, /build/capos-rt/src/entry.rs and /build/capos-config/src/validation.rs where a host build of the same tree carries the worktree path. So init-side panic output names a directory that does not exist on the host.
• The total sccache budget doubles. The container defaults to its own cache directory (see above), and neither cache is a subset of the other, so the bound is the host’s plus the container’s – currently 10G each. Both are set explicitly (SCCACHE_CACHE_SIZE, and CAPOS_CONTAINER_SCCACHE_CACHE_SIZE for the container) rather than inherited from sccache’s default. A server that is already running keeps the size it started with.
• Build mode is recorded in provenance, and the two are not comparable. make build-provenance reports the artifact-bound Build mode, container image, skipped verification set, and producing xorriso identity from the ISO sidecar. A host artifact and a container artifact differ by construction, so a comparison across modes flags the coincident mode change while stating that the record pair alone cannot prove it caused artifact-hash drift; see the retention and comparison policy in docs/trusted-build-inputs.md.
• The image is a build input. It is pinned by digest and inventoried in docs/trusted-build-inputs.md, which also carries the refresh procedure.