Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Self-Hosted Task Backend

Detailed decomposition for hosting the multi-agent development task backend on capOS itself. Today that backend is loopyard, a companion host-side project (PostgreSQL-backed; not part of this repository): it computes runnable task sets from typed dependencies, validates status transitions, checks conflict domains, and issues lock leases with generation fencing, all enforced inside the database so no client can corrupt task state, with CLI/MCP/REST as thin transports over one API surface. The long-term intent — recorded in Stateful Task and Job Graphs and capOS-Hosted Agent Swarms — is for capOS to serve that coordination role in-system through typed capability services instead of an external SQL server.

Visible Outcome

A capOS instance serves the task backend for the multi-agent development workflow that builds capOS: host-side workers resolve runnable tasks, acquire fenced lock leases, report status transitions, and receive conflict reports from a capOS-hosted coordinator service, speaking the same task-source and lock wire contracts that loopyard serves today. loopyard’s schema and invariants are the reference semantics the capOS services must reproduce, and task state must be exportable/importable in both directions for migration.

Gap Inventory

What exists and what is missing, largest gap first:

  1. Transactional structured record store. The coordinator needs record-level atomic multi-record commits, secondary lookups, and constraint-style invariant enforcement. Current durable storage is below that level: the userspace Store/Namespace service persists a whole-state snapshot (CAPOSUS1), the kernel persistent_store fixture is a disk-backed CAPOSST1 Store, and writable_fs is a single-writer CAPOSWF1 filesystem. Crash-safe A/B superblock commits and torn-grow mount recovery are landed building blocks one level down. The write-ahead-log record store itself is now landed: the capos-lib CAPOSRS1 transactional WAL (capos_lib::recordstore — checksum-chained torn-tail discard, epoch fence, atomic multi-record commit frames, secondary index, host-tested and fuzzed) is wired to the real BlockDevice cap by the demos/record-store-blockdevice service and proven in QEMU for reboot durability and bounded forced-poweroff crash recovery (make test-record-store). The coordinator’s persistence is now switched onto this store (2026-07-15 17:00 UTC): the demos/task-coordinator-persist-proof process drives the real task-coordinator-logic coordinator write-through over a granted BlockDevice cap — every accepted mutation commits the affected task’s full snapshot (record + status + fenced lease) as one atomic WAL frame keyed by task:<key> and secondary-indexed by status:<label>, and on boot rebuilds the live coordinator from the replayed log. make test-task-coordinator-persist proves, across three boots of one disk image, reboot reload of the runnable set / statuses / lease generations plus a bounded forced-poweroff torn-write recovery in which the interrupted task is atomically absent. The snapshot codec (encode_record/decode_record with a fail-closed value-level decode) and boot-rebuild seam (restore_task) are host-tested in task-coordinator-logic. Log compaction is now landed (2026-07-15 17:52 UTC): the compaction-capable CAPOSRS1 format (RecordStore::format_compacting / compact) folds the append-only log down to the live record set into a fresh generation over an A/B superblock plus two log segments — writing the folded set into the inactive segment under epoch + 1 and flipping the generation-fenced superblock last, so an interrupted compaction recovers to either the pre- or post-compaction consistent state — host-tested and fuzzed in capos_lib::recordstore and proven in QEMU (demos/record-store-compaction-proof, make test-record-store-compaction) for reboot log-space reclamation plus a bounded forced-poweroff in the compaction flip window. Still missing at this level: wiring the same durable seam into the Endpoint-serving task-coordinator-service (an optional-BlockDevice-cap durable path over its serve loop).

    A bounded QEMU dogfooding path is now landed without claiming to close that production-shaped gap. The Endpoint-serving service can optionally consume a single-writer, spawn-granted writable 9p Directory; it persists one canonical current/temp/backup record per task and recovers before serving. make test-task-coordinator-9p proves two-boot reload, host inspection, and a post-restart mutation. The path accepts at most 20 tasks and 60 directory entries, fails closed on malformed, stray, duplicate, oversized, or ambiguous state, preserves fencing generations, and invalidates active leases at restart because the guest monotonic clock is boot-relative. Its v2 records carry the lease actor; the decoder remains compatible with v1 records and drops any v1 lease because that layout cannot attribute it, while preserving the task generation. The identity-free persistence-work readback attributes logical read-requested/read-returned and write-submitted bytes, bounded chunk high-waters, and transaction byte high-waters without changing persistence behavior; Resource Governance owns that descriptor. Its host-filesystem plus QEMU Tfsync proof remains bounded dogfooding rather than production storage authority, and supplies no per-actor charge, admission budget, protected recovery reserve, physical-I/O attribution, or allocator accounting.

    That remaining step is blocked on a spawn-grant authority gap, not on the durable seam. Under the qemu feature a BlockDevice is grantable only to the init process, through the bootstrap CapSet (cap::BootCapFactory’s build, the #[cfg(feature = "qemu")] KernelCapSource::BlockDevice arm over the boot virtio-blk device). The spawn path cap::process_spawner::build_child_caps carries a KernelCapSource::BlockDevice arm only under #[cfg(not(feature = "qemu"))] (the brokered-NVMe arm), and no KernelCapSource::BlockDeviceTarget arm at all, so a spawned service that requests either source falls through to Err("spawn grant unsupported kernel source"). task-coordinator-service is a spawned service in system-task-coordinator.cue, and make run-task-coordinator is a qemu build, so an optional BlockDevice grant cannot reach it on that proof path. This is why every disk-holding QEMU proof (task-coordinator-persist-proof, record-store-blockdevice, storage-persist-service, multi-virtio-blk-smoke) makes the disk holder the init process with services: []. The manifest offers no way around it: CapSource is Kernel or Service only, so init cannot delegate a cap it already holds to a service in the graph.

    Three ways forward, in preference order:

    • Wire KernelCapSource::BlockDeviceTarget into build_child_caps under qemu. The bootstrap arm already scopes this source to a manifest-declared non-boot disk by PCI identity and fails closed on boot-device identity, absent selector, and no match (virtio::block_device_target_for_pci_selector), so the authority being extended to spawned services is a PCI-scoped non-boot disk rather than the boot medium. This keeps system-task-coordinator.cue in its current shape (init spawns coordinator + client, exitWhenServiceExits) and needs a second QEMU disk plus kernelParams.blockDeviceTarget.pci. It is a kernel authority change and needs a decision before implementation.
    • Make the coordinator the init process on a durable proof manifest. Mirrors storage-persist-service: the coordinator holds the disk, spawns the client itself, and the client signals shutdown out-of-band because exitWhenServiceExits is unavailable with services: []. No kernel change, but it reshapes the existing acceptance proof and pulls demos/task-coordinator-client-smoke/ into the slice.
    • Prove the durable path on the non-qemu brokered-NVMe arm, where build_child_caps already grants a spawned service a BlockDevice. This is the production-shaped path but requires a verified NVMe controller and an ordered device_mmio grant, i.e. a run-cloud-provider-*-class vehicle rather than run-task-coordinator.

  2. The coordinator service itself. A userspace service owning task records, statuses-as-data with validated transitions, dependency edges with cycle rejection, runnable-set computation, conflict-domain checks, and lock leases with expiry and generation fencing — the Stateful Task and Job Graphs Stage E operator task surface and Stage G agent workflows, with invariants enforced in the service the way loopyard enforces them in PL/pgSQL. The phase-1 in-memory local proof is landed: the demos/task-coordinator-service process owns the records and enforces those invariants server-side over a demo-local Cap’n Proto Endpoint protocol (demos/task-coordinator-proto), with the pure rules host-tested in demos/task-coordinator-logic (make run-task-coordinator, make task-coordinator-logic-test). The phase-2 HTTP/JSON API surface is also landed (2026-07-14 03:32 UTC): demos/task-coordinator-api-service serves the coordinator’s task-source and lock contracts (show/list/runnable-set, create/transition, lease acquire/renew/release with generation fencing) as JSON over the Phase C userspace network-stack listener, with the bounded HTTP/JSON rules host-tested in demos/task-coordinator-api-logic and a host-side harness driving the contracts through a hostfwd TCP port (make test-task-coordinator-api). Lock-response holder-metadata parity with the loopyard lock adapter is landed (2026-07-14 08:16 UTC): contention refusals echo the live blocking lease as a metadata blob in loopyard’s contention shape — see the mapping below. Still missing at this level: durable persistence, authentication, and any production backend authority (the demos are bounded QEMU proofs).

    Core-table whole-scan attribution is landed as a bounded read-side proof. The coordinator brackets conflict-domain acquisition, dependency/cycle validation, runnable-set construction, and state-transfer size projection and validation (mutation admission plus export/import), listTasks projection, and lockList projection as six exclusive dispatch classes, and the occupancy sample’s own table traversal as a seventh. Its identity-free occupancy record publishes saturating per-class visits and their sum, per-class scanned-entry high-waters, the largest observed scan class over all seven, and the largest over the six dispatch classes alone. The traversal outscans every dispatch class once any task exists, so it holds the overall position by construction and the dispatch position is the one that still names driven serve-path work. The traversal class is counted on every sample, including the samples whose record the due-check withholds, and is excluded from the due-check so the observer cannot trigger its own emission. It also publishes Rust-owned stored length and allocation capacity for both map keys, dependency and conflict-domain vector backing and string buffers, and lease worker strings, plus the capacity high-water and a capacity-below-length mismatch counter. The fixed-width lease actor is inline rather than a heap string. Ordered-map nodes, allocator-internal overhead, and physical memory remain outside the byte totals. Heap occupancy, heap high-water, and scan-entry high-water changes emit immediately; scan-visit-only and mismatch-only changes retain exact-power-of-two damping. The record carries no task, actor, worker, lease-holder, generation, expiry, or capability material. The publisher takes no new clock sample and changes no task, lease, admission, persistence, or ingress authority. The authoritative coordinator descriptor inventory now also covers the bounded 9p persistence-work readback and its remaining production gaps; see Resource Governance.

    Lock wire-contract mapping, verified field-by-field against the installed loopyard lock adapter (loopyard vibe lock, src/loopyard/vibe.py _op_acquire/_op_update/_op_release), operation by operation:

    • acquire, success — loopyard: {"acquired": true, "metadata": {<client-stored blob> + "task_id", "run_id", "path"}}; adapter: {"acquired": true, "metadata": {"task_id", "run_id", "generation", "expires_at_ms"}}. loopyard round-trips a client-supplied blob (lease_seconds, heartbeat_at, fencing_token, workspace, …); the adapter synthesizes its blob from coordinator lease state — generation is the fencing token and expires_at_ms the staleness authority. path (a lock-root file path loopyard defaults in) has no capOS equivalent and is absent.

    • acquire, contention — loopyard: {"acquired": false, "metadata": {<holder's stored blob>, "run_id": <holder run>, "task_id", "path"}}; adapter: {"acquired": false, "error": "lease-held"|"conflict-domain", "metadata": {"task_id", "run_id": <holding worker>, "generation": <holder generation>, "expires_at_ms": <holder expiry>[, "held_task"]}}. The error label is an additive diagnostic (loopyard carries none); held_task is a capOS extension naming the blocking task on conflict-domain refusals (loopyard’s lock layer has no cross-task conflicts).

    • renew/update, success — loopyard: {"updated": true, "metadata": {...}}; adapter: same shape with the synthesized blob.

    • renew/update, refusal — loopyard: bare {"updated": false} (its client raises LockBackendError on it, no holder echo); adapter: {"updated": false, "error": "stale-generation"} — additive label, no metadata, matching loopyard’s metadata-free refusal.

    • release — loopyard: {"released": true|false}; adapter: {"released": true} / {"released": false, "error": "stale-generation"}.

    • Host worker wiring (landed in 05324347): tools/capos-task-backend-adapter.py exposes the installed vibe-loop task-source and fenced-lock command contracts over the bounded HTTP/JSON API. The coordinator remains authoritative for task state and fencing generations; an owner-only local ledger maps opaque vibe-loop fencing tokens and recognized workspace metadata onto those generations. The adapter rejects non-loopback service URLs, bounds all input and response data, maps HTTP failures to stable command exits, and hashes oversized run identifiers into coordinator-sized worker names without losing their local identity. make task-backend-vibe-adapter-test covers the command contracts, including an installed vibe-loop process. make test-task-backend-vibe-adapter-9p drives the same adapter through two QEMU boots and proves stale-token rejection, generation advancement, and workspace-metadata continuity. Arbitrary extension metadata, remote authentication, public ingress, and production backend authority remain outside this bounded path.

      The coordinator-side half of the status/list gap is now landed (2026-07-16 17:02 UTC): the Endpoint protocol carries read-only lockStatus (9) and lockList (10) returning a LockInfo (key/worker/generation/expiresAtMs) for leases live at the served instant, over Coordinator::lock_view/lock_views. These were the missing primitive: the HTTP adapter is a stateless transport holding no lease state, and TaskInfo.generation is the task’s fencing token, which survives lease expiry — so showTask/listTasks could not answer “is a lock held, by whom, until when”. Query and refusal liveness share one Coordinator::live_lease predicate so they cannot drift. Proven by make run-task-coordinator (live holder, conflict-blocked-but-unlocked, unknown key, expiry exclusion before reclaim, post-reclaim holder) and make task-coordinator-logic-test. The HTTP adapter routes on top are now landed (loopyard-lock-status-list-adapter-routes): GET /v1/locks/list returns {"locks":[...]} (one lock-metadata blob per live lease) and GET /v1/locks/status/<key> returns {"locked":true, "metadata":{...}} for a live holder or a bare {"locked":false} for a released or never-locked key (not a 404), matching loopyard’s GET /locks / GET /locks/{name} read contract. Both share lock_metadata_json, extracted from the existing lock response builder, so no second response shape was added. Proven end-to-end by make test-task-coordinator-api and the task-coordinator-api-logic host tests. The same Endpoint service now has the optional bounded 9p persistence path described above; it remains in-memory when no state Directory cap is granted. Still missing at this level: authentication and production backend authority.

  3. Public ingress and TLS. Host-side workers need to reach the backend. Private GCE self-hosted Web UI serving is proven; public exposure and TLS remain gated by the explicit on-hold cloud-gce-public-self-hosted-webui-ingress-tls task. TLS building blocks (capos-tls, the ACME http-01 solver) exist.

  4. A reusable multi-client API service layer. The remote-session Web UI already runs a persistent accept/recv/send/close loop with per-connection deadlines and slow-client bounds over the Phase C userspace stack, but as a demo-grade single flow. The phase-2 API adapter (demos/task-coordinator-api-service) reuses that serving shape and is the first in-tree capos-service lifecycle consumer. Its bounded multi-client extension landed 2026-07-14 13:00 UTC: each accepted Phase C connection receives a distinct socket service-object capability and receiver badge over the shared socket-service endpoint, while the network-stack loop multiplexes per-slot recv/send/close calls from smoltcp readiness. The adapter maps those readiness observations through the capos-rt poll/select bridge into a two-entry connection table with per-request deadlines and fail-closed HTTP 503 refusal on exhaustion. Each submitted ring call retains its CallId through completion; if its deadline expires while the target is still pending, the adapter terminates instead of resuming with an orphaned call. Service-object helper faults use bounded, ordered object release, helper termination, and process reaping. The network stack’s accepted-holder lease is 60 seconds, covering the adapter’s five-second coordinator budget while retaining the two-second per-socket-call slow client bound. The QEMU harness observes client A’s new admission after a log-position snapshot before it opens client B, then proves B completes while A holds a partial request; it also proves that a third client is refused while both slots are occupied and that repeated connection churn does not exhaust the process capability table. The reusable HTTP/JSON service layer landed 2026-07-27 11:43 UTC (task record): capos-http-json-service now owns configurable bounded request framing, JSON/octet response construction, route/handler composition, connection admission, absolute read/send deadlines, readiness-driven multiplexing, and retained submitted-call ownership. The coordinator adapter supplies the /v1/tasks/*, /v1/locks/*, /v1/state/export, and /v1/state/import routes plus its coordinator-call budget; its JSON result/error shapes and all existing wire limits remain coordinator-owned. The layer receives no coordinator, storage, or session capability. The other remote-session Web UI modes retain their serial serving path. Authentication and production backend authority remain absent.

  5. Remote actor authentication. Endpoint-level lease attribution landed 2026-07-27 09:41 UTC (task record): every acquired coordinator lease records the opaque, service-scoped caller session reference already supplied by the Session Context invocation boundary. The service checks the Endpoint ABI’s live-session flag on every dispatch. renewLease and releaseLease require that same live caller session and return callerSessionMismatch separately from staleGeneration; read-only lockStatus/lockList surface the recorded actor through LockInfo, and the HTTP/JSON adapter projects that same service-scoped correlation as one fixed-width lowercase-hex actor field on its shared lock metadata shape. Only live, actor-bound leases reach those routes: persistent restart invalidates leases, state transfer omits them, and legacy records do not reconstruct them. The adapter rejects malformed internal lock metadata instead of emitting a plausible all-zero actor identifier. make task-coordinator-logic-test covers same-session success, cross-session refusal and refusal ordering, plus expiry/reclaim; make run-task-coordinator drives two broker-selected spawned callers in QEMU, while make test-task-coordinator-api distinguishes a direct Endpoint lease from an adapter-acquired lease on both HTTP lock read routes. Bounded HTTP API-token admission is now landed (2026-07-29 14:20 UTC) (task record). initConfig.taskCoordinatorApiTokens optionally carries at most four SHA-256 token hashes paired with fixed-width actor labels. An absent set preserves the unauthenticated development behavior. A configured set protects every /v1/tasks/*, /v1/locks/*, /v1/state/export, and /v1/state/import route: missing, malformed, unknown, duplicate, or oversized authorization fails as the same typed 401 before routing or coordinator submission. Tokens are capped at 64 bytes, authorization field-values at 71 bytes, and hash comparison uses a constant-time primitive across the complete bounded set. Successful API lease acquisition records a bounded generation-keyed actor projection, and the existing lock_metadata_json shape uses it on both lock read routes. The adapter admits authenticated renew/release calls only when the presented token actor owns that task’s API projection, refusing cross-token mutation before any coordinator call. Direct Endpoint leases retain their session actor and remain read-visible, but authenticated HTTP clients cannot renew or release them. The projection ledger has one slot per coordinator task (256), rejects an impossible excess before lease acquisition rather than terminating the service, and is cleared on import together with coordinator lease invalidation.

    Optional credential-surface admission landed 2026-07-30 04:02 UTC (task record). The static hash set remains the development default, including the prior unauthenticated behavior when it is absent. A declared taskCoordinatorApiCredentialPath instead requires the adapter’s CredentialStore, one fixed UserSession, and a manifest-declared SHA-256 prefilter. The prefilter rejects arbitrary bearer values before they can request Argon2 work, CredentialStore verifies the admitted bearer, and the fixed session must answer info() as live. Its opaque session/principal identifiers derive the service-scoped actor used by the existing generation-keyed projection. The bearer carries no session selector and the credential verifier returns no principal identity: deployment configuration, not client input, binds this one credential path to this one session. Missing session/hash grants, a denied credential, or a stale session returns typed 401; verifier unavailability and transport failure return typed 503, while verifier overload returns 503 with bounded Retry-After and millisecond guidance. Declaration never falls back to static hashes. The local QEMU gate runs the static two-token proof and the fixed-session credential proof separately, preserving cross-token mutation coverage while proving the credential admission and direct-Endpoint lease boundary.

    The host command adapter now carries that local credential contract (task record). CAPOS_TASK_BACKEND_API_TOKEN is an environment-only input; when present, the adapter sends it as a bearer on every coordinator request without placing it in argv, command output, or the owner-only metadata ledger. It refuses redirects so the credential cannot be forwarded beyond the exact configured loopback endpoint. It enforces the coordinator’s printable-ASCII grammar and 64-byte token / 71-byte authorization limits before opening a request, while an absent or empty value preserves the prior unauthenticated request shape. Typed 401 refusals use stable non-retryable exit 5. Typed 503 refusals retain transient exit 1 and expose only bounded numeric Retry-After and retryAfterMs guidance; malformed or oversized guidance is ignored and the adapter does not retry internally. make task-backend-vibe-adapter-test covers authenticated success, redirect refusal, missing and unknown credentials, reflected-label redaction, local oversized refusal, hostile retry guidance, the installed runtime-owned default lifecycle, and output / ledger redaction. The lifecycle proof no longer pins worker-owned compatibility mode: generation-bound complete and reset HTTP commands let the runtime close active -> review -> done after integration or settle active -> ready after failure. The adapter admits either command only when its owner-only ledger maps the runtime’s environment-only opaque fence and run identity to the submitted coordinator worker/generation. Malformed, oversized, missing, stale, or cross-run hook claims fail before mutation. The adapter advertises the runtime’s fenced reset capability. Its post-release fallback carries no fence, requires the task lease to be absent, and uses the coordinator’s ordinary validated active -> ready edge; this lets a failed fenced settlement recover without treating an ambient or stale token as authority. The same private ledger provides the separate main-integration mutex required by runtime-owned integration. Because the runtime does not heartbeat that mutex, the adapter deliberately ignores command-lock lease durations: the record remains held until explicit release or the runtime’s same-owner process-liveness recovery, so elapsed wall-clock time cannot grant overlapping integration authority. The mutex stores no task state and never enters the coordinator’s task or conflict-domain model. Every validated coordinator status edge – ready -> active, active -> ready, active -> review, review -> active, and review -> done – is bound to the acquiring Endpoint caller session while the task has a live lease. The authenticated HTTP path additionally requires the lease’s generation-keyed bearer/session actor projection before it submits any transition. In the deliberately unauthenticated posture, the API service’s shared Endpoint caller session remains the authority check; therefore its own HTTP-acquired leases can use the hooks while a lease acquired by another direct Endpoint session remains read-visible but not HTTP-transitionable. An expired, released, or absent lease reserves no transition authority. The active -> ready edge still clears any attributed lease and advances the issued-generation high-water mark before the task can re-enter the dependency-derived runnable set, so pre-reset renew and release tokens are stale. No validated status edge remains lease-unbound; non-transition mutations such as task creation and dependency changes remain outside this lease-holder rule. Transitions retain the same durable write-through, conflict domains, dependencies, and runnable-set derivation, with no export/import format or status change. make test-task-coordinator-api executes both lifecycle hooks in capOS under the disabled, static-token, and credential-backed authentication postures. make test-task-backend-vibe-adapter-9p selects the static-token manifest posture and drives the existing two-boot fencing and workspace-metadata sequence through authenticated adapter calls.

    This remains bounded local proof-grade credential admission, not production authentication. The fixed session and public capos credential/hash are local proof fixtures. The console verifier is not an API credential registry and cannot attest a credential-to-principal or credential-to-session relationship; supporting multiple accounts or actors requires a different identity-bearing verifier/lookup contract. There is no credential issuance, rotation, revocation, persistent session lookup, TLS/public ingress, or production account-policy integration. It grants no public-ingress authority and does not close this gap. The coordinator remains authoritative for task state and fencing generations; the adapter receives no coordinator or storage authority beyond its existing narrow client endpoint. The Web UI login remains demo-grade.

  6. Client transport. The bounded REST/JSON command adapter for vibe-loop’s task-source and fenced-lock contracts is landed in 05324347. A reusable typed client remains future work through the capos SDK remote transport (capOS SDK And Dual Transport); the transitional host-backend remote transport (slice 4a there) can carry typed clients before the full capnp-rpc upgrade.

  7. Time quality and data portability. The coordinator reads a granted WallClock for its boot-relative monotonic lease deadline and publishes a boot-unique lock epoch with bounded provenance. This gap is the authoritative current-state record for coordinator lease restoration. The done task-coordinator-seeded-wallclock-lease-expiry-domain title overstates what finally landed: that task first added seeded expiry persistence, but its final remediation commit ed22ca69 removed it and retained fail-closed boot-relative invalidation. The earlier task-coordinator-lock-clock-provenance-read-surface delivered only the boot epoch and provenance fields on lock reads; it did not make deadlines comparable across boots.

    kernelParams.seedUtcSeconds selects an operator-declared UTC base at the boot monotonic origin for manifests that set it. The coordinator persistence manifest does not set it, and setting it would not establish elapsed time: Phase 1.x UTC is that fixed base plus monotonic-since-boot and restarts from the base on every reboot. Seeded and fallback readings both retain untrusted provenance. Durable coordinator records therefore keep the boot-relative lease shape: restore invalidates every lease, republishes the cleared record, and preserves its issued-generation high-water fence. make test-task-coordinator-9p proves that fail-closed invalidation and durable clearing across two boots of one host share. This is bounded local QEMU evidence, not trusted time or production clock authority; a progressing cross-reboot source and explicit domain contract remain Phase 2 ClockDiscipline/NTP work. The bounded coordinator-local export/import contract landed 2026-07-27 05:08 UTC and its fencing and admission invariants were tightened 2026-07-27 05:33 UTC (task record): TaskCoordinator.exportState emits one deterministic snapshot, bounded by MAX_TASKS and a 60 KiB payload below the Endpoint ceiling, containing statuses, dependency edges, conflict domains, and fencing generations in their observable stored order. Task creation, dependency addition, status transition, and durable restore refuse any mutation that would cross that byte bound, so every admitted live state remains exportable. importState treats the strict versioned field format as untrusted, builds a fresh candidate through the same create_task, add_dependency, and transition rules as live requests, and swaps it in only after complete validation. Unknown/malformed fields, invalid statuses, dangling references, cycles, duplicate keys, limit violations, invalid fields, and exhausted generations have typed refusals. Leases are absent from the format. The coordinator retains a MAX_TASKS-bounded per-key issued-generation high-water mark across replacement and temporary omission; import preserves the snapshot generation or raises it to that floor, so pre-import renew/release tokens are stale and the next claim cannot reuse a value issued before import. make run-task-coordinator proves the ordered round trip, stale-token rule, and replayed-snapshot high-water behavior in QEMU; the make task-coordinator-logic-test host gate covers hostile snapshots and atomic refusal. The optional writable-9p path admits import through a directory transaction: it validates and encodes the complete candidate before mutation, prepares task temporaries plus one canonical generation-floor record, syncs a rollback marker before the first destructive rename, backs up the prior set, publishes the replacement, and commits through the marker rename. Ordinary persistent mutations publish only the changed task record and the floor record in the same transaction, so they cannot split across a crash; complete-set replacement remains specific to import. The floor record retains at most the coordinator’s 256 entries and is bounded at 18,695 bytes. File reads and writes are split into requests of at most 4,096 bytes, so the logical floor record fits the writable-9p transport in at most five chunks. Keeping retired keys in one record avoids consuming the 20 live-task slots or one directory entry per key. The 64-entry directory ceiling is 60 task current/temporary/backup slots, three floor-record slots, and one transaction marker. A new identity beyond the 256-entry floor capacity retains the existing typed retained-generation-capacity-exhausted refusal. Boot recovery uses the same current/temporary/backup planner to roll the whole set backward before that decision or forward afterward, then restores generation floors before task records or dispatch. The resource-governance descriptor is authoritative for persistence-work accounting and proof coverage. Keys omitted from the snapshot lose their live record but retain their coordinator-local floor; snapshots still contain no floor table. Rejected imports preserve the prior live and host-visible set. The two-boot proof recreates an omitted key above its pre-restart high-water and rejects the pre-restart token. This is a host-filesystem plus QEMU Tfsync proof, not the fenced CAPOSRS1 WAL or a production-storage durability claim, and it adds no BlockDevice authority. The bounded HTTP transfer routes landed 2026-07-27 07:21 UTC (task record): GET /v1/state/export returns the coordinator’s snapshot bytes unchanged, and POST /v1/state/import forwards its opaque body to importState. Adapter request and response payloads use the coordinator’s 60 KiB snapshot ceiling; the whole HTTP request remains capped at 64 KiB, and the protocol tests prove a 60 KiB snapshot’s multi-segment Cap’n Proto import call fits below the Endpoint ceiling. Incomplete or oversized imports never reach the coordinator. Successful exports use application/octet-stream, while import results and typed refusals retain the API’s existing JSON result/error shapes. The adapter does not inspect or repair the snapshot format. make test-task-coordinator-api proves a deterministic export/import round trip, a maximum-size opaque import reaching the coordinator, and atomic hostile malformed and unknown-field refusals through the host-forwarded port. Local host inspection and lossless round-trip tooling landed 2026-07-27 18:09 UTC: tools/capos-task-coordinator-snapshot.py decode projects a coordinator snapshot into deterministic UTF-8 JSON while preserving task, dependency-edge, and conflict-domain order, and encode validates that exact documented JSON schema before recreating importable snapshot bytes. The tool accepts only the canonical five-field TCS1 layout, reads snapshot input only up to the coordinator’s 60 KiB ceiling, shares the encode_record/decode_record key and list bounds, rejects unknown fields, invalid statuses, dangling references, cycles, duplicate keys, exhausted generations, and limit violations with stable classified exits, and writes no output before full validation. make task-coordinator-logic-test pins the Python codec to a snapshot produced by Coordinator::export_state and covers deterministic JSON, byte-identical decode/encode, hostile inputs, and no-overwrite output behavior. The tool reads and writes local files only; it makes no coordinator network call. The local loopyard interchange projection landed 2026-07-27 20:35 UTC: to-loopyard maps snapshot JSON into the strict loopyard.interchange/v1 record set, and from-loopyard validates that interchange before recreating snapshot-shaped JSON. Both directions preserve task, dependency-edge, and conflict-domain order. The interchange matches the installed adapter’s task shape: coordinator conflict domains occupy the ordered resources array and paths must be empty because this coordinator does not model path domains separately. The outbound form declares its metadata loss explicitly by using each task id as the title and normal as the priority; inbound conversion validates and drops those fields because the coordinator has no title or priority state. The adapter’s portable [A-Za-z0-9][A-Za-z0-9._-]{0,63} id grammar is narrower than the coordinator’s UTF-8 key grammar, so outbound conversion explicitly refuses otherwise-valid coordinator keys that cannot become adapter task ids. Unknown fields, invalid statuses, non-empty paths, dangling references, cycles, duplicate keys, exhausted generations, oversized inputs, and limit violations fail before output is written. Both directions prove that the converted task set still encodes below the coordinator’s 60 KiB TCS1 ceiling before publishing output. The projection remains local-file only and makes no coordinator, loopyard, or PostgreSQL call. The offline apply-plan step landed 2026-08-02 02:47 UTC: plan-loopyard TARGET OUTPUT [--current CURRENT] validates the target and optional current loopyard.interchange/v1 documents under the same strict interchange rules, with the projection’s 512 KiB bounded-JSON ceiling for each input and a separate proof that each decoded state still fits the coordinator’s 60 KiB TCS1 ceiling, before creating any output. It emits deterministic loopyard.apply-plan/v1 JSON in task-record, resource-set, dependency-edge, and status-transition order. Every desired or current entry is classified new, unchanged, or conflicting; resource arrays use set semantics while retaining input order as evidence, dependency additions/removals retain their source order, and the dependency_edges section is the sole dependency authority. Task-record payloads deliberately omit dependencies, so arbitrary valid interchange order cannot make record creation depend on a later record. Only direct coordinator-valid status edges are actionable. Unsupported task removal, non-plannable record metadata drift, and invalid status edges remain explicit operator-decision conflicts. The command is local-file only, deterministic for identical inputs, and never replaces an existing output path. Plan serialization is bounded by a structurally derived 8,519,680-byte ceiling that covers the maximum target/current task and dependency counts rather than reusing the smaller interchange-input ceiling. verify-plan PLAN TARGET [--current CURRENT] is the local-file-only verification companion. It strictly validates the supplied loopyard.apply-plan/v1 document under the planner’s 8,519,680-byte output ceiling and both interchange inputs under their 512 KiB input ceiling. It rejects missing, unjustified, or contradictory plan entries, reconstructs the post-plan target state from the plan’s task, resource, dependency, and status evidence, and proves that re-planning from that state to the target is an all-unchanged fixed point with no actionable transition. It also re-plans the supplied inputs and requires the plan to match the deterministic projection byte-for-byte. Verification writes no output and makes no coordinator, loopyard, or PostgreSQL call. Still missing: authenticated loopyard-side ingestion tooling that applies this plan to PostgreSQL. That importer requires operator credentials and authorization; the offline planner neither authenticates nor mutates loopyard. Endpoint lease actor attribution is landed as described in gap 5, but leases remain deliberately absent from this export/import format, so the state-transfer contract is unchanged. The optional static API-token rule now covers both transfer routes before their bodies or snapshots can reach the coordinator. The fixed-session local proof using credential_store / user_session is described in gap 5. The host vibe-loop adapter can now authenticate its task-source and fenced-lock calls. Its export-state OUTPUT command now fetches the opaque snapshot through that same loopback-only, redirect-refusing, environment-bearer client and creates an owner-only output without replacing an existing path. Its import-state --confirm-replace-all INPUT command reads at most the coordinator’s 60 KiB ceiling before opening the request and forwards those bytes unchanged. Both commands preserve the adapter’s typed 401 and 503 exit behavior and bounded retry guidance. The adapter does not inspect, repair, or re-encode snapshots, and the coordinator remains authoritative for task state and fencing generations. This adds no loopyard/PostgreSQL ingestion, credential issuance or rotation, public ingress, or production backend authority. Production credential registry/session lookup, authenticated loopyard-side ingestion, and production ingress authority remain missing.

Sequencing

Phases ordered so each lands on local QEMU evidence before touching the gaps behind it:

  1. In-memory coordinator local proof (landed): coordinator service plus client demo proving runnable-set computation, transition validation, dependency-cycle rejection, conflict domains, and fenced leases (expiry, generation increment, stale-token rejection) in QEMU, with the pure rules under host tests. Proofs: make run-task-coordinator, make task-coordinator-logic-test. Root task record: task-backend-coordinator-inmemory-local-proof.
  2. API surface local proof (landed 2026-07-14 03:32 UTC): HTTP/JSON adapter over the userspace network stack exposing the coordinator to a host-side client speaking the task-source/lock command contracts against local QEMU. The task_api_mode of the Phase C network-stack server spawns the coordinator and the task-coordinator-api adapter; the host harness (tools/qemu-task-coordinator-api-smoke.sh) drives the JSON contracts through a SLIRP hostfwd port. Proofs: make test-task-coordinator-api, make task-coordinator-logic-test. Task record: task-backend-api-surface-local-proof. The bounded concurrent-connection extension landed 2026-07-14 13:00 UTC with per-connection socket service objects, readiness-driven multiplexing, and fail-closed table exhaustion; task record: network-stack-concurrent-socket-objects-task-api-serving.
  3. Durable record store: the transactional record store over BlockDevice, then switch the coordinator’s persistence from in-memory state to it, with reboot and crash-recovery proofs. First increment landed: the capos-lib CAPOSRS1 WAL core (host logic, task-backend-record-store-core-host-logic). Second increment landed: the BlockDevice wiring proof — demos/record-store-blockdevice persists the WAL through the typed BlockDevice cap and proves reboot durability plus bounded torn-write crash recovery across three boots of one disk image (make test-record-store, task-backend-record-store-blockdevice-wiring-local-proof). Third increment landed (2026-07-15 17:00 UTC): the coordinator’s persistence is switched onto the store — demos/task-coordinator-persist-proof drives the real task-coordinator-logic coordinator write-through over a granted BlockDevice cap, committing each mutation’s task/status/lease snapshot as one atomic frame and rebuilding live state from the log on boot; make test-task-coordinator-persist proves reboot reload and bounded forced-poweroff crash recovery across three boots of one disk image (task-backend-coordinator-persistence-record-store-local-proof). Fourth increment landed (2026-07-15 17:52 UTC): WAL compaction — the compaction-capable CAPOSRS1 format (RecordStore::format_compacting / compact) reclaims append-only log space by folding the live set into a fresh generation over an A/B superblock plus two segments, with the same crash-consistency guarantee (an interrupted compaction recovers to the pre- or post-compaction state via the generation/epoch fence); demos/record-store-compaction-proof and make test-record-store-compaction prove reboot log-space reclamation and a bounded forced-poweroff in the compaction flip window (task-backend-recordstore-wal-compaction). Remaining at this level: a durable optional-BlockDevice-cap path in the Endpoint-serving task-coordinator-service. A parallel QEMU-only inspection increment landed in task-backend-9p-persistence: the service uses a spawn-granted writable 9p Directory and bounded current/temp/backup task records; make test-task-coordinator-9p proves restart reload and clean host-visible state. This shortcut does not replace the BlockDevice path. The dependency-ordered task-backend-9p-api-persistence follow-on is implemented in commit 2c291abc: the existing Phase C userspace-network topology grants the same bounded state cap only to the coordinator while its storage-blind HTTP/JSON adapter remains host accessible. make test-task-coordinator-api-9p proves API-visible reload, restored-lease invalidation, stale-renew rejection, a new fencing generation, and post-restart mutation across two boots over one host share. Its device-composition prerequisite is the explicit virtio_9p_host_fixture feature described in Virtio-9p Host Directory Passthrough. The feature carries only the 9p fixture into the not(qemu) Phase C build, and make test-virtio-9p-net-coboot proves a writable 9p Directory and the userspace Nic path serve in one boot. The final local integration increment, task-backend-vibe-adapter-qemu-proof, is implemented in commit 05324347: a real vibe-loop-compatible host adapter drives the persistent API while the coordinator retains the only storage authority. make task-backend-vibe-adapter-test verifies the installed CLI’s runtime-owned default lifecycle, and make test-task-backend-vibe-adapter-9p proves authenticated adapter calls plus fenced lock and workspace metadata behavior across two QEMU boots. This adds only the bounded local bearer path described in gap 5, not public ingress, production authentication, or production storage authority.
  4. Actors, time, and portability: the coordinator-local bounded export/import contract and its bounded HTTP transfer routes are landed (gap 7). Direct Endpoint lock leases are now attributed to their acquiring caller session and reject cross-session renew/release separately from stale fencing tokens (gap 5). Bounded static API-token admission and HTTP actor projection, including both transfer routes, are also landed, and the host vibe-loop adapter can present a bounded environment-only bearer. Remaining work is production identity-bearing session/credential lookup, progressing trusted-time synchronization for cross-reboot lease and timestamp comparability, authenticated host-side loopyard/PostgreSQL ingestion tooling, and production ingress/authentication authority.
  5. Live deployment: serve the backend behind the public ingress/TLS milestone once that separate track is authorized and closed.

Phases 1 and 2 can proceed against in-memory state while phase 3 is designed; the coordinator’s storage seam should keep the record-store swap mechanical.

Design Grounding