feat: implement the kernal-api capability spine and fully rebase zccache embedded

[zackees]

Context

The architecture is settled, but it now needs an executable delivery plan spanning both repositories:

zccache embedded -> kernal-api -> running-process -> Tokio/native OS

Today zccache still exposes/uses Tokio and running-process directly, and zccache-platform contains a second cross-platform implementation tree for process, filesystem, IPC, executable, and host operations. The platform crate is already a product-type-free dependency leaf, so most of its code is a strong candidate to become kernal-api capability implementation rather than remain duplicated inside zccache.

This issue is the implementation umbrella. zackees/zccache#1518 remains the consumer-side migration tracker; zackees/running-process#1143 tracks any narrowly scoped substrate changes.

Proposal

Implement this as ordered phases. Each phase must add focused RED -> GREEN contract tests before moving callers, and each phase must leave both repositories releasable.

Phase 0: compatibility inventory and baselines

• Produce a checked-in mapping for every public zccache-platform operation under process, fs, ipc, executable, and host:
  • reuse an existing kernal-api contract;
  • extend a kernal-api contract;
  • move generic native code into kernal-api; or
  • retain product policy in zccache with a written reason.
• Capture current zccache embedded shutdown/cancellation, process lifecycle, IPC, filesystem, and broker behavior in focused tests.
• Preserve the frozen broker framing, zccache payload protocol 0x7A63, endpoint behavior, and single-round-trip request path.
• Record clean and representative incremental build baselines through soldr cargo build --timings and zccache's sanctioned performance workflow where runtime behavior is affected.

Phase 0.5: trim and prove the running-process substrate feature

• Define the smallest running-process feature surface required by kernal-api for native process/OS behavior.
• With default features disabled, this substrate must not compile broker protobuf generation, CLI/config parsing, SQLite, PTY/ConPTY acquisition, daemon binaries, or other client/daemon-only machinery.
• Gate protobuf build dependencies (prost-build/protox) and generated broker code behind the feature that consumes them; a process-only dependency must not run the broker schema build.
• Record soldr cargo tree and clean-build timing for the substrate alone. Compare it to the phase-0 kernal-api baseline before accepting the dependency.
• Make running-process#1143 a landing prerequisite if this requires changes in that repository.
• Do not begin the facade adapter by accepting a larger graph and promising to trim it later: dependency shape is part of the phase contract.

Phase 1: establish the native substrate adapter

• Add the exact running-process dependency and feature set needed by kernal-api; never add the reverse dependency.
• Expose approved canonical running-process contracts through kernal-api namespace re-exports/aliases, with identical Rust type identity rather than mirrored definitions. Preserve genuine facade policy where needed; unrelated Tokio, raw generated protobuf and native OS internals remain private. Coordinate feat: re-export canonical running-process Inherited and Independent spawn API #189/refactor: audit duplicated running-process APIs and plan canonical re-exports #196.
• Add characterization tests for child containment, owner death/kill-on-drop, bounded capture, detached launch, process inspection, endpoint identity, broker framing, manifests, service definitions, routes, and refusal classes.
• Add a narrowly scoped running-process primitive only when the mapping proves the existing API cannot preserve behavior.

Phase 2: complete the embedded async and I/O spine

• Provide facade-owned runtime handles, cancellation tokens, tasks, clocks, deadlines, channels, async I/O traits/types, and blocking-work dispatch needed by zccache embedded.
• Provide the kernel network stack with mandatory connection deadlines and progress/idle timeout primitives; do not encode a single wall-clock timeout for downloads that continue making progress.
• Make Tokio Console/diagnostic integration composable so zccache can keep its logging layers while gaining deadlock/task inspection.
• Rebase zccache embedded public APIs and RuntimeHooks onto kernal_api types. No public embedded signature or core implementation may name Tokio or tokio-util after this phase.

Phase 3: move generic host and executable mechanics

• Move or map zccache-platform::{host, executable} functionality into facade-owned kernel modules.
• Cover OS/architecture facts, user/runtime directories, current image identity, executable discovery, native naming, image comparison, and safe replacement behavior.
• Keep compiler-target policy in zccache. For example, explicit Rust target triples and compiler/linker selection are not host-kernel responsibilities.
• Move product-specific candidate lists or UI wording out of the platform layer instead of generalizing them artificially.

Phase 4: move generic filesystem mechanics

• Move or map file/path identity, change markers, volume identity, allocated size, hard-link limits/counts, link classification, symlink creation, private directories, permission/mode handling, canonical/verbatim paths, durable directory sync, shared append, atomic replace, no-replace rename, staged directory installation, and contention classification.
• Add facade-owned mapped-file access and advisory file-lock types where zccache currently needs them; resource lifetime and unlock-on-drop must be explicit.
• Preserve zccache cache-materialization semantics, especially stored mtimes, hardlink safety, and atomic installation. Those product policies stay in zccache while the OS mechanisms move down.
• Delete each native implementation from zccache-platform immediately after its kernel replacement is in use; do not maintain dual implementations.

Phase 5: move generic process mechanics

• Move or map command preparation, hidden-window/process-group setup, owner-death containment, priority, process inspection, termination, detached stdio/log redirection, native exit classification, and jobserver primitives.
• Express behavior through semantic specs such as command, stream, containment, priority, and child policies rather than exposing mutable Tokio commands or child handles.
• Rebase daemon launch, compiler probes, compiler children, and CLI daemon deployment onto the kernel process facade.
• Keep zccache-specific service names, environment switches, exit-code policy, and operator diagnostics in zccache.

Phase 6: move generic IPC and broker access

• Move or map local endpoint, listener, stream, peer identity, permissions, connect retry/progress timeout, and listener-pool behavior into kernal-api.
• Add facade-owned broker identity, frame, route/error, service-definition, manifest, and verified-backend contracts backed by running-process.
• Rebase zccache-ipc and zccache-protocol without changing golden bytes, negotiation, endpoint selection, or round-trip count.
• Keep the broker daemon implementation in running-process for this migration. Hoisting the generic broker-daemon pattern into kernal-api is explicitly later work after zccache, Soldr, and fbuild stabilize.

Phase 7: remove the zccache platform/backend layers

• Migrate all consumers currently aliasing zccache_platform (zccache, CLI core, daemon core, compiler, IPC, artifact, depgraph, core, and download protocol).
• Delete zccache-platform once its generic mechanics are represented by kernal-api. Any surviving zccache-specific policy must live in the owning product crate, contain no host syscall/import, and have a documented reason it is not a kernel capability.
• Remove all direct zccache dependencies/imports for running-process, Tokio, tokio-util, interprocess, crash-handler, libc, windows-sys, and other kernel-owned backends from core application code.
• Replace the transitional enforce_platform_boundary baseline with the strict kernal-api boundary/platform Dylints. No occurrence-count grandfathering remains.

Phase 8: release and prove the result

• Release and exactly pin the required kernal-api version before removing temporary workspace patches.
• Verify disabled features omit their dependency trees and embedded builds request only the capabilities they use.
• Compare the phase-0 and final clean/incremental timings, duplicate dependency tree, peak compiler memory, and cache reuse.
• Apply the same facade shape to Soldr and fbuild only after the zccache embedded migration is stable.

Acceptance criteria

• Each phase contains focused failing contract tests or a reproducible failing boundary check before implementation, followed by GREEN evidence.
• The minimal running-process substrate graph excludes broker protobuf generation, CLI/config parsing, SQLite, PTY/ConPTY acquisition, and daemon-only dependencies; its measured clean-build cost is accepted before kernal-api adopts it.
• zccache embedded public APIs and core implementation contain no Tokio, tokio-util, running-process, or native-OS types.
• zccache-platform is removed; any rejected move is documented as product policy and contains no generic host implementation.
• rg 'zccache_platform|zccache-platform|running_process|running-process' finds no production dependency or import in zccache, apart from intentional migration/history documentation.
• Kernel boundary Dylints reject normal, aliased, target, build, and test dependencies on owned backends, plus host cfg/native imports outside the kernel.
• There is one async runtime/network/process/allocator/crash/console implementation path in the resolved release graph.
• Embedded cancellation and shutdown, child containment, bounded probes, detached deployment, filesystem identity/replacement, IPC peer security, and broker compatibility tests pass on Linux, macOS, and Windows.
• Broker frame golden bytes and payload protocol 0x7A63 are unchanged, and no IPC round trip is added.
• Build-timing evidence reports clean and incremental deltas against phase 0; regressions require an explicit follow-up decision rather than being hidden by dependency consolidation.
• Both repositories' standard checks and package-from-archive tests pass before the migration is considered complete.

Decisions

• Priority is P1 because this work is the first proof of the architecture and removes duplicated security/build ownership.
• File this umbrella in kernal-api, where capability readiness and cross-client ownership are controlled; use refactor: migrate zccache completely from running-process to kernal-api zccache#1518 for consumer PR sequencing.
• Prefer mapping to existing kernel behavior over copying zccache-platform; port code only where the kernel has a verified semantic gap.
• Treat running-process dependency trimming as a prerequisite, because the current async-process-only graph still compiles unrelated unconditional build dependencies and would otherwise increase the kernel baseline.
• Target deletion of zccache-platform, because it is already product-type-free. Product policy discovered during extraction moves to the relevant zccache crate.
• Use capability-gated, reviewable releases rather than a flag-day cross-repository commit.
• Do not hoist the broker daemon itself during this migration.
• Do not create speculative traits or internal crates; introduce boundaries only for multiple implementations or measured compile benefit.

Open questions

• Does native crash-context formatting belong entirely in the kernel crash facade, or should zccache retain only presentation after receiving a kernel-owned exit report?
• Which current executable candidate lists are generic host discovery versus compiler-specific zccache policy?

Related issues

• feat: unify async OS HAL, profiling, symbolization, and allocator instrumentation #1 defines the canonical kernal-api ownership architecture.
• perf: benchmark kernal-api compilation boundaries and release packaging #3 tracks measured crate/package organization.
• refactor: define running-process as the native substrate for kernal-api running-process#1143 defines the native substrate contract.
• refactor: migrate zccache completely from running-process to kernal-api zccache#1518 tracks the complete consumer migration.

Phase-0 ownership split and landing gate

Phase 0 has one shared outcome but two owners; it is not a separate duplicate issue.

• feat: implement the kernal-api capability spine and fully rebase zccache embedded #5 owns migration compatibility inventory and behavior characterization. Before an adapter is accepted, check in a mapping for every public zccache-platform operation under process, fs, ipc, executable, and host: reuse, extend, move into kernel, or retain as product policy with a reason. Add named characterization fixtures for embedded shutdown/cancellation, child lifecycle, filesystem identity/replacement, IPC peer security, frozen broker framing, payload protocol 0x7A63, endpoint behavior, and the single-round-trip invariant.
• perf: benchmark kernal-api compilation boundaries and release packaging #3 owns the reproducible timing/dependency measurement protocol and results. It records toolchain/host, exact Soldr commands, feature sets, repeated clean/incremental samples, resolved dependency graphs, peak memory/cache-reuse evidence, and the comparison required before/after migration.
• refactor: define running-process as the native substrate for kernal-api running-process#1143 owns the substrate gate. A kernel adapter may not add running-process until its default-disabled substrate feature has proved that broker protobuf generation, CLI/config parsing, SQLite, PTY/ConPTY, daemon binaries, and other client/daemon-only machinery are absent from the resolved graph and build scripts.

The first independent implementation children are:

• feat(process): privately adapt the trimmed running-process substrate #6 ? process-only private adapter; blocked by the accepted substrate gate.
• feat(async): add cancellation, deadlines, and progress-aware I/O policy #7 ? fa?ade cancellation/deadline/progress-I/O policy; independent of the process substrate.
• feat(hash): add the kernel-owned BLAKE3 content facade #8 ? concrete kernel-owned BLAKE3 fa?ade; independent of the process substrate.

Platform/filesystem and broker children remain intentionally unfiled until the inventory identifies an exact current semantic gap; creating them now would duplicate this umbrella with speculative scope.

Active implementation coordination

The user has authorized completing this umbrella together with running-process#1202, #189 and #196. For equivalent running-process APIs, direct Rust re-exports and namespace aliases now supersede the older blanket facade-owned-type prohibition. All migration, protocol, lifecycle, feature-isolation, downstream-release and verification requirements remain in scope; this note does not claim they are completed.

Dependency edits use a temporary git checkout under _vender/<dependency>/ for integration. Publish and wait for the new dependency release, then remove temporary checkout/path patches and select its exact public version. Per user instruction, implementation and test authoring across repositories precede the build/lint/test phase.

[zackees]

Phase 0.5 is cleared: running-process#1143 and ordered children #1144-#1147 are now merged/closed. The supported running-process selection is default-features = false, features = [kernel-substrate]; it resolves only the reviewed native/async graph, has no reverse kernal-api edge, and is enforced for host plus published Windows x86_64/ARM64 MSVC graphs.

Validation: isolated Linux Bosn rc 0, 217/217 root library tests, 40 packages, 12.905s clean / 1.932s repeat, lock SHA d798de507e13b25099cec3692c902c63d989b4e1145905cc6e90b29480592a49. This clears the substrate prerequisite for Phase 1; #5 remains open for the adapter and migration phases.