Skip to content

feat(plugin-go): the shared golist GOCACHE becomes a declared scratch - #435

Merged
raphaelvigee merged 3 commits into
raphaelvigee/scratch-lineagefrom
raphaelvigee/scratch-go-golist
Sep 3, 2026
Merged

feat(plugin-go): the shared golist GOCACHE becomes a declared scratch#435
raphaelvigee merged 3 commits into
raphaelvigee/scratch-lineagefrom
raphaelvigee/scratch-go-golist

Conversation

@raphaelvigee

Copy link
Copy Markdown
Member

Third of three (on top of #434). The retrofit that proves the mechanism.

plugin-go's golist_gocache.rs was this feature, implemented once, for one
driver, by hand: a directory under the engine home keyed by a hand-written
struct, with no lock, no eviction, no remote, no visibility, and available to
nothing else. It is now an ordinary scratch target —
//@heph/go/gocache:cache@<factors> — which supplies all of those.

The provider serves it from an early handle_get arm, before package decoding,
exactly as the hermetic toolchain is served: there is no such directory on disk,
and asking the filesystem about it would only produce a confusing "not found".
The golist driver takes a scratch reference and reads the mount off RunRequest
instead of resolving a directory itself.

This is the design's acceptance test — if the general mechanism cannot express
the special case, the design is wrong.
It can, with one correction worth stating
plainly.

GOROOT leaves the key, and that is an improvement

The old key included the resolved GOROOT path, because Go's action IDs
incorporate it. A scratch declaration is resolved before the driver runs and
GOROOT is not known until it has — so the path cannot be in the key.

Dropping it is not a loss:

  • Under gotool: "host", GOROOT is a function of the Go version, which is in
    the key. Nothing changes.
  • Under a hermetic toolchain the SDK stages at a different absolute path in
    every sandbox, so the old key gave every sandbox its own slot and every one of
    them fell back to cold. Dropping the path gives them one shared slot.

It does not make a hermetic build's action IDs match across sandboxes —
those still embed the staged GOROOT — so the win there stays bounded until the
SDK is staged at a stable path (see hermetic-goroot-stable-path). It does mean
the cache stops being fragmented for no reason.

The remaining factors (go version, goos, goarch, build tags, goexperiment, race)
live in the addr as readable args rather than a hash, because they show up in
heph tool scratch ls and "which cache is this?" is a question a hash cannot
answer. The engine keys the slot on the addr, so two runs agreeing on every
factor share a directory and two that disagree on any of them do not — asserted
per factor.

access = "shared", because Go's build cache is content-addressed and
self-verifying and concurrent access is what go build -p N already does;
serializing it would turn the 2.4× into a large loss. That property was
previously expressed by having no lock at all; it is now stated.

Local-only for now. Publishing a Go build cache is worth doing and is a separate
change with its own before/after, since whether the transfer beats a cold
go list is a measurement rather than a guess.

What still needs a number — please label this perf-test

.github/workflows/perf.yml runs on a PR labelled perf-test and times a
500-package Go corpus against N−1 across cold / full-hit / incremental.

cold is the scenario that matters here. The 2.4× was always an intra-run
win — 500 packages sharing one GOCACHE within a single cold build — and cold
wipes .heph3 on both sides, so it measures like for like. This change should be
neutral-to-better there.

That is a claim for the gate to settle, not for a local run. The seeded-GOCACHE
experiment on this same corpus cut go list CPU by 60% and moved wall time by
exactly zero, which is the standing reminder that an obvious cache win here
can be worth nothing.

Note the label does not retrigger the job on an already-open PR (pull_request
fires on opened/synchronize/reopened, not labeled) — label, then push or
re-run the job.

Safety

A missing mount falls back to a sandbox-local directory rather than failing: an
older host that does not carry scratch mounts on RunRequest must still be able
to run this driver, and a cold cache is slow, never wrong.

Not in this PR

The other three Go sites, each wanting its own number:

  • driver_compile.rs still creates a per-sandbox .heph-gocache — the same
    pattern this removes for golist, left behind because that fix was driver-local.
    Smaller upside (§2 measured all compilation at 15s against golist's 778s).
  • thirdparty.rs's GOMODCACHE/GOPROXY host passthrough — the largest CI
    effect, and the only one that narrows a hermeticity hole rather than widening
    one.
  • pkg_analysis.rs enumerates those vars for hashing and moves with them.

@raphaelvigee raphaelvigee added the ci/perf-test Opt in to running the perfbench (Tier A/B) job on this PR label Aug 29, 2026
raphaelvigee added a commit that referenced this pull request Aug 29, 2026
It ran on #435 and #437 across all three supported targets, over cold/full-hit/
incremental, and reported no regression. Stated as the negative it is — the job
fails on REGRESSION and did not — rather than as a measured win, which would need
the per-scenario numbers in the step summary that this document does not carry.

Also records why the gate had never run despite being asked for twice: the label
alone does nothing. `pull_request:` takes the default event types, `labeled` is
not among them, and a re-run replays the payload from before the label existed.
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/scratch-go-golist branch from 31ed2c0 to 49a5001 Compare August 31, 2026 09:39
raphaelvigee added a commit that referenced this pull request Aug 31, 2026
It ran on #435 and #437 across all three supported targets, over cold/full-hit/
incremental, and reported no regression. Stated as the negative it is — the job
fails on REGRESSION and did not — rather than as a measured win, which would need
the per-scenario numbers in the step summary that this document does not carry.

Also records why the gate had never run despite being asked for twice: the label
alone does nothing. `pull_request:` takes the default event types, `labeled` is
not among them, and a re-run replays the payload from before the label existed.
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/scratch-go-golist branch from 49a5001 to 827d7b0 Compare August 31, 2026 09:52
raphaelvigee added a commit that referenced this pull request Aug 31, 2026
It ran on #435 and #437 across all three supported targets, over cold/full-hit/
incremental, and reported no regression. Stated as the negative it is — the job
fails on REGRESSION and did not — rather than as a measured win, which would need
the per-scenario numbers in the step summary that this document does not carry.

Also records why the gate had never run despite being asked for twice: the label
alone does nothing. `pull_request:` takes the default event types, `labeled` is
not among them, and a re-run replays the payload from before the label existed.
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/scratch-go-golist branch from 827d7b0 to f021585 Compare August 31, 2026 10:31
raphaelvigee added a commit that referenced this pull request Aug 31, 2026
It ran on #435 and #437 across all three supported targets, over cold/full-hit/
incremental, and reported no regression. Stated as the negative it is — the job
fails on REGRESSION and did not — rather than as a measured win, which would need
the per-scenario numbers in the step summary that this document does not carry.

Also records why the gate had never run despite being asked for twice: the label
alone does nothing. `pull_request:` takes the default event types, `labeled` is
not among them, and a re-run replays the payload from before the label existed.
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/scratch-go-golist branch from f021585 to 750719c Compare August 31, 2026 11:03
raphaelvigee added a commit that referenced this pull request Aug 31, 2026
It ran on #435 and #437 across all three supported targets, over cold/full-hit/
incremental, and reported no regression. Stated as the negative it is — the job
fails on REGRESSION and did not — rather than as a measured win, which would need
the per-scenario numbers in the step summary that this document does not carry.

Also records why the gate had never run despite being asked for twice: the label
alone does nothing. `pull_request:` takes the default event types, `labeled` is
not among them, and a re-run replays the payload from before the label existed.
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/scratch-go-golist branch from 750719c to 4c4d7f7 Compare August 31, 2026 12:59
raphaelvigee added a commit that referenced this pull request Aug 31, 2026
It ran on #435 and #437 across all three supported targets, over cold/full-hit/
incremental, and reported no regression. Stated as the negative it is — the job
fails on REGRESSION and did not — rather than as a measured win, which would need
the per-scenario numbers in the step summary that this document does not carry.

Also records why the gate had never run despite being asked for twice: the label
alone does nothing. `pull_request:` takes the default event types, `labeled` is
not among them, and a re-run replays the payload from before the label existed.
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/scratch-go-golist branch from 4c4d7f7 to 1db23cd Compare September 2, 2026 08:54
raphaelvigee added a commit that referenced this pull request Sep 2, 2026
It ran on #435 and #437 across all three supported targets, over cold/full-hit/
incremental, and reported no regression. Stated as the negative it is — the job
fails on REGRESSION and did not — rather than as a measured win, which would need
the per-scenario numbers in the step summary that this document does not carry.

Also records why the gate had never run despite being asked for twice: the label
alone does nothing. `pull_request:` takes the default event types, `labeled` is
not among them, and a re-run replays the payload from before the label existed.
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/scratch-go-golist branch 3 times, most recently from f6b5be1 to 05fbad3 Compare September 2, 2026 09:54
raphaelvigee added a commit that referenced this pull request Sep 2, 2026
Renamed in place on the repo (`gh label edit`), so it keeps its id and
stays applied to the PRs already carrying it — #435 and #437 both still
have it.

The functional reference is `perfbench`'s `if:`; the rest are comments
here and in `perf.yml`. Not touched: the `perf-test` *skill* in
`.claude/skills/` and the references to it in `.claude/agents/` and
`.cargo/config.toml`. That is samply profiling, unrelated to the label
beyond sharing a name.

One transition window worth knowing about. A `pull_request` run resolves
its workflow from the merge of head into base, so an open PR carrying
the label gets the renamed `if:` as soon as this lands on master, with
no rebase needed. Before it lands, those PRs hold a label the workflow
on their own base does not match — perfbench will not fire for them in
that window. It is an opt-in check on two PRs, so the cost is a re-label
or a rebase, not a lost signal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQ8kRpuBjgxsSf15RjLDZz
raphaelvigee added a commit that referenced this pull request Sep 2, 2026
Renamed in place on the repo (`gh label edit`), so it keeps its id and
stays applied to the PRs already carrying it — #435 and #437 both still
have it. With `ci/force-ci` from the commit below, both CI labels now
share one namespace; a lone `pr/` prefix on one of them was a typo
waiting to happen.

The functional reference is `perfbench`'s `if:`; the rest are comments
here and in `perf.yml`. Not touched: the `perf-test` *skill* in
`.claude/skills/` and the references to it in `.claude/agents/` and
`.cargo/config.toml`. That is samply profiling, unrelated to the label
beyond sharing a name.

One transition window worth knowing about. A `pull_request` run resolves
its workflow from the merge of head into base, so an open PR carrying
the label gets the renamed `if:` as soon as this lands on master, with
no rebase needed. Before it lands, those PRs hold a label the workflow
on their own base does not match — perfbench will not fire for them in
that window. It is an opt-in check on two PRs, so the cost is a re-label
or a rebase, not a lost signal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQ8kRpuBjgxsSf15RjLDZz
raphaelvigee and others added 3 commits September 3, 2026 01:33
`golist_gocache.rs` was this feature, implemented once, for one driver, by hand:
a directory under the engine home keyed by a hand-written struct, with no lock,
no eviction, no remote, no visibility, and available to nothing else. It is now
an ordinary `scratch` target — `//@heph/go/gocache:cache@<factors>` — which
supplies all of those.

The provider serves it from an early `handle_get` arm, before package decoding,
exactly as the hermetic toolchain is served: there is no such directory on disk,
and asking the filesystem about it would only produce a confusing "not found".
The golist driver takes a scratch reference and reads the mount off `RunRequest`
instead of resolving a directory itself, so nothing creates or tears down a cache
per sandbox — the only thing that ever moved cold wall time.

This is the design's acceptance test: if the general mechanism could not express
the special case, the design would be wrong. It can, with one correction worth
stating.

## GOROOT leaves the key, and that is an improvement

The old key included the resolved GOROOT path, because Go's action IDs
incorporate it. A scratch declaration is resolved *before* the driver runs and
GOROOT is not known until it has, so the path cannot be in the key.

Dropping it is not a loss:

- Under `gotool: "host"`, GOROOT is a function of the Go version, which *is* in
  the key. Nothing changes.
- Under a hermetic toolchain the SDK stages at a different absolute path in every
  sandbox, so the old key gave every sandbox its own slot and every one of them
  fell back to cold. Dropping the path gives them one shared slot.

It does not make a hermetic build's action IDs match across sandboxes — those
still embed the staged GOROOT — so the win there stays bounded until the SDK is
staged at a stable path. It does mean the cache stops being fragmented for no
reason.

The remaining factors (go version, goos, goarch, build tags, goexperiment, race)
live in the addr as readable args rather than a hash, because they show up in
`heph tool scratch ls` and "which cache is this?" is a question a hash cannot
answer. The engine keys the slot on the addr, so two runs agreeing on every
factor share a directory and two that disagree on any of them do not — asserted
per factor.

`access = "shared"`, because Go's build cache is content-addressed and
self-verifying and concurrent access is what `go build -p N` already does.
Serializing it would turn the 2.4x into a large loss. That property was
previously expressed by having no lock at all; it is now stated.

Local-only for now: publishing a Go build cache is worth doing and is a separate
change with its own before/after, since whether the transfer beats a cold
`go list` is a measurement rather than a guess.

## What still needs a number

The perf gate is `.github/workflows/perf.yml`, which runs on a PR labelled
`perf-test` and times a 500-package Go corpus against N-1 across cold/full-hit/
incremental. `cold` is the scenario that matters here — the 2.4x was always an
intra-run win, 500 packages sharing one GOCACHE within a single cold build, and
`cold` wipes `.heph3` on both sides so it measures like for like. This change
should be neutral-to-better there; that is a claim for the gate to settle, not
for a local run, since the seeded-GOCACHE experiment already showed an obvious
win moving wall time by exactly zero.

A missing mount falls back to a sandbox-local directory rather than failing: an
older host that does not carry scratch mounts on `RunRequest` must still be able
to run this driver, and a cold cache is slow, never wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FArWjycMDyWeSfHHtpgtoU
The `perfbench` job is gated on the `perf-test` label, and `heph.yml`'s
`pull_request:` trigger takes the default event types — which do not include
`labeled`. So applying the label does not start a run, and re-running replays
the original payload, where the label was still absent. A push with the label
already applied is what actually reaches the gate.

Empty on purpose; `master` is squash-only, so it does not survive the merge.
The field was removed from the `scratch` declaration in the base PR, but this
one kept setting it. A declaration rejects unknown fields, so that is not a stale
comment — it is a hard parse failure of every Go target referencing the cache.

No unit test noticed, because the spec test asserted individual config keys and
never parsed the thing it built. It parses now, and that test discriminates:
putting `platform` back fails with `unknown entries found: ["platform"]`.

A Go build cache's entries depend on the **target** (`GOOS`/`GOARCH`) and the
toolchain — both already in the addr — not on the machine that ran the compiler,
which is what `platform = "os_arch"` was folding in. Dropping it also stops a
laptop cross-compiling to `linux/amd64` and a CI runner building it natively from
getting different slots for identical content.
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/scratch-go-golist branch from 05fbad3 to ff75fac Compare September 2, 2026 23:41
@raphaelvigee
raphaelvigee merged commit a7fc0ba into master Sep 3, 2026
22 checks passed
@raphaelvigee
raphaelvigee deleted the raphaelvigee/scratch-go-golist branch September 3, 2026 08:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/perf-test Opt in to running the perfbench (Tier A/B) job on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant