Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions docs/contrib/local-check.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,15 @@ matches, for example a red unit again. The script reads every list from its owne
`python3 tools/recipes.py pure-crates`, the Python self-tests from the `selftests=(` array of `tools/check-recipes.sh`,
and the host units from `python3 tools/recipes.py host-nightly`.

`python3 tools/recipes-key-test.py --self-test` checks the public key/ledger CLI in isolated, offline Cargo
workspaces. Called-source edits must invalidate a recorded green result even when size and mtime stay unchanged.
Absolute paths and raw identifiers retain conservative owner/dependency bounds when structural resolution is
uncertain; Cargo-selected roots and declared Rust modules are source regardless of their filename suffix.
The persistent structural cache must preserve these inputs across fresh processes.
The selected profiles of every box call remain inputs, including shell defaults and inherited model overrides.
Changing a selected profile or called source invalidates its consumer; an unrelated private profile or unreached
source does not. Shared box inputs remain mandatory for every box command.

The mapped checkpoint reader is `bloomery-safetensors`; `gate-safetensors` owns its dtype, span and shard tests.
`f32_values` also checks directly constructed `Raw` values: shape and dtype must account for the complete
byte span without overflow. A partial value or a mismatched span is `Error::Format`, not a truncated conversion.
Expand Down
3 changes: 3 additions & 0 deletions tools/affected-gates.sh
Original file line number Diff line number Diff line change
Expand Up @@ -36,5 +36,8 @@
# the crate graph's true answer. The saving is on leaf changes (serve, tokenizer, sampler, one gate
# binary, a runner, docs); the larger value is the list itself: no forgotten gate, and a gate left out
# is a printed record, not a judgment. It cannot see $BLOOMERY_DATA, the ik trees, or the card.
# Ledger keys deliberately differ: `recipes.py key` may exclude proven-unreached dependency sources
# while retaining own-target, build/proc-macro, script and opaque bounds. This affected list remains the
# broad compile/selection contract; narrowing a key does not remove a recipe from the affected list.
set -euo pipefail
exec python3 "$(dirname "$0")/recipes.py" affected "$@"
3 changes: 3 additions & 0 deletions tools/bloomery/fixture_dirs.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@
"""

import re
import sys

_REACH_LOADED_CODE = sys._getframe().f_code

FIXTURE_DIRS = "tools/fixture-dirs.tsv"
REF_PATHS = "tools/ref/ref-paths.sh"
Expand Down
477 changes: 477 additions & 0 deletions tools/bloomery/reach_cache.py

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions tools/check-recipes.sh
Original file line number Diff line number Diff line change
Expand Up @@ -387,6 +387,7 @@ pytools_list() {
"tools/model_tax.py --self-test"
"tools/owners_check.py --self-test"
"tools/pr-gate-lines.py --self-test"
"tools/recipes-key-test.py --self-test"
"tools/ref/asm-census.py --self-test"
"tools/ref/cache-options-test.py --self-test"
"tools/ref/check-int-twins.py --self-test"
Expand Down
23 changes: 19 additions & 4 deletions tools/gate-batch.sh
Original file line number Diff line number Diff line change
Expand Up @@ -359,10 +359,14 @@
# A skip names its source, `src=lead` or `src=round`, in run.log and the dry run. A rebase moves the
# key of every item whose closure the landed commits touched, so a landing still reruns those; the
# saving is in the items the rebase left alone.
# Each item's input key is `tools/recipes.py key` (its section header lists the parts): the files its
# targets read (the walk `just affected` uses; the whole tree for a command run on the Mac, a cargo
# subcommand the parser does not model, or a script that walks the tree), the text of its recipe and
# of every recipe it depends on, the scripts those name, the cargo globals and the cuda-oxide pin, its
# Each item's input key is `tools/recipes.py key` (its section header lists the parts): its own target
# sources, uncut reached dependency sources, build/procedural-macro compilation inputs, named script
# reads and manifests. Compiled regular-source macro invocations add structural roots even in otherwise
# unreached modules; implicit impls include non-.rs #[path] modules. Opaque expansion/linkage keeps
# conservative bounds; a Mac command, unmodelled cargo subcommand or tree-walking script still keys
# the whole tree. `just affected` keeps its broader
# compile closure: an affected recipe need not have a changed ledger key. Keys also include the text
# of the recipe and every dependency recipe, the scripts those name, cargo globals and cuda-oxide pin, its
# ARGS, its full BLOOMERY_BOX_ENV (the caller's, the item's, its lane's card), the Mac-side variables
# box.sh carries, and a box manifest read once per batch through one box.sh call (`recipes.py
# box-manifest` behind box.sh's guard with BLOOMERY_BOX_WAIT=0: refused while the lease or a hold is
Expand All @@ -375,6 +379,17 @@
# is not bounded by its recipe text, so a data change reruns every item — the safe side; a model
# directory's rows enter only the keys of the items that can open it (a /models literal in the item's
# files or recipe text, its profile among them, or a /models value in the box env).
# Structural reach is cached privately in ~/.cache/bloomery/reach/<checkout-id>/graph.json. The cache
# contains no green verdicts: source/code/metadata/read/probe content must still match, and malformed or
# stale snapshots rebuild. Code-identity compilation is memoized by exact source bytes, filename and
# optimization level; cached recipe inputs still recheck current content, and key output is validated again.
# Immutable file-local syntax facts are memoized by exact text; bindings and macro resolution stay
# graph-specific. Schema descriptors are reused, never value-validation verdicts. Module ancestors are
# collected once per root/file through the existing base index, not once per reached item.
# The snapshot persists structural owners/nodes/edges, not the recomputable resolver lookup memo.
# Cache hydration discards eager resolver results; lookups are recomputed from the validated structure.
# Runtime reach is not profile-pruned for keys. The v2 key format deliberately invalidates pre-reach ledger
# entries once; an old green cannot authorize a narrower new key.
# Keys are computed before the lease check. An item whose key is in the ledger does not run: run.log
# gets `<stem> rc=skip green-at=<commit> <date> lane=<L>`. Every other item runs as without --ledger;
# one whose final rc is 0 is recorded after its key is computed again (against the same pre-batch box
Expand Down
Loading
Loading