ci(release): gate stable tags on the dig-constants real-genesis floor - #178
Conversation
Review of the envelope fix caught that `inclusion_proof` was gated on `start == 0`,
which is the same defect this change exists to remove, relocated to large resources.
`inclusion_proof` is in `ChunkObject.required` in docs.dig.net's openrpc.json and is
documented there as "Sent on every window for getContent/getManifest"; the retired
dighub-retrieval Lambda emitted it unconditionally, gating only `chunk_lens` on the
first window. Gating the proof meant windows 1..N of any resource over 3 MiB carried
no way to verify, and a client that begins mid-resource — a resumed or ranged read —
never received one at all. A well-formed response the client cannot verify, with no
error raised anywhere, is exactly the failure mode of #2071. chia-offer.on.dig.net is
15962 bytes and fits in a single window, which is why testing against it could not
have surfaced this.
`chunk_lens` keeps its first-window-only rule, which both normative sources agree on:
it describes how to split the reassembled resource, which a client cannot act on
until it holds every window.
SPEC.md §5.5.0 is corrected and completed:
* the field table now matches ChunkObject field-for-field, and says so, so the two
normative sources cannot drift again without one of them contradicting the other
in writing
* states the 3 MiB window, and that this node IGNORES the `length` request
parameter while openrpc documents it as honoured-then-clamped, so a client sizes
its stride from the length it is GIVEN
* names all three independent definitions of the window size (WINDOW here,
RPC_MAX_CHUNK in hub's retrieval Lambda, RPC_CHUNK in on.dig.net's service
worker) and references DIG-Network/dig_ecosystem#2076 to consolidate them into
dig-constants
* drops the inaccurate "every path, one shared builder" claim: the response-window
cache replays a proxied upstream result verbatim rather than rebuilding it, which
is deliberate (it preserves provenance) and is now stated as the exception
The window-size consolidation is deliberately NOT done here. dig-node carries four
versions of dig-constants (#2072) and PR #178 adds a release gate requiring it to be
single and current, so adding a constant means a version reconciliation this outage
fix should not be coupled to.
Tests: a new test asserts the proof on both the first and last window of a
multi-window resource, and that only the first carries chunk_lens. Falsified by
re-gating the proof on `start == 0` — the last-window assertion fails. 670/670
dig-node-core lib tests pass.
Refs DIG-Network/dig_ecosystem#2071
Co-Authored-By: Claude <noreply@anthropic.com>
8fb2870 to
d4ceb66
Compare
|
Holding this PR deliberately — the gate is correct, and merging it now would freeze dig-node releases. I ran the gate's own logic against Four distinct But only two of those seven rows are dig-node's own crates. The rest live in other repos and must bump and publish first (release-first, §4.1), and every 0.x minor between them is semver-BREAKING — Until it lands, this gate refuses every stable dig-node tag. dig-node shipped two emergency releases tonight — v0.93.9 to restore Sequencing: #2072 first, this second. Recorded as a hard dependency on that ticket. Nothing wrong with the PR; it is waiting on the world it is checking. One thing worth adding while it waits, since it makes the gate safer to land: consider whether the first release after this merges should print the offending consumer list ( |
The Release-script tests job globs scripts/tests/*.test.sh and main carries one test per script. #178 added check-dig-constants-current.sh and no test, so that job passed by running three pre-existing tests and never exercising the new gate at all -- a green that measured nothing. Each case is built against the nearest WRONG gate rather than merely against the right one: - the duplicate fixture holds TWO copies of which ONE IS THE TIP, so the currency check cannot be what fires; had both copies been stale, a gate that had lost the duplicate check entirely would still exit 1 and look green. - every case asserts the REASON, not just exit 1, because exit 1 alone cannot distinguish which of the two checks fired. - the fail-closed case uses an otherwise PERFECT lock, so an unreadable index is the only thing it can be measuring. - an honest control (single copy, at the tip, PASSES) keeps a gate that refuses everything from satisfying the whole file. Adds a CURL_BIN seam to the gate, mirroring check-glibc-floor.sh's READELF_BIN. It is what makes the read-failure case expressible at all, and the fixture versions are pinned out of band (9.x) so that removing the seam would make every expected substring stop matching rather than let the cases drift back onto the live index unnoticed. Refs #178 Co-Authored-By: Claude <noreply@anthropic.com>
…nner The script was committed mode 100644 while the workflow invoked it as `./scripts/check-dig-constants-current.sh`, which is Permission denied on a runner -- the gate would have failed the stable job for a reason unrelated to dig-constants, on every release. Sets the exec bit and switches the invocation to `bash scripts/...`, matching how build-binaries.yml already calls check-glibc-floor.sh, so the step no longer depends on a mode bit surviving a checkout. Refs #178 Co-Authored-By: Claude <noreply@anthropic.com>
Verified green, deliberately NOT merging yet — sequencing decisionHead Two real finds along the way, both worth recording:
Why this is heldThis gate will refuse a stable cut the moment it merges. Main's lock carries four That is the gate working exactly as designed. But merging it now means no stable dig-node release is possible until the drift is collapsed — and stable auto-update was down for ~15 hours today (#2290) precisely because of a dig-node release problem. Landing a release-blocking gate while that is fresh, with no hotfix path, is the wrong order. So: #2072 collapses the drift first, then this merges immediately after. The gate goes in green rather than pre-tripped. Dispatching #2072 now. Also queued behind PR #198 (the #2290 release-path fix), which edits the same |
Decision: this gate needs re-scoping before it merges. Holding, with a concrete redesign.The #2072 lane measured what happens when you try to satisfy it. Both clauses are currently unsatisfiable inside dig-node, and the "current" clause is unsatisfiable by design, not by circumstance. "current" is structurally unreachable
So reaching the tip is not a dep bump. It is a chia 0.26 → 0.36 migration across dig-gossip, digstore, dig-nat, dig-download, chia-peer and chia-query — a separate epic. The generalisation matters more than this instance: "single" is not reachable in dig-node eitherSeven holders, and only two are dig-node's own. The rest are pinned by published metadata a consumer cannot edit: What the gate should actually enforceThe lane's work also revealed the real defect class this gate exists to prevent, and it is not "is it the newest":
So the property worth gating is:
Optionally a third, as a warning not a block: "a newer I am rejecting the lane's suggestion to land this as-written with "current" demoted to a warning, because "single" would still block, so the outcome is the same freeze. Floor-plus-single is the version that is both satisfiable and protective. Why not just merge it and accept the freezedig-node main is 0.101.0; the latest stable tag is v0.99.9. Four versions of merged work are unreleased, including #2108's unauthenticated local inventory-enumeration fix. Stable auto-update was down ~15 hours today (#2290). Merging a gate that freezes stable cuts behind a multi-repo chia migration, in that state, leaves no hotfix path — and a gate that has to be bypassed in an emergency is a gate that will be bypassed and then ignored. What staysEverything else about this PR is good and should survive the re-scope: the 8-case test file, the 7-mutation falsifiability proof, the Keeping this open and held. #2072's PR #199 lands the part that is reachable now. |
…the chia tip dig-node's lock held FOUR dig-constants versions in one binary (0.1.0, 0.4.0, 0.5.1, 0.8.0). Only two of the seven holders are dig-node's own crates; the rest are pinned by upstream crates' PUBLISHED ranges, which a consumer cannot edit. The copy that mattered was 0.1.0, reached through dig-clvm 0.1.1 (a git rev in dig-wallet). dig-constants 0.1.0 shipped an all-zeros PLACEHOLDER DIG L2 genesis challenge with all six AGG_SIG domains derived from it — self-consistent, so no derivation test could see it — and 0.4.0 finalized the real values. dig-wallet's spend-validation ValidationContext therefore described a different chain identity than the rest of the node. Nothing was mis-signed: that call site sets DONT_VALIDATE_SIGNATURE and the signing domain is injected by the caller. Moving dig-clvm to crates.io 0.2 (also closing a §3.6 git-dep) removes the copy. dig-clvm's own requirement was `>=0.1.0, <0.2.0`, so it could never have resolved forward off the placeholder on its own — which is why this sat as long as it did. The suite could not see any of that, before the fix or after it: no source pins the genesis literal, and every runtime check compares `dig_constants::DIG_MAINNET.genesis_challenge()` against itself, which passes identically under the real value and under the placeholder. So the guard goes where the defect is actually decided, in `dependency_tree.rs` against the lock: no dig-constants copy below 0.4.0. Stated as a FLOOR, not `!= "0.1.0"` — 0.2.x and 0.3.x carry the same placeholder, and this is the same property the release gate enforces at 0.4.0 (#178). Proven RED by reverting both dependency edits: 0.1.0 returns to the lock and this is the only test in the workspace that fails. The node's own two crates move 0.4 -> 0.9. Values are identical from 0.4.0 through 0.10.0, so this is value-neutral. 0.9 — not the 0.10.0 tip — is correct: 0.10.0 moved to chia-protocol 0.36.1 / chia-wallet-sdk 0.34 while this workspace builds against 0.26 / 0.30 with dig-gossip's vendored chia-protocol fork, so depending on it links a second chia_protocol and DIG_MAINNET.genesis_challenge() stops type-unifying (measured: 11 errors). Reaching 0.10 is the chia 0.36 migration, not a dependency bump. The three copies that survive (0.4.0/0.5.1/0.8.0/0.9.0) are held down by upstream crates' PUBLISHED ranges. Their full DIG_MAINNET const bodies are IDENTICAL, cost limits included, so what remains is a type-unification nuisance rather than an identity split; collapsing it is a cleanliness cascade, not a correctness fix. Version: minor, for the dig-wallet spend-validation constants changing value (placeholder -> real genesis) — observable behaviour. Workspace 0.101.0 -> 0.102.0; dig-wallet is already 0.13.0 from #189. Refs DIG-Network/dig_ecosystem#2072 Co-Authored-By: Claude <noreply@anthropic.com>
…the chia tip (#199) dig-node's lock held FOUR dig-constants versions in one binary (0.1.0, 0.4.0, 0.5.1, 0.8.0). Only two of the seven holders are dig-node's own crates; the rest are pinned by upstream crates' PUBLISHED ranges, which a consumer cannot edit. The copy that mattered was 0.1.0, reached through dig-clvm 0.1.1 (a git rev in dig-wallet). dig-constants 0.1.0 shipped an all-zeros PLACEHOLDER DIG L2 genesis challenge with all six AGG_SIG domains derived from it — self-consistent, so no derivation test could see it — and 0.4.0 finalized the real values. dig-wallet's spend-validation ValidationContext therefore described a different chain identity than the rest of the node. Nothing was mis-signed: that call site sets DONT_VALIDATE_SIGNATURE and the signing domain is injected by the caller. Moving dig-clvm to crates.io 0.2 (also closing a §3.6 git-dep) removes the copy. dig-clvm's own requirement was `>=0.1.0, <0.2.0`, so it could never have resolved forward off the placeholder on its own — which is why this sat as long as it did. The suite could not see any of that, before the fix or after it: no source pins the genesis literal, and every runtime check compares `dig_constants::DIG_MAINNET.genesis_challenge()` against itself, which passes identically under the real value and under the placeholder. So the guard goes where the defect is actually decided, in `dependency_tree.rs` against the lock: no dig-constants copy below 0.4.0. Stated as a FLOOR, not `!= "0.1.0"` — 0.2.x and 0.3.x carry the same placeholder, and this is the same property the release gate enforces at 0.4.0 (#178). Proven RED by reverting both dependency edits: 0.1.0 returns to the lock and this is the only test in the workspace that fails. The node's own two crates move 0.4 -> 0.9. Values are identical from 0.4.0 through 0.10.0, so this is value-neutral. 0.9 — not the 0.10.0 tip — is correct: 0.10.0 moved to chia-protocol 0.36.1 / chia-wallet-sdk 0.34 while this workspace builds against 0.26 / 0.30 with dig-gossip's vendored chia-protocol fork, so depending on it links a second chia_protocol and DIG_MAINNET.genesis_challenge() stops type-unifying (measured: 11 errors). Reaching 0.10 is the chia 0.36 migration, not a dependency bump. The three copies that survive (0.4.0/0.5.1/0.8.0/0.9.0) are held down by upstream crates' PUBLISHED ranges. Their full DIG_MAINNET const bodies are IDENTICAL, cost limits included, so what remains is a type-unification nuisance rather than an identity split; collapsing it is a cleanliness cascade, not a correctness fix. Version: minor, for the dig-wallet spend-validation constants changing value (placeholder -> real genesis) — observable behaviour. Workspace 0.101.0 -> 0.102.0; dig-wallet is already 0.13.0 from #189. Refs DIG-Network/dig_ecosystem#2072 Co-authored-by: Claude <noreply@anthropic.com>
Refuse a stable tag when any dig-constants copy in the workspace lock predates 0.4.0, the
release that replaced the all-zeros PLACEHOLDER DIG L2 genesis challenge with the real one.
WHAT THIS REPLACES, AND WHY
The gate originally required dig-constants to be SINGLE and equal to the published tip. That
condition is unsatisfiable, so merging it as written would have frozen stable releases:
* "current" is structurally unreachable. dig-constants 0.10.0 moved to chia-protocol 0.36.1 /
chia-wallet-sdk 0.34 while this repo builds on 0.26 / 0.30, including the chia-protocol fork
dig-gossip vendors through [patch.crates-io]. Adopting 0.10 links a second chia_protocol and
produces 11 errors shaped `expected BytesImpl<32>, found chia_protocol::bytes::BytesImpl<32>`.
That recurs on every chia-line jump, for every consumer, forever.
* "single" is not reachable by this repo at all. The copies are pinned by PUBLISHED metadata a
consumer cannot edit (dig-gossip >=0.2,<0.5, dig-nat 0.18.0, digstore-chain ^0.5, dig-download
0.17.0); collapsing them needs a five-repo publish cascade (dig_ecosystem#2072).
So the gate now blocks on the property it actually exists to protect. dig-constants 0.1.0 shipped
an all-zeros placeholder genesis with all six AGG_SIG domains correctly DERIVED from it — a
self-consistent set, and therefore invisible to every test, since each runtime check compares the
constant against itself. It reached production through a dig-clvm git rev into dig-wallet's spend
validator (dig_ecosystem#2316). Everything from 0.4.0 up is value-NEUTRAL here: the full
DIG_MAINNET const body is byte-identical across 0.4.0 / 0.5.1 / 0.8.0 / 0.9.0. A floor catches the
real defect exactly; tip-equality caught it only incidentally.
The rule is stated over the CLASS, not as `!= "0.1.0"` — 0.2.x and 0.3.x carry the same
placeholder. It is worded to match the lock-level assertion #199 already added
(no_dig_constants_copy_predates_the_real_genesis_challenge), so a reader sees one property
enforced at two levels rather than two rules.
Duplication and a newer published release WARN instead of blocking, and the duplication warning
names the package pinning each copy — read out of the same lock the checker reads, never a second
list that could drift. A gate that blocks on a condition only another repo can fix gets bypassed
the first time someone needs a release, and a bypassed gate is worse than a warning: it teaches
its readers that the gate is noise. Promote it once #2072 lands.
Fail-closed is preserved where it is load-bearing: the blocking check reads only the lock, so a
missing lockfile or one with no dig-constants at all is REFUSED rather than vacuously passed. The
crates.io index now feeds only the advisory notice, so an unreadable index degrades to a warning —
blocking a release on a network blip for an informational clause would reintroduce exactly the
unsatisfiability this re-scope removes.
Verification: 21 cases in scripts/tests/, all offline behind the $CURL_BIN seam with fixtures
pinned out of band at 9.x so the seam stays load-bearing. The floor is pinned from BOTH sides
(0.3.9 fails, 0.4.0 passes), and holder attribution is proven to name the holder of the BAD copy
and not the holder of a healthy one. An 11-mutation battery kills every case, including the
`!= "0.1.0"` mutant, both floor off-by-ones, and duplication-blocks-again.
The script is committed mode 100755 AND invoked as `bash scripts/...`, matching how
build-binaries.yml calls check-glibc-floor.sh: a mode-100644 checkout must not fail a release with
a bare "Permission denied".
Co-Authored-By: Claude <noreply@anthropic.com>
f2144e4 to
37818ca
Compare
MichaelTaylor3d
changed the title
Co-Authored-By: Claude <noreply@anthropic.com>
#197) * fix(dig-node-core): bind admit root to ChunkPool content (#2246 #2240) The capsule-admit trust gate (ChainAnchoredModuleVerifier::rejection_reason) recomputed the merkle root from the capsule's own MerkleNodes section and compared it to the committed CurrentRoot. That is hollow: rule 4 already forces CurrentRoot == chain_root, a single untagged leaf's root IS that leaf, and decode_merkle_leaves accepts arbitrary bytes -- so MerkleNodes=[chain_root] with an empty/garbage ChunkPool passed for free, admitting a contentless capsule that was then cached and DHT-announced (phantom-holder / holder-set poisoning). Bind the SERVED content instead: recompute each resource leaf as resource_leaf(concat(ChunkPool ciphertexts)) per KeyTable entry, sort by static_key (the producer's order), fold, and require the root == committed root (== chain_root). Also require the recomputed leaves equal decode_merkle_leaves (MerkleNodes) so served inclusion proofs stay consistent with content. MerkleNodes is no longer trusted for the admit decision. Fail-closed on absent/undecodable KeyTable/ChunkPool or any out-of-range chunk index; an empty store folds to sha256([]). Covers both reshare-admit and cache.pushCapsule land (shared gate; full ChunkPool is materialized before the gate on both paths). No digstore change. 0.100.2->0.100.3. Closes #2246, Closes #2240. Co-Authored-By: Claude <noreply@anthropic.com> * perf(dig-node-core): pre-index ChunkPool to kill admit-gate quadratic scan The #2246 admit gate resolved each KeyTable chunk reference via read_chunk, which is O(global_index) (re-scans the ChunkPool from offset 0 per call), so N references over an M-chunk pool were Theta(N*M). Zero-length chunks add 0 bytes, slipping past the MAX_STORE_BYTES cap, so a pool of M zero-length chunks + one current-gen entry referencing index M-1 N times pinned a core for ~Theta(module^2) per unauthenticated reshare request. Pre-index the pool once into per-chunk byte ranges (O(1) lookup, byte-identical to read_chunk) so content_leaves is O(pool + refs); additionally cap cumulative references at MAX_STORE_BYTES/4 as defense-in-depth for the zero-length case. Co-Authored-By: Claude <noreply@anthropic.com> * chore(release): bump to 0.102.2 after #178 moved main Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
1 participant
Re-scoped. Blocks a stable tag on a 0.4.0
dig-constantsfloor; warns (never blocks) on duplication and on a newer published release.Why the original rule could not merge
The first version refused a stable tag unless
dig-constantswas single and equal to the crates.io tip. It was correct code with a real test suite — and unsatisfiable, so merging it would have frozen stable dig-node releases indefinitely. Both halves, measured:dig-constants0.10.0 moved tochia-protocol0.36.1 /chia-wallet-sdk0.34; this repo builds on 0.26 / 0.30, including thechia-protocolforkdig-gossipvendors via[patch.crates-io]. Adopting 0.10 links a secondchia_protocol— 11 errors shapedexpected BytesImpl<32>, found chia_protocol::bytes::BytesImpl<32>. This recurs on every chia-line jump, for every consumer, forever. A gate keyed to the tip does not protect a property; it periodically bans releasing.dig-gossip>=0.2,<0.5,dig-nat0.18.0,digstore-chain^0.5,dig-download0.17.0. Collapsing them needs five cross-repo publishes (DIG-Network/dig_ecosystem#2072).The rule shipped
dig-constants0.1.0 shipped an all-zeros placeholderDIG_MAINNET_GENESIS_CHALLENGE, with all six AGG_SIG domains correctly derived from the placeholder. Self-consistent, therefore invisible to every test — each runtime check compares the constant against itself. It reached production through adig-clvmgit rev into dig-wallet's spend validator (dig_ecosystem#2316).Stated over the class, not as
!= "0.1.0"— 0.2.x and 0.3.x carry the same placeholder. Worded to match the lock-level assertion #199 already merged (no_dig_constants_copy_predates_the_real_genesis_challenge), so a reader sees one property enforced at two levels, not two rules.Dropping "current" costs nothing: every release from 0.4.0 up is value-neutral here — the full
DIG_MAINNETconst body is byte-identical across 0.4.0 / 0.5.1 / 0.8.0 / 0.9.0. The floor catches the real defect exactly; tip-equality caught it only incidentally.Decision: SINGLE warns, FLOOR blocks
origin/main's lock today carries 0.4.0, 0.5.1, 0.8.0, 0.9.0 — the floor is satisfied, single is not. Blocking on SINGLE would mean this gate lands red on day one, on a condition only another repo can fix. That gate gets bypassed the first time someone needs a release, and a bypassed gate is worse than a warning: it teaches its readers the gate is noise. Every copy at or above the floor agrees on the chain identity, so the safety argument for blocking is not there either.So duplication reports loudly and names the package pinning each copy, making it actionable, and lets the release through. Promote to blocking once #2072 lands. Holder names are derived from the same lock the checker reads (cargo disambiguates
"dig-constants 0.5.1"in a dependency list precisely when several versions resolve), never a hand-maintained second list that could drift from the real graph.RED evidence
Against the real
origin/mainlock with a 0.1.0 copy injected the waydig-clvmactually held it pre-#199:Against the unmodified lock:
EXIT=0, with the four-copy duplication warning namingdig-gossip/dig-nat, digstore-chain/dig-download/dig-clvm, dig-node-core, dig-node-service.Verification
21 cases (
scripts/tests/), fully offline behind the$CURL_BINseam, fixtures pinned out of band at 9.x so the seam stays load-bearing — if the seam were removed the asserted tip would stop matching rather than the cases quietly passing against the network. (The original PR's first run was a false green for exactly that reason: four cases passed only because the live tip happened to equal the fixture tip.)The cases that rule out the nearest wrong gates:
!= "0.1.0"check, which would let the placeholder through under another version number.::error::lines must namedig-clvm(holder of the bad copy) and must not namedig-gossip(holder of a healthy one). A gate printing every consumer satisfies a naive "contains dig-clvm" check while telling the reader nothing;dig-gossipappears elsewhere in the same output, so the assertion is scoped to the error lines.::warning::, not an::error::— an error-typed annotation reads as a failure to every human looking at the run even when the step is green.dig-constantsis refused; "no copy is below the floor" is trivially true of no copies.11-mutation battery, all killed, including:
!= "0.1.0"; floor off-by-one up (0.5.0) and down (0.3.0); duplication blocks again; duplication annotated::error::; attribution drops the bare form; attribution ignores the version; unreadable index blocks again; empty-FOUNDguard removed; yanked filter removed; package name matched by substring instead of equality.Preserved from the original
The
$CURL_BINseam, the exec-bit fix, and fail-closed — with one deliberate narrowing worth calling out. The blocking check now reads only the lock, so the lock is what fails closed: a missing lockfile, or one with nodig-constantsat all, is refused. The crates.io index feeds only the advisory notice, so an unreadable index degrades to a warning. Failing a release on a network blip for a purely informational clause would reintroduce exactly the unsatisfiability this re-scope removes. Both behaviours have a test.The exec bit needed
git update-index --chmod=+x—core.filemodeon Windows silently dropped achmod +x, which is how the original landed at 100644. It is committed 100755 and invoked asbash scripts/..., matching howbuild-binaries.ymlcallscheck-glibc-floor.sh.Blast radius
Additive. Two new files; one new step in
nightly-release.yml'sstablejob, placed before version resolution so a breach means no tag exists to deploy. No Rust source changed —Cargo.lock's only delta is thedig-node-serviceversion, verified against the manifest withcargo metadata --lockedbefore pushing.scripts/tests/*.test.shis already globbed byci.yml, so the suite is gated with no workflow change. Rebased ontomain(d59b4e8, v0.102.0, post-#199); version 0.102.0 to 0.102.1 (patch: CI/docs, no behaviour change to the shipped binary).SPEC.mdsection 11.3a,runbooks/release.md, andDEVELOPMENT_LOG.mdupdated in the same unit.