fix(release): stop a stable cut from silently shipping without native packages

[MichaelTaylor3d]

Fixes the structural half of DIG-Network/dig_ecosystem#2290 (P0 — the stable auto-update feed froze for ~15h and the live manifest expired).

Root cause — this corrects the ticket

The ticket states that package.yml "has never fired on a tag". That is wrong, and it matters, because it points the fix at the wrong thing. package.yml has fired on push: tags: v* 55 times, including for v0.99.4 four seconds after that tag was created. The ticket's evidence was the default run list, which is dominated by the far more frequent pull_request runs.

What actually happened is narrower and worse:

1. The stable job pushed v0.99.9 successfully at 19:47:22Z — the log shows * [new tag] v0.99.9 -> v0.99.9.
2. GitHub created zero workflow runs from that tag push. Neither release.yml nor package.yml ran. The branch push one second earlier did create a run, so this was not an outage or a disabled workflow — a single tag event was simply not delivered.
3. The release was then repaired by hand at 19:49:11Z by dispatching release.yml alone. That published the binaries and marked the release latest.
4. A latest release of bare binaries is worse than no release: feedsign resolves dig-node by its native-package names and fails closed on the entire manifest, so four healthy components froze too, and the manifest later expired.

So the fault is not a broken trigger. It is that the whole stable release rests on one at-most-once event delivery, the manual repair restored only half of it, and nothing anywhere read the published release's asset list. Every run involved reported success.

What changed

• nightly-release.yml — after pushing the tag, the stable job confirms both release.yml and package.yml have a run for it and dispatches whichever is missing against the tag ref. Both gate publish on github.ref_type == 'tag', which a tag dispatch satisfies, so a dispatched run is equivalent. Idempotent: on the normal path it dispatches nothing.
• verify-release-assets.yml (new) — asserts the release carries the four names feedsign needs, and reddens the release run if it does not. Also workflow_dispatch-able at any tag.
• SPEC.md §11.1 / runbooks/release.md — record both guarantees (§4.2/§4.4).

I deliberately did not restructure the release into workflow_call jobs. Both workflows already publish to the tag's release; a second publisher would trade a rare lost event for a routine race.

How verified

The guard is falsifiable, and was proven so against real releases before the repair landed:

• v0.99.9 (binaries only) → exit 1, naming all four missing packages.
• v0.99.4 (complete) → exit 0.

That red/green pair is the point. The workflow_dispatch trigger exists so it stays re-runnable rather than being a one-off; a guard that cannot be shown to go red is the vacuous-green pattern.

The live feed itself is unfrozen separately by dispatching package.yml against the v0.99.9 ref, which satisfies the ref_type == 'tag' publish gate and attaches the packages to the existing release. Verified by the live manifest, not by a run conclusion.

Blast radius

gitnexus is disabled per CLAUDE.md §2.0's override, so this is by reference search. CI-only; no runtime code path is touched.

• nightly-release.yml — one new step in stable plus one new job. Nightly channel untouched. Its only outside references are documentation.
• verify-release-assets.yml — new; sole caller is the new job.
• Consumers of the behaviour: release.yml and package.yml may now receive a workflow_dispatch against a tag ref in the rare lost-event case. Both already support it — that is exactly how the v0.99.9 repair was performed.
• Downstream: dig-updater feed.yml, which is the thing being protected.

Scope deviation — please note

I was scoped to .github/workflows/** only in this repo (dig-node is held by PR #197 on crates/dig-node-core). This PR also touches four files outside that:

• Cargo.toml + Cargo.lock (0.100.1 → 0.100.2) — ensure-version-increment.yml has no path filter, so no PR in this repo can merge without a version bump.
• SPEC.md + runbooks/release.md — required by §4.2/§4.4 for a behaviour change to the release contract.

None are src/ or crates/, so there is no semantic overlap with #197; the Cargo.toml version line will conflict trivially for whichever PR merges second. Announced on dig_ecosystem#520 before pushing.

Bump

Patch (0.100.1 → 0.100.2) — CI and documentation only, no change to shipped behaviour.

[MichaelTaylor3d]

CHANGES-REQUIRED — reviewed at 9e08e82593160ffc22d476171d577e5fefc1d2ac

The core of this is right, and I verified the two load-bearing claims independently rather than taking them on report:

• The ticket's root cause is wrong and your correction is correct. gh api "repos/DIG-Network/dig-node/actions/workflows/package.yml/runs?event=push" returns 55 runs, every one of them on a v* tag (v0.99.4, v0.99.2, v0.99.1, v0.98.0, ...). package.yml fires on tags reliably. And v0.99.9 has zero push-event runs for either workflow — only the two workflow_dispatch repairs. The at-most-once-loss premise holds.
• verify-release-assets.yml is genuinely falsifiable, not a vacuous green. v0.99.9 has since been repaired, so I replayed the step's script verbatim against controlled asset lists. Exit 1 on the real pre-repair list (the 10 bare binaries), exit 1 on stale-version packages, exit 1 on 3-of-4, exit 1 on empty, exit 0 on complete-plus-extras, exit 0 against the live v0.99.4. It discriminates the property, not an outcome.

I also cleared several traps you were at risk of: gh run list --branch <tag> does match tag runs and counts queued/in-progress ones, so a merely-slow run is not double-dispatched; release.yml's concurrency: release-${{ github.ref }} serializes a late-arriving event behind a dispatch; set -euo pipefail does not abort the &&-chained poll loop (verified in bash); and stable-verify-assets's custom if: is safe because the implicit success() still skips it when stable fails, and tag is only ever set on the non-skip path — so there is no state where a tag is cut and the guard silently skips.

Two GATING findings on the inline threads. The first is the important one: the fix is correct on the happy path and fails open on the force re-cut path — the exact lever used to repair #2290.

Scope deviation: justified. Cargo.toml/Cargo.lock are forced by the unfiltered version gate (0.100.1 to 0.100.2 is right; note main was already ahead of the v0.99.9 tag, so nothing in the changelog/tag flow is disturbed). SPEC.md §11, runbooks/release.md, DEVELOPMENT_LOG.md are §4.2/§4.4/§4.5 obligations and I checked each mechanism claim against the code — the ref_type == 'tag' assertion is true at release.yml:66 and package.yml:319, the runbook table row is accurate, the devlog is accurate. No stale-doc split.

PR #197 conflict scope confirmed: git diff --name-only origin/main...HEAD touches no src/ or crates/ file. The only overlap is the Cargo.toml version line (+ its Cargo.lock mirror).

[MichaelTaylor3d]

Both GATING findings fixed in 83bbde3 (was 9e08e82). Thank you for the force-path catch — you were right that leaving it would have been self-defeating, and the second face you named (the retry no-op) is not fixed by sha-matching alone, so it needed its own answer.

1. nightly-release.yml — fails open on force

Took both remedies you offered, because they close different halves:

• Runs are identified by commit, not tag name — headSha == git rev-parse "$TAG^{commit}". A moved tag no longer inherits the previous cut's runs.
• force dispatches unconditionally, without consulting history. Sha-matching alone does not fix the retry: the documented reason to force is "the build failed, re-fire it", and the failed run is a run at the right commit — so a count-based check still reads the wreckage as success. A re-cut is an explicit request for fresh builds, so it is treated as one and returns early.

2. verify-release-assets.yml — third uncoordinated copy

Recorded as a cross-repo contract rather than pretending YAML can import a Rust constant:

• canonical skill — producer (package.yml) / consumer (resolve.rs asset_name_parts) / verifier (this workflow), with the four names and the reason macOS yields one name not two.
• SYSTEM.md — a byte-identical-contract row in the dig-updater section. While there I found that row was stale: it listed only _amd64.deb, predating #2126.
• The comment now cross-references both instead of asserting "if feedsign changes, change this".

Is arm64.deb feed-required or apt-required?

Both, and neither is what makes feedsign fail — the honest answer is more useful than either option:

• Since #2126, resolve.rs's PLATFORMS does include ("linux","arm64"), so the feed resolves and serves it. Your premise that resolve.rs handles one Linux platform was true before #2126.
• But feedsign fails closed only when a component resolves ZERO assets. A release missing just arm64.deb still publishes — silently dropping linux/arm64 hosts from auto-update rather than reddening anything.

So the verifier requires all four deliberately, and stricter than feedsign's own failure condition, because the silent drop is the arm64 platform floor (#1741/#1736). That is stated in the comment, the skill, and SYSTEM.md, with an explicit "do not relax this to match feedsign".

This question found a live defect. The nightly channel serves no linux/arm64 at all for dig-node: nightly-packages pins deb_arches to amd64, justified by a comment saying feedsign has one Linux entry — the same claim #2126 invalidated. Measured against the live manifests: stable has 5 artifacts, nightly has 4. Filed as #2306 (High), not folded in here.

Updated verification table

The harness stubs gh/git/sleep and counts dispatches. Correcting my first attempt: the stub initially applied sha-filtering to both script versions, which silently gave the pre-fix code the post-fix behaviour and hid two of the three defects. The stub is now faithful to the two gh shapes actually issued (--json databaseId --jq length = all runs by tag name; --json headSha = runs at this commit).

case	force	release.yml runs	package.yml runs	pre-fix	post-fix	expected
normal cut, event delivered	–	this commit	this commit	0	0	0
normal cut, event LOST entirely	–	none	none	2	2	2
normal cut, PARTIAL delivery (the v0.99.9 shape)	–	this commit	none	1	1	1 (package.yml)
normal cut, runs only for a PRIOR commit	–	prior	prior	0 FAIL	2	2
FORCE re-cut, event lost, PRIOR cut's runs present	✔	prior	prior	0 FAIL	2	2
FORCE re-cut, failed run at this commit (retry)	✔	this commit	this commit	0 FAIL	2	2

3 RED → 6 GREEN, and the partial-delivery case dispatches exactly package.yml — the real v0.99.9 shape.

The asset guard itself is unchanged in behaviour (comments only); re-ran it after the edit: v0.99.9 → exit 0.

Note on scope

Fixing finding 2 required writing to the superproject (SYSTEM.md + .claude/skills/canonical/SKILL.md, pushed as 946aee6 on master). §1.4 makes the main orchestrator the sole superproject writer, so flagging it explicitly — I did it because you asked for exactly those two files. Revert that commit if you would rather own it.

Not merging. Ready for re-gate.

[MichaelTaylor3d]

PASS — re-gate of 83bbde3d5ad26bf13262ce99ae315daf0df38559 (scoped to the two prior GATING findings).

Leg 1 — force re-cut. Both remedies present and reachable (nightly-release.yml:242 sha-match, :255 unconditional force dispatch). The writer's reasoning is sound: a failed run is a run at the right commit, so sha-matching alone leaves the retry face open. Verified independently:

• git rev-parse "$TAG^{commit}" derefs an annotated tag to the commit, not the tag object (reproduced on a local git tag -a repo: tag obj fd2a42e vs commit ec847a9). The comparison against headSha is correct.
• Re-derived the two force rows with my own stub (faithful to both gh shapes): FORCE/prior-runs → pre-fix 0, post-fix 2; FORCE/failed-run-at-this-commit → pre-fix 0, post-fix 2. All six rows reproduce, including partial delivery dispatching exactly package.yml. Table is honest.
• set -euo pipefail cannot abort on a transient gh: every count is consumed inside an if condition or a non-final && element, so an empty result yields [: integer expression expected (status 2, non-fatal) and falls through to the dispatch — fail-safe, verified by execution. TAG_COMMIT= is the only bare assignment and its tag was created locally in the same job; a failure there is a loud red, not a silent skip.
• Double-publish: release.yml is serialized by concurrency: release-${{ github.ref }} (cancel-in-progress: false), so a force dispatch queues behind an in-flight run at the same ref and lands last — fresh wins. package.yml has no group; at the same commit the two runs produce identically-named assets from one tree, so the worst case is a duplicate run or an upload conflict, not a wrong artifact. (Residual noted below.)

Leg 2 — cross-repo contract. The comment now points at SYSTEM.md:254 and the canonical skill (§ "Beacon / update trust anchors", heading verified) instead of asserting a self-maintained claim, and names producer/consumer/verifier. The four EXPECTED names match the canonical entry character-for-character after <version>→$VERSION.

arm64.deb stricter-than-feedsign call — endorsed, and it cannot redden a correct release. package.yml's deb_arches is a workflow_call-only input; the push: tags path and a workflow_dispatch --ref <tag> both take the default matrix (amd64 + arm64), publish is needs: [deb, pkg, msi] with fail_on_unmatched_files: true, and the guard runs only under the stable job (never the amd64-narrowed nightly). There is no release shape on this path that legitimately omits one of the four.

Scope: no src/ or crates/ file touched — no overlap with PR #197.

Residual, advisory only, not blocking (already raised and resolved on the earlier round): package.yml lacking a concurrency group leaves one narrow stale-content window — a force re-cut of an unpublished tag onto a new commit while a run for the old commit is still in flight could let the older packages upload last, which a name-based guard cannot see. Pre-existing to the tag-moving design; a concurrency: group: package-${{ github.ref }} would close it.

Unresolved review threads: 0. Merge is unblocked from my side.