Skip to content

fix(ci): prune marks, sweep approvals, replace usage period, fold sweep triggers (D-1319, D-1342, D-1317, D-1320) - #531

Merged
acamarata merged 7 commits into
mainfrom
dw10-plugins-12
Oct 29, 2026
Merged

acamarata merged 7 commits into
mainfrom
dw10-plugins-12

Conversation

@acamarata

Copy link
Copy Markdown
Contributor

Four open debt rows, one commit each. All in free/ci.

D-1319: provider marks never pruned

  • Store.ProviderMarkPrune(ctx, name, keyPrefix, olderThan) deletes old marks of one name (empty prefix means every key).
  • The coordinator maintenance loop (hourly, with DedupPrune) applies DefaultMarkRetention: github-reconcile run| and demand| marks 7 days, every other mark of that name (term, enq, redeliver) 30 days. Config.MarkRetention overrides it; an empty slice keeps everything.
  • github-reconcile is a literal in the coordinator because reconcile imports coordinator; a test in reconcile fails if the two names drift.
  • Tests: store prune (prefix, name, other names untouched), coordinator prune with a fake clock, name-drift test.

D-1342: approval voids and trust rows

  • The coordinator maintenance beat (30 s) now calls Store.SweepApprovals with the trust/source void rule, so approval.void audit rows no longer wait for ci runs pending, a facts write or a new approval.
  • DeletePipelines also deletes trust_approvals and trust_run_facts rows of the pruned pipelines (inside the existing batch transaction). A standing approval of a pruned run is deleted without a void row; its approval.grant audit row stays.
  • Tests: tick voids a stale approval on a real store file; prune removes only the pruned run's trust rows.

D-1317: usage ledger keeps stale provider rows

  • Store.ReplaceUsage(provider, fromDay, toDay, rows) deletes the provider's rows in the day range and inserts the new ones in one transaction; economics.ReplaceUsage validates each line like RecordUsage.
  • The GitHub importer replaces the whole period it read (Importer.Replace, default economics.ReplaceUsage; replaces the per-line Record hook). A rename of repo or sku, or a changed billing_account, no longer double counts.
  • Decision: an empty report writes and deletes nothing. A truncated or malformed answer must not erase known usage, because the allowance would then read larger (the unsafe side). A billing account that really went quiet keeps its rows until they leave the month.
  • Tests: repo rename, sku rename and account change each leave the allowance at one count; empty report keeps rows; store replacement scope and refusal.

D-1320: head-scan enqueue does not fold into a finished row

  • Delivery.Reconcile (set by the head scan) becomes IntakeRequest.Reconcile: such a trigger folds into a done row of the same source and semantic key inside the 72 h horizon, whatever the event. Failed rows never absorb it; non-reconcile, non-webhook triggers behave as before.
  • When a reconcile delivery folds into a done row, or is a duplicate of one, the coordinator calls Config.Rereport(row). ci serve wires it to re-send the stored result through the status outbox (report.Sender.Repost, new seq).
  • Re-report rules: only a GitHub, whole-gate run whose stored evidence is bound and has result pass, fail or error. Partial, unbound and cancelled runs never posted a status, so none is sent. The description names the run, never evidence text that a publication profile might filter.
  • Also moved Clock out of coordinator/server.go into clock.go: the D-1319 field pushed it over the 300-line gate.
  • Tests: store folding matrix (same source/key, other source, other key, failed row, horizon, duplicate); coordinator re-report once per fold and duplicate, never for a live or plain delivery; rereportDone states and skips against a real store.

Commands

Run in free/ci with -mod=vendor and a private GOCACHE:

  • go build ./..., go vet on the touched packages: clean.
  • go test ./... (whole module): everything passes except three tests that do not touch this change. internal/logs TestFollowDeliversEachLineWithinASecondAcrossRotation is a timing flake that passed on rerun. internal/legacy TestSecretsEnvLiveKeyGateTrackedValueOnly (a global gitignore hides .environments/prod/.env) and cmd/agent TestServeStdioExitsNonZeroWithE663OnAProdStackMarker fail identically on a clean origin/main.
  • python3 scripts/ci/check-file-size.py: clean.

@acamarata

Copy link
Copy Markdown
Contributor Author

Review fix (cd0370c): TestMaintainSweepsStaleApprovals now runs maintain on the fake clock. It checks that a matching approval stands after the first beat, then changes the head, advances one Beat, and asserts the void row appears. With the c.sweepApprovals(ctx) call removed from coordinator/dedup.go the test fails (timed out waiting for the void). With it restored the test passes, and go vet and go test -mod=vendor ./... pass for free/ci.

@acamarata

Copy link
Copy Markdown
Contributor Author

Review fixes pushed in c2eb91e.

  1. A head-scan (Reconcile) trigger now folds into a done intake row only when that run's result can be re-sent: GitHub source, whole-gate (no named pipeline), newest evidence bound, of the row's own revision, result pass/fail/error. This is checked inside the enqueue query (resendable in store/node_intake.go), so a partial, unbound, cancelled or evidence-less run falls through to a new enqueue instead of suppressing the rebuild for 72 h. New tests (store: 5 cases; coordinator: sweep builds after an unreportable run) fail on the old query and pass now.
  2. Docs and comments in providers_usage.go, usage/keys.go and Scheduling-and-Cost.md now say economics.ReplaceUsage.

Checks: go test -mod=vendor on store and coordinator pass; go vet ./... clean. Remaining failures in the module are unrelated to this change: TestBuildV15EndToEnd (android artifact build), and the cmd/agent and internal/run artifact/lease tests (E712 disk-floor refusal on this machine).

@acamarata

Copy link
Copy Markdown
Contributor Author

Fixed in fb1ce55: the re-send no longer rebuilds the state from evidence. It replays the state of the newest recorded outbox payload for the commit (the status the run was authorized to post, after the release veto and publish gates). A done row with no recorded decision is not folded into, so the sweep builds the head instead. Test: a passing gate with a failed publication re-sends failure; against the base code it re-sent success (also covers no recorded decision, and success recorded for a failed run). cmd, store, coordinator, cmd/agent pass; internal/run fails only on the disk-floor environment check unrelated to this change.

acamarata and others added 7 commits October 29, 2026 13:02
… (D-1342)

The test called sweepApprovals directly, so removing the call from maintain still passed. It now runs maintain on the fake clock and asserts the void row appears after a beat.
… re-sent (D-1320)

A done intake row covers partial, unbound and cancelled runs, which never
post a status. Folding a sweep into one re-sent nothing and held off the
rebuild for 72 h. The fold now requires a whole-gate GitHub row whose newest
evidence is bound, of its own revision, with a pass, fail or error result;
otherwise the sweep enqueues a new run. Docs name economics.ReplaceUsage.
…t (D-1320)

A failed release publication vetoes a passing run in the status outbox only,
so rebuilding the state from evidence re-sent success. The sweep now replays
the state of the newest recorded outbox payload, and a done row with no
recorded decision is not folded into (the head is built instead).
@acamarata

Copy link
Copy Markdown
Contributor Author

Rebased onto origin/main and resolved merge conflicts in free/ci/internal/store/sched_ledger_budget.go (kept main's ledgerValues validation + pruneLedger alongside branch's ReplaceUsage helper) and free/ci/cmd/serve_wire.go (kept both serveNetscope and cfg.Rereport). Module builds with -mod=vendor; force-pushed to dw10-plugins-12.

@acamarata
acamarata merged commit 9c2877a into main Oct 29, 2026
27 of 32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant