Engine improvements: fix loops, votes, packet pins, gates, and registry - #38
Open
erikreinert wants to merge 199 commits into
Open
erikreinert wants to merge 199 commits into
erikreinert wants to merge 199 commits into
Conversation
erikreinert
force-pushed
the
feature/engine-improvements
branch
2 times, most recently
from
September 1, 2026 23:39
583fae1 to
30eb937
Compare
56 commits hardening the run engine and the CLI around it. Workflow grammar — `when` predicates gain `or` and `contains_any`; steps can declare a severity floor and stall limit; cross-issue linked input form; `max_fix_loops` counting documented. Fix loops — entry scoped to per-trigger clusters; loop parks when the routing verdict repeats or the fix commit didn't move; steering notes carry into the next round; run-scoped batch gate-override covers repeated identical failures (DKT-546). Votes — approved votes route on concerned casts; vote records exposed; retry refused once a proposal has decided; silent seats named with their path; weighted score separated from ballot count; vote-step spend counted against the run budget. Claims and budget — claims can declare a variant-scaled cost; a forced reap no longer spends the attempt budget; routing metadata recorded at claim; run report pairs requested and resolved tier; aggregate builtin gets a routing floor; warning when usage is backfilled against an unclaimed step. Packet pins — pin only the closure a run's workflows reach; check the pin set is closed; validate a packet before taking a lease; repin drops gone files and adds newly required ones; registered workflow source drift detected on disk and shown in run reports. Gates — gate processes get the step's base commit; stub gate results marked with a reason and tracking issue; override-pass refused when it would skip a gate; failed gate named when a step parks; advisory pre-gate verdicts marked; gates named as make targets. Registry and scope — cross-project workflow/schema registration and a drift audit; orphaned registrations surfaced; `--deprecated` hides retired versions and `show NAME` skips them (DKT-616); a widened scope can reach an active run, with a warning when it can't; unresolvable stale targets flagged and waivable. CLI fixes — `events follow --tail` starts at the newest N; `events list` finds a run in any project; `next --run` returns the full ready set; dispatch close does backfill and verify in one step; trust changes refuse a missing actor/cwd; loop redirects no longer bind an unrelated step's output; operator pause survives a step record; in-repo docket skill copy retired.
erikreinert
force-pushed
the
feature/engine-improvements
branch
from
September 1, 2026 23:47
30eb937 to
10b6f6a
Compare
…ainst built dist Removes the standalone test/qa jobs — go build/test already run inside build-shell/build via the vorpal pipeline, and qa.sh now runs against the actual docket-dist-* artifact from the build job instead of a throwaway `go build` binary, so e2e exercises what actually ships. repo-gates is renamed to gates to match its new position as the first, fast-fail job; secret-scan.sh's comment is updated to match.
…breaks - a tree comparison on a moving branch tip read any later commit on the same paths as a divergence, so clean cherry-pick integrations warned "integration diverged" on every review row - a disproved ancestry now asks first whether HEAD carries the target's patch, via git cherry refined by zero-context patch-id comparison, including a squashed integration's combined diff - the tree probe remains as a fallback and may only acquit; the advisory says "diverged" only when the patch probe measured the patch missing
…olve - step resolve --worktree, with override-pass or rerun-gates, recomputes the recorded issue.diff and target sha from the named checkout and records it as a new artifact superseding the step's previous one - no resolution re-recorded the diff before, so after an out-of-band patch every downstream review packet still rendered the pre-patch commit and a full fix round ran to report nothing needed fixing - the re-pin is event-logged as issue-diff-repinned with both shas and rides the resolve output as issue_diff_repin - an empty diff over a recorded change is refused rather than pinned - extract computeIssueDiff from the routing stage so the re-pin and step record share one diff implementation - fold the resolve variants into ResolveStepWith on a ResolveOptions struct
- a type-filtered listing of eleven documents was 83KB of JSON because every row carried its full body - rows now carry id, type, status, title, author, timestamps, and body_bytes; bodies stay in doc show - --with-body restores the previous full-document output for bulk callers
- issue list, next, plan and board now drop description from each JSON row and carry description_bytes in its place - a filtered listing of a few dozen described issues ran to tens of kilobytes to answer a question that needs ids, titles and status - --with-body restores the previous full-row shape on all four verbs; issue show is unchanged
- the previous-round pass matched on emitted kind alone, so a re-synthesis received the prior round's whole raw review fanout beside the digest of it - a re-entry step now reads only its own name at the prior ordinal plus the standing set the loop body declares it acted on - one security-change round-2 synthesis carried a 263KB packet of four raw judge payloads it was also carrying as the reconciled aggregate
- a vote panel is seated from step show and only probes the context bundle when the row names a target - the bundle carried target_sha and target_worktree but the step view did not, so every panel read its own HEAD instead of the judged tree - the view now resolves the same pair in the same transaction, emitting both keys only when the step consumes issue.diff and a round record exists - steps with no target emit neither key rather than an empty or approximated value
(cherry picked from commit f01b09faa828b3229e413832023f7ffd261c726f)
…ng gap - derive the reserved-label set (selected by no workflow, declined by every workflow) from the live corpus instead of hand-listing it, so the sweep tracks corpus changes automatically - fail loudly, naming the label, when one is declined by only some workflows and selected by none (an orphaned exclusion) instead of silently reporting it as an unrouted gap - apply the non-vacuity floor to the routable vocabulary only, after reserved labels are set aside, across all three sweep tests
- ParseRelationType no longer accepts related_to/related-to; it backs the wire format and the workflow-spec vocabulary, both of which are meant to stay exactly the documented set - move the typo tolerance to a new parseRelationTypeArg helper shared by link add and link remove, where it only ever affects a human typing at a terminal - add tests pinning the parser, the JSON decode, and the CLI helper so a future alias can't widen the wire format silently again
- new run_notes table (schema v26): append-only, run-scoped, capped at 16 KiB per note, dead with the run - docket run note add RUN-N --text|--file records one; refuses an empty or oversized note, an unknown run, or a done/abandoned run - docket run note list RUN-N renders them back, human and --json - context assembly reads notes as a sixth source alongside the existing five, so step context/claim/render all carry them - the default packet template renders each note verbatim as its own section right after the request body, so an operator ruling made before dispatch reaches every step of the run instead of getting rediscovered per step - documented in the engine spec and reliability delta
…g it - a gap file's leading header block (Severity/Priority/Kind/Labels lines right after the title, ending at the first line that isn't one) now sets the filed issue's priority, kind, and labels instead of every gap landing at priority none, kind task, no labels - Severity maps to priority (blocker->critical, high->high, medium->medium, else none); an explicit Priority line wins either way it's ordered against Severity - unrecognized keys are skipped without ending the block, so a Home: line composes with the header; invalid values fall back to the default rather than refusing the whole issue - the body is stored verbatim, header included - documented in step complete/record's help text
A gate step that ends up skipped (routing resolved elsewhere, an on_fail=skip rejection, a false `when`, or a quorum miss) used to leave its downstream `after` successors ready to run anyway, since a skipped predecessor still counts as terminal. Successors had no way to say "only run me if that predecessor actually fired." Add a step-level `after_fired` list: naming a predecessor there means this step is skipped in the same transaction the moment that predecessor is skipped, cascading transitively through the graph. Every `after_fired` entry must also appear in `after`, so the step still waits for the predecessor to reach a terminal state before the skip (or the run) is decided.
- the claim wrote step_inputs but no read verb used it, so step context and step show re-resolved a handed-out step over the run's current artifacts and reported inputs and a target sha the worker never saw - a claimed step now reads back the bindings its claim recorded; a pending or never-claimed step still resolves live - a re-claim clears the last attempt's bindings before recording its own - the claim records its bindings before pre-gates run so a read that lands mid-claim finds them - step context --live keeps the current-state resolution reachable
…e with it - golangci-lint and staticcheck cache issues by package content but store the absolute source path and re-open it to find a suppressing nolint comment - a pre-gate reconstruction is deleted within the minute, so a later run over the same content replayed a stale entry, could not find the nolint, and failed a clean tree - point GOLANGCI_LINT_CACHE and STATICCHECK_CACHE at a scratch sibling of the reconstruction that is removed with it; durable trees keep their shared caches
…ays blocked - Expansion gated on every issue in every earlier topological level, so an unrelated phase-1 sibling left at todo by abandon-issue stalled a whole chain whose only real predecessor was done - An issue now expands once its own depends_on predecessors are done, matching how the planner reads the same graph - Activation reports every issue it left unexpanded with the predecessors holding it, on the summary line and as blocked_issues in the JSON envelope
…'s domain - a workflow may declare `[match] domain_paths`; it binds nothing and is read only by an activation lint - an issue whose whole scope lies inside another workflow's domain, lacking only that workflow's labels, is named in the activation report - exactly-one-match refuses zero and several matches but cannot see one wrong match; a mis-labelled TUI test issue bound the baseline pipeline and silently lost the UI gates - warns and never refuses, stays quiet on partial overlap, kind mismatch, or a firing unless_labels
- a gate needing an earlier step's artifact had to list the issue's steps and pick itself out by an instance-name convention, which broke silently - exports the step reference on both completion gates and pre-gates, and leaves the variable absent rather than STEP-0 when no step is known
…s reaped - failed_attempts/reaped_claims tally every ending a step ever had and go ambiguous once a history mixes both; a router needs only the last one - a session killed mid-wave had ten leases reaped and an on_failure policy escalated all ten as if they had failed - schema v27 adds steps.last_claim_end, stamped by the fail and reap paths, surfaced as prior_attempt_end on next, dispatch open, step show and list - next.go also makes IssueStepList return an empty slice instead of nil so a stepless issue serializes "steps": [] rather than null
- a nil slice marshals as "steps": null and forces every JSON consumer to guard against it
`docket trust probe [--run RUN-N] --json` runs every non-action trust-roster entry once against a throwaway worktree of clean HEAD, using the same resolve/spawn/timeout path gates use at record time, and removes the worktree unconditionally including on interrupt. `docket trust list --json=v2` items now carry `class: "gate"|"action"`, resolved from the workflow corpus's own `action = "<name>"` declarations, so a caller can tell the two apart without grepping workflow TOMLs itself. This replaces the gate-probe.js workflow script's equivalent, which spent one agent per gate and could not honor per-entry timeouts.
`docket next --run` executor rows now carry model/effort/variant resolved from the run's pinned policy.toml, and vote rows carry the same per voter. Resolution walks the attempt- and round-based escalation ladder, redirects around [security]-forbidden models via [escalation.fallback], and clamps to [security].ceiling on a sensitive row -- a direct port of wave.js's and tribunal.js's resolve()/resolveSeat(), verified against tests/wave-escalation-walk.test.sh's own table and structural assertions. A run with no pinned policy.toml leaves every row exactly as it was before this feature existed. policy.toml was already pinned per run (packet_closure.go's policyPinRef); this reads those bytes back instead of leaving routing to a second, independently-maintained parser downstream.
`dispatch close` now verifies, for every write-class step recorded in the run, that its own recorded commit is an ancestor of the shared branch's HEAD or patch-equivalent to a commit on it (a cherry-pick mints a new sha for identical content), refusing CONFLICT with the step, sha, and worktree otherwise. A cherry error counts as unintegrated rather than being assumed equivalent. `--skip-integration-check REASON` is the operator override, and the close event records integration: verified|skipped plus the checked shas either way. Reuses the same ancestor/patch-equivalence probes the stale-target advisory already shells out with, rather than a second implementation of `git cherry` plumbing.
- Replaces the six read-only probes a conductor cleared by hand before the first dispatch of an attach with one read-only verb - Every check runs; a missing --run or --source reads as SKIP and leaves the report unclean rather than silently checking fewer things - Straggler worktrees are reported but never move the clean bit
- config set refuses vote.rule.<name>.roster and .weighting by name, pointing at the vote-step field that replaced each; stored rows are ignored - SetConfigTx writes inside a caller's transaction and returns the prior value at that scope - A write to any vote.rule key records a config-changed event in the same transaction, carrying key, scope, prior, new value and writer - Attribute the issue-body-refreshed kind so the closed-set tests pass
- A package failing early in a long go test run lost its failing test names to the fixed 20-line tail, so the verifier could not tell which test failed
- validateOnExhausted emits V41 five ways, but RuleIDs and the test table both omitted it, so the set-equality check could not see the gap
- Cluster rounds and the issue loop count agree unless another cluster spent an ordinal first, so a second case runs one to tell them apart
- The span tripwire still pinned v34 after the loop-history columns bumped the schema, failing the db suite; the amendment records why v35 was needed
- Treat `--- `/`+++ ` as file headers only between a `diff --git` line and its first hunk; inside a hunk they are content and count toward diff.lines, so a removed `-- banner` no longer under-sizes the change - Derive diff.empty from the ledger's own record-or-drop test over the in-scope portion, so rename-, mode-, and binary-only diffs the ledger keeps as real changes read as non-empty and route review - When a retry computes an empty diff and the guard keeps the earlier recorded change, measure diff.* from that recorded body instead of the empty one, so a retry cannot skip review over an unreviewed change
… time - Move the diff.lines, diff.files, and diff.empty names into the workflow package so validator and evaluator share one spelling - V45 refuses a diff.* predicate on a step that holds no tree (action, type, fanout, or holds_tree = false), where no measurement exists and the predicate would silently never match - V46 refuses a non-integer literal under an ordered operator on diff.lines or diff.files, moving the refusal from a parked run to the author's terminal - Exempt exactly the three reserved names from V21a on payload-declaring steps: no schema can declare them, and the refusal told authors to add diff.lines to a payload schema; lookalikes such as diff.bogus are still refused
- Add the reserved diff.* family to the payloads and thresholds TDD: measurement, absent-record rule, the V45/V46 lint, and the V21a exemption - Add the on_exhausted row to the engine spec's step key table, the three loop-history facts a refused entry writes, and the step row wire shape that carries them
…peat - The byte-identical guard compared the newest recorded body against the computed one, and "no record yet" read as an empty body, so a first empty diff matched it and was dropped despite the stated rule that a genuine first "nothing changed" records - Report whether a record exists separately from its body and suppress only a repeat of an actual record
- V46 now covers == and != on diff.lines and diff.files, since the engine parses a count literal as an integer whatever the operator - V47 refuses an ordered operator or a non-boolean literal on diff.empty, so no registered definition reaches a record-time refusal
- Describe the three reserved fields and their register-time rules in the engine spec threshold grammar - Record V47, V46 under equality, and the first-empty-diff record rule in the payloads and thresholds TDD
- A fanout step expands to executor rows, and each sibling that holds the tree measures and routes on its own recorded diff, so refusing the predicate there was stricter than the engine - Refuse it only where no measurement exists: action and type steps, and executor or fanout steps declaring holds_tree = false
… parks - The attempts-exhausted park stored the worker's fail note verbatim, so an empty note parked a step with no reason and `step show` omitted the field as if the step had never parked; a refused fix loop on a rejection stored `<reject note>; <bound reason>`. - Every reader documents park_reason as the engine's own text. Both sites now write `attempts-exhausted: <n> of <max>` or `fix-loop-exhausted: <bound reason>`; a spent triage panel's sentence is prefixed with the class. - The caller's note stays where it was: the routing record, the step-failed or step-rejected event, and the trail comment. - SetStepRoutingWithParkReasonTx takes the park text apart from the routing reason; SetStepRoutingTx delegates with the two equal, so the engine-text sites are unchanged.
- A step whose triage panel had already ruled lost the worker's fail note when its budget ran out again: the panel's sentence replaced it in the routing record, the step-failed event and the trail. It is now appended, and the park reason carries the budget count ahead of it. - A park whose reason is blank, such as a failed gate that printed nothing, stored an empty park_reason that `step show` omits. The write now falls back to the park class word. - Tests pin the park_reason wire key in raw run-report and step-show JSON, the exact composed formats, and the spent-panel path.
- `docket schema deprecate <name>@<version>` retires a schema version the way `workflow deprecate` retires a workflow: a timestamp on the row, never a delete. Runs that pinned it keep validating against it; new `payload` references are refused by register, lint, and activation's auto-registration, naming the schema and the restore remedy. - Refuses a version a workflow still in service names as payload, listing the referencers, with no override; refuses the builtin. `--restore`, `--project`, and `--all-projects` mirror the workflow verb, with `in-use` as its own per-project outcome. - `schema list` hides retired versions unless `--deprecated`; `schema show NAME` resolves the highest version still in service; `registry audit` reports a fully retired orphan schema as retired. - Store migration v36 adds `schemas.deprecated_at_ms`, NULL meaning in service, with the column-probing rewind guard and no back-fill. - `workflow list` and `workflow lint` take `--project <ref>` with the write verbs' ref forms and error, so another project's registry can be listed and linted against from any checkout, including a project whose checkout is missing from this machine.
- Reports each project's verdict (new, unchanged, conflict, invalid) in the per-project report register --all-projects writes, so a store-wide sweep can be checked before it runs
- A repo-relative path under an instance-config root was stored verbatim, so verify-pins, pin show, and resume could never resolve it - Files under a root now pin by their config-relative ref from any cwd - Absolute paths outside every root keep their form; relative ones are refused with VALIDATION_ERROR before anything is written - Template pin lookup matches the normalized ref as well as the typed path
- The id remap probed the whole store, so re-importing a project's own export remapped every row and failed on the first label mapping's foreign key - An id held by this project's own row is now the same row and is skipped; only ids held by another project are remapped - Labels match by project and name, so mappings land on the existing label - Issue-label and issue-file mappings whose rows are missing are skipped instead of aborting the import
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Workflow grammar —
whenpredicates gainorandcontains_any; steps can declare a severity floor and stall limit; cross-issue linked input form;max_fix_loopscounting documented.Fix loops — entry scoped to per-trigger clusters; loop parks when the routing verdict repeats or the fix commit didn't move; steering notes carry into the next round; run-scoped batch gate-override covers repeated identical failures (DKT-546).
Votes — approved votes route on concerned casts; vote records exposed; retry refused once a proposal has decided; silent seats named with their path; weighted score separated from ballot count; vote-step spend counted against the run budget.
Claims and budget — claims can declare a variant-scaled cost; a forced reap no longer spends the attempt budget; routing metadata recorded at claim; run report pairs requested and resolved tier; aggregate builtin gets a routing floor; warning when usage is backfilled against an unclaimed step.
Packet pins — pin only the closure a run's workflows reach; check the pin set is closed; validate a packet before taking a lease; repin drops gone files and adds newly required ones; registered workflow source drift detected on disk and shown in run reports.
Gates — gate processes get the step's base commit; stub gate results marked with a reason and tracking issue; override-pass refused when it would skip a gate; failed gate named when a step parks; advisory pre-gate verdicts marked; gates named as make targets.
Registry and scope — cross-project workflow/schema registration and a drift audit; orphaned registrations surfaced;
--deprecatedhides retired versions andshow NAMEskips them (DKT-616); a widened scope can reach an active run, with a warning when it can't; unresolvable stale targets flagged and waivable.CLI fixes —
events follow --tailstarts at the newest N;events listfinds a run in any project;next --runreturns the full ready set; dispatch close does backfill and verify in one step; trust changes refuse a missing actor/cwd; loop redirects no longer bind an unrelated step's output; operator pause survives a step record; in-repo docket skill copy retired.