Skip to content

ci: merge auto-eligible bot PRs one at a time, behind a circuit breaker, and batch the release - #257

Open
88plug wants to merge 5 commits into
mainfrom
claude/fresh-bot-merge
Open

88plug wants to merge 5 commits into
mainfrom
claude/fresh-bot-merge

Conversation

@88plug

@88plug 88plug commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

Merges auto-eligible bot PRs without a human once every piece of evidence is green at the head, one PR per pass, stops merging whenever main or a release is red, and batches the releases those merges trigger. Part P7 (bot-merge) of the freshness spec.

Stacked on #253 (bot-pr-ci: hack/gate_lib.sh, hack/bot-ci-dispatch, bot-sweep.yml). Until #253 merges, this diff also shows its two commits; the commit of this part is the top one. Merge #253 first.

What changes

hack/bot-merge (new), run hourly as job merge of bot-sweep.yml. One pass:

  • Candidates: open PRs into main from this repository's own autobump/* branches, opened by github-actions[bot], labelled autobump. Never autobump/amend-*, ledger-amendment, Dependabot or fork PRs; never a PR that adds, changes or renames a file under .github/.
  • A candidate merges only when, at its head SHA:
    • hack/pins policy MB SHA says auto, MB being the merge-base of main and SHA (the PR's own diff, so a ledger or release commit on main is not read backwards as the PR's change), or a maintainer approved that SHA (review APPROVED, commit_id = head, user.type User, association OWNER, MEMBER or COLLABORATOR). A maintainer's latest review requesting changes blocks. Labels never change the verdict.
    • ci_status SHA BRANCH is green (the dispatched ci.yml run counts; a held action_required run does not).
    • For every engine in the policy's canary list: the canary.yml runs at the SHA are its pull_request and workflow_dispatch runs whose head is this repository's PR branch (read through the REST API, which names the head repository: canary.yml runs on pull requests into any base, so a fork's PR from the same commit is not this PR's evidence), held (action_required) ones aside. All must be complete and the newest must have succeeded. Each mapped engine is decided by the newest canary-result artifact, made for that SHA, that holds it, and must show verdict: pass, hard: true, exact: true. Per-engine because canary.yml's isa-skip re-dispatch reruns only the skipped engines. A newer run with no readable artifact blocks. A hard pass with exact: false labels the PR canary-drift and needs a maintainer approval. isa-skip, too-big and every other verdict never merge.
    • base_ok holds, git merge-tree merges the SHA onto main without a conflict (base_ok passes over release commits, and each release rewrites K3D_GPU_VERSION in packaging/install.sh, the line above the k3d pin), and rules_stale does not hold. Otherwise it dispatches freshness.yml -f group=<g> (no reason: freshness.yml takes known-issue:<id> only) to regenerate the branch: at most 3 per group per UTC day, counted from today's dispatched freshness runs, and never while a freshness run for that group is queued or running.
    • Circuit breaker: the newest completed ci.yml run on main (push or dispatch) succeeded, and no open ci-alarm issue's key starts with ci-main, ci-missing, release: or red:canary. A breaker that cannot be read blocks every merge and fails the pass.
    • Fewer than MAX_BOT_MERGES_PER_DAY (default 6) bot merges on main today (UTC), unless the policy marks the PR urgent.
  • Order: urgent first, then the oldest. The merge is gh pr merge N --squash --match-head-commit SHA --subject ... --body ...; the body ends with Bot-merge: pr=N group=G urgent=B, which later passes use to count today's merges and to find unreleased ones.
  • After a merge it dispatches ci.yml on main, because a GITHUB_TOKEN merge starts no push workflow. A refused merge prints GitHub's reason and ends the pass.
  • Every candidate keeps one comment (starting <!-- bot-merge -->), edited in place only when its text changes. The comment states what blocks the PR, or the evidence it merged on (policy, CI run, canary run, sweep run).

Release batching, in the same pass after the merge step. It dispatches auto-release.yml -f mode=changed (at most once) when all of these hold:

  • main holds Bot-merge commits that no aur-v* tag contains;
  • main HEAD's ci.yml run succeeded;
  • no auto-release, build-and-push, aur-deploy or packages run is queued or running;
  • no auto-release run has succeeded since the newest of those merges (so a merge that touched no shipped path is not re-dispatched every hour);
  • and one of: an urgent fix is among them; no other bot PR is queued with an allowed verdict and green CI; the newest tag is 6 h old.

hack/bot-ci-dispatch --workflow canary.yml now dispatches with -f engines=auto, and only for heads whose policy (from the merge-base, as above) lists canary engines. It fetches the head from origin to read the policy and never checks it out. bot-sweep.yml ci-fill runs it after the ci.yml step, and its checkout fetches full history for the policy diff.

bot-sweep.yml job merge:

  • needs: ci-fill;
  • permissions contents, pull-requests, actions and issues: write;
  • timeout-minutes: 20;
  • checks out main only, with persist-credentials: false.

It runs with if: !cancelled(), so an error in ci-fill does not stop merging: the merge job reads every piece of evidence itself. rules_stale runs the merged tree's model_opt.py in a subshell without GH_TOKEN.

Interfaces this part consumes from parallel parts

Each was checked against the sibling branch or its working tree.

  • P1 (alarm-release), checked on the branch of ci: alarm on red or missing automation, and harden the release #252:
    • auto-release.yml takes the choice input mode with the value changed;
    • alarm issues are titled [ci-alarm] <key>: <title>;
    • the watchdog raises the keys ci-main, ci-missing, release:<ver> and red:<file>, so a red canary nightly is red:canary.yml.
  • P6 (freshness):
    • hack/pins policy returns the engine list as canary (its in-progress pinlib/policy.py emits "canary": sorted(...)). bot-merge fails closed and comments when that list is missing.
    • freshness.yml is not on a pushed branch yet. This part dispatches it with group only; its in-progress version rejects any reason but known-issue:<id>.
    • Regeneration is counted per group from run titles. A freshness run whose title names no group counts against every group's cap, which errs toward fewer regenerations. So freshness.yml should set a run-name that contains group=<g>.
  • P5 (canary), checked on the branch of ci(canary): runtime and surface canary for engine and node images #258:
    • canary.yml takes the inputs engines and attempt, and uploads an artifact named canary-result holding canary-result.json with {sha, ref, engines: {<id>: {verdict, hard, exact, ...}}}. sha is the PR head for pull_request runs and the branch head for dispatches.
    • Its pull_request trigger makes a held run on every bot PR, and its redispatch job re-dispatches only the isa-skip engines on the same ref (attempt + 1, at most 3). The canary reader handles both.
    • engines=auto selects by the merge-base diff, and the policy's canary list comes from P6. An engine the policy maps but auto did not select shows as "canary result lacks engine E" in the PR's comment, and the PR waits for a human.

Tests

Suite Result
tests/bot_merge_test.sh (new) 67/67
tests/bot_ci_dispatch_test.sh 19/19
tests/gate_merge_test.sh 67/67
tests/ci_wiring_test.sh 97/97
tests/claims_check.sh clean
tests/ledger_drift_check.py 41 files, 81 rows
actionlint 1.7.12 (the digest in hack/ci-images.env) clean
shellcheck v0.11.0 at warning severity clean

tests/bot_merge_test.sh runs the script against a fake GitHub (tests/bot_merge_fake_gh.py) and real git fixture repositories.

  • It has one case for each rule in the task, plus each release-batching condition. On a successful merge, the fake writes the squash commit onto the fixture origin's main.
  • The suite also passes against the stand-in gate_lib.sh written before ci: CI on bot PRs by dispatch, gate acceptance and alarm surfacing #253 existed. The fixtures hold under both readings of ci_status, which differ on a queued run newer than a completed one.
  • Mutation check: 20 mutants of hack/bot-merge each fail at least one case. Examples: drop the head pin; count bot approvals; ignore the approval SHA; disable the breaker, the cap or the regeneration cap; skip the .github check; let isa-skip pass; skip the release busy check.
  • The canary reader has eight cases of its own: a held run newer than the dispatched one, an isa-skip re-dispatch that passes and one that fails, a newer failure superseding a pass, a newest run red on an unmapped engine, a newest run without an artifact, another branch's run at the same SHA, and a null engine entry. Nine mutants of the reader each fail at least one of them: count held runs, accept any branch, read the newest run only, skip artifact-less runs, drop the newest-succeeded check, drop the pending check, read oldest first, drop the SHA check, drop the missing-engine finding.
  • A mutant of the dispatch fix (stderr folded into the policy output) fails the new canary case.
  • The fixture hack/pins judges the diff it is given, as the real one does: a file the BASE..HEAD diff changes and the PR's own diff does not makes the verdict review and the group mixed. Cases for a ledger-only main move (merged, the policy called with the merge-base), a release commit that conflicts with the head (regenerated, no merge attempt, and the next PR merges in the same pass), a fork's pull_request canary run at the same SHA (not counted) next to this PR's own approved one (counted), and an unreadable breaker (exit 1). Seven more mutants each fail at least one case: policy against origin/main (in bot-merge and in the canary dispatch), no conflict check, a reason sent to freshness.yml, no head-repository filter, no head-branch filter, an unreadable breaker that exits 0.

ci.yml gets job bot-merge-tests, placed just before alarms, because #253's wiring test requires alarms to stay the last job. ci_wiring_test.sh gains three checks:

  • the merge job needs ci-fill;
  • it checks out main only and runs hack/bot-merge;
  • the script never runs checkout, switch, worktree or reset.

Merge order

Merge #253 first, then this one only after these are on main: #252 (P1: auto-release.yml input mode, without which every release dispatch fails), #265 (P3: pin-guard, which CI must carry before a bot merge relies on it), #258 (P5: canary.yml) and P6 (hack/pins, freshness.yml). Without hack/pins:

Not done here

The end-to-end acceptance happens after merge, with eyes on each step:

  1. One auto-eligible digest or patch PR is bot-merged.
  2. The dispatched main CI goes green.
  3. The batched auto-release is dispatched.
  4. A new aur-v tag is cut.
  5. The images reach Docker Hub and GHCR.
  6. The watchdog's release-integrity check passes.

Separately, a review-class PR should stay open with its blocking comment.

No repository settings, App permissions, secrets or branch protection change. If the App-token upgrade lands later, the PR author changes from github-actions[bot] to the App's bot account. The candidate filter in hack/bot-merge and in hack/bot-ci-dispatch would then need that login added.

…pen alarms

A PR that GITHUB_TOKEN opens or pushes gets pull_request runs held for
approval (#240 sat with CI action_required), while a workflow_dispatch run
needs none. ci.yml now takes workflow_dispatch, and hack/bot-ci-dispatch runs
it on each same-repo autobump/* branch opened by github-actions[bot] that has
no run at its head, re-running a first-attempt failure once. It keeps one bot
ci.yml run in flight (BOT_CI_MAX_INFLIGHT) and defers while a human PR's CI is
queued, since one run is about 39 jobs against a 20-40 job pool. The three
nightly PR jobs call it after create-pull-request, and the new hourly 'Bot
sweep' workflow (job ci-fill) fills in what they deferred.

hack/autobump-branch-reset generalises the digest job's stale-branch step to
all three PR jobs: a bump branch whose workflow files differ from main's is
deleted before create-pull-request, whose force-push GitHub would refuse.

The merge gate's checks move to hack/gate_lib.sh. ci_status reads ci.yml at a
SHA through a pull_request run or a dispatch run on the PR branch; once the
dispatched run is green, held entries of 'CI' and 'Auto-merge Dependabot PRs'
no longer block. base_ok ignores release-only bot commits on main, so the
three release commits per launcher release no longer force a sync. The gate
prints every open ci-alarm issue and refuses (exit 4) a PR that changes
.github/ or shipped paths while one is not acked; ledger, docs, bench and test
PRs are never refused.

ci.yml gains a last job, alarms, which warns on every open ci-alarm issue and
on a missing, disabled or quiet Watchdog workflow, and never fails.

dependabot-auto-merge.yml is deleted: it ran gh pr merge --auto on an
unprotected main, merged nothing in 500 runs, and held a run on every bot PR.
…er, and batch the release

hack/bot-merge runs as job merge of the hourly bot sweep. Each pass merges at
most one open autobump/* PR that github-actions[bot] opened from this
repository (never autobump/amend-*, never a ledger-amendment, Dependabot or
fork PR, never a PR that touches .github/). A PR merges only when, at its head
SHA: hack/pins policy origin/main SHA says auto, or a maintainer (a User with
OWNER, MEMBER or COLLABORATOR association) approved that SHA and no maintainer's
latest review requests changes; ci_status is green; every canary engine the
policy lists passed hard and exact in the newest canary.yml run's
canary-result artifact for that SHA (exact false labels canary-drift and needs
an approval; isa-skip, too-big and fail never merge); base_ok holds and
ledger-rules.json would stay current, else freshness.yml is dispatched to
regenerate the branch (three times per group per UTC day at most, never while a
freshness run for it is in flight); main's newest completed ci.yml run is green
and no ci-main, ci-missing, release: or red:canary alarm is open; and fewer than
MAX_BOT_MERGES_PER_DAY (6) bot merges landed today, unless the policy marks the
PR urgent. Urgent PRs go first, then the oldest.

The merge is a squash pinned with --match-head-commit, and its body ends with a
"Bot-merge: pr= group= urgent=" line that later passes count. A GITHUB_TOKEN
merge starts no push workflow, so the pass dispatches ci.yml on main after it.
Every candidate keeps one comment that states what blocks it, or the evidence it
merged on, edited in place. A refused merge prints GitHub's reason and ends the
pass.

Release batching follows the merge step: auto-release.yml is dispatched with
mode=changed when main holds Bot-merge commits no aur-v* tag contains, main's
HEAD ci.yml run succeeded, no release workflow is queued or running, no
auto-release run succeeded since the newest of those merges, and an urgent fix
is among them, or no other bot PR is queued with an allowed verdict and green
CI, or the newest tag is 6 h old.

hack/bot-ci-dispatch --workflow canary.yml now dispatches with engines=auto and
only for heads whose policy lists canary engines; ci-fill runs it after the
ci.yml step. The merge job checks out main only, with no persisted credentials,
and runs after ci-fill even when ci-fill had an error, since it reads every
piece of evidence itself.

tests/bot_merge_test.sh runs the script against a fake GitHub
(tests/bot_merge_fake_gh.py) and real fixture repositories, one case per rule
and per release-batching condition (54 cases); 20 mutants of the script each
fail at least one case. ci_wiring_test.sh checks that the merge job needs
ci-fill, checks out main only and runs hack/bot-merge, and that the script never
checks out a ref.
… held or foreign run

The canary evidence came from the newest canary.yml run at the head SHA, whatever
it was. Two runs the canary workflow makes broke that. A pull_request run of a
bot PR is held for approval (action_required, no artifact); when one is newer
than the dispatched run, the PR stayed blocked. And an isa-skip re-dispatch
reruns only the skipped engines, so its artifact lacks the engines the first run
passed, and a multi-engine group could never merge. A dispatched run on another
branch at the same SHA also counted as this PR's evidence.

The runs at the head are now its pull_request runs and its workflow_dispatch runs
on the PR branch, held ones aside, the same set ci_status and bot-ci-dispatch read.
All must be complete and the newest must have succeeded. Each mapped engine is
decided by the newest canary-result artifact for the SHA that holds it, so the
first run's pass and the re-dispatch's pass combine. A newer run with no readable
artifact for the SHA blocks, since what it found is unknown, and an engine entry
that is not a result object is not a verdict. The evidence comment links every
run whose result counted.

Eight new cases cover the held run, the re-dispatch (passing and failing), a newer
failure superseding a pass, a newest run red on an unmapped engine, a newest run
without an artifact, another branch's run and a null entry (62 cases in all). Nine
mutants of the canary reader each fail at least one case.
@88plug
88plug force-pushed the claude/fresh-bot-merge branch from ad571bc to f49ad67 Compare October 30, 2026 00:43
… read only this PR's canary runs

The pins policy now reads merge-base(main, head)..head, in bot-merge and in the
canary dispatch. Against origin/main it read main's later commits backwards: one
ledger commit turned an auto patch into changes-defaults (checked with the
freshness branch's hack/pins), and a stale head's group became "mixed", so it was
neither merged nor regenerated.

base_ok passes over release commits, but those can touch the head's lines: each
release rewrites K3D_GPU_VERSION in packaging/install.sh, on the line above the k3d
pin. GitHub refused such a merge on every pass, and since the refused PR was the
oldest, nothing else merged and no release was batched. A head that git merge-tree
cannot merge onto main is now regenerated like a stale one.

The regeneration dispatch sends group only: freshness.yml takes reason as
known-issue:<id> and fails any other value.

The canary reader lists runs through the REST API and keeps only those whose head
is this repository's PR branch. canary.yml runs on pull requests into any base, so
a fork could open one from the bot's commit and its result counted as this PR's.

A breaker that cannot read main's CI runs or the alarm issues still blocks every
merge, and now also fails the pass, so the sweep goes red instead of staying green
while merging nothing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant