Recipes are prompts: shrink orchestrator context via hook/server-delivered routing and a compiled skeleton rendering #4482

Context

Surfaced during the #4478 review discussion: that PR (like nearly every PR touching pipeline behavior) grew the recipes — scope-verdict routing on 8 step sites across 4 recipes plus three new gate/loop steps. The #4478 changes stay as merged; this ticket is about the pattern, not that PR.

The recipe is a prompt. The interpreter is the orchestrator LLM. The YAML format exists for build-time programmatic validation — at runtime the consumer reads it as natural language and does not need the syntax. Contributors (especially AI-assisted ones) treat the recipe as a program and steadily balloon it.

Reframed motivation (maintainer): adherence is NOT the pain point. Orchestrators almost never step out of bounds in practice — each step's required inputs come from the previous step's outputs, so the dataflow makes it structurally hard to go wrong; observed mistakes have been minor and non-critical. The problem is context bloat and token cost: the orchestrator is the first thing that reads the recipe, the recipes are already ~110KB, and every PR adds more.

Measured composition (implementation.yaml, 113,644 bytes)

Step census: 28 of 134 steps are run_skill (LLM-session dispatches needing judgment). 46 run_python + 10 run_cmd are deterministic decision/gate steps (~33% of file bytes); most of the remainder is deterministic MCP dispatch (push_to_remote, wait_for_ci, enqueue_pr, …). The orchestrator personally reads, interprets, and executes all of it.

Delivery today: below the measured 195,000-byte exemption ceiling the whole raw YAML is served inline; above it, ENVELOPE mode makes the orchestrator pull flow_records + per-step sections via get_recipe_section. The *.json twins are never agent-visible. Hidden ingredients are excluded from the ingredients table but inlined as literals into step bodies.

External evidence (key findings)

1. Length, not format, is the dominant failure axis. Procedural instruction-following accuracy falls from 63% at 5 steps to 20% at 95 steps (arXiv:2605.00817). Format swaps (YAML/JSON/TOON) move tokens ±20% and are shape-dependent; tabular compaction wins 30–60% only on uniform records.
2. Structure helps compliance — numbered/keyed structure beats prose for procedures even at higher token cost; markdown-like register is the models' native distribution. A hyper-dense DSL is the wrong target for imperative content.
3. Lossy semantic compression (LLMLingua-family) is disqualified for instructions — validated only for reference/context material; instructions degrade unevenly.
4. Pure just-in-time delivery has a measured cost: step-by-step agents without lookahead hit repetitive-failure loops ~5x more often (66.9% vs 13.9%) at 2–4.7x tokens vs full-horizon planning (arXiv:2605.08477), with weaker models hurt most.
5. The evidence-backed middle path (HiPlan, arXiv:2508.19076; also Swarm/LangGraph/Temporal/Step Functions in production): compact global skeleton upfront (preserves lookahead) + full step detail delivered just-in-time by an external engine that holds the state machine.

What already exists in this codebase

The building blocks are live, built for adjacent purposes:

The single missing component: nothing evaluates on_result/on_success/on_failure server-side. No routing evaluator exists anywhere — routing is 100% LLM-read prose. The conditions are literal comparisons over an already-parsed schema; the evaluator is a small pure function.

Constraint notes from the hook side: hook scripts are stdlib-only standalone executables seeing only tool_name/tool_input/session_id (#3758); hook-injected text bypasses _response_budget enforcement (design gap); no SessionEnd event (#677); documented parallel-tool-call race.

Design direction (maintainer input)

1. Routing instructions can be REMOVED from the recipe text the orchestrator reads. The recipe keeps the instruction content — the major steps and what each does — but on_success/on_failure/on_result-style routing is mechanical and can be delivered by automation at the moment it is needed, telling the orchestrator the way.
2. Two mechanisms observed to improve success:
   • The hook/server instructs the orchestrator with the routing information at step completion.
   • It can go further and deterministically direct some steps itself, so the orchestrator is not used at all for them (the 56 deterministic run_python/run_cmd steps are the candidate population).
3. Instruction-echo compliance pattern: requiring the orchestrator to repeat the directive before acting ("You are required to repeat this message" → the model says "Going to step 7 …") measurably improves follow-through. Any delivered routing directive should carry a required echo line as part of its format.
4. YAML stays the single authoritative source (per Instruction rules are duplicated across surfaces with no ownership layer — copies drift independently #4372's ownership principle); anything agent-facing is derived by tested, deterministic functions — no second hand-maintained artifact.

Proposed layers (for discussion, not commitment)

• Layer 1 — server-evaluated next_step routing hints (small, additive, flagged): pure evaluate_step_routing(step, result) over active_recipe_steps; run_skill/run_python handlers append next_step: {name, matched_condition, dispatch_line, echo_line} beside the existing pipeline_tracker field. Delivered by the MCP handler, not a hook script — handlers are typed, tested, and budget-checked, while hook-injected text escapes the byte budget. Hooks stay enforcement-only (Unified hook checkpoint framework: declarative tool-call enforcement for L1 skill sessions #3758's lane). Measurement: orchestrator token totals, get_recipe_section read counts, and hint-vs-LLM routing agreement (divergences are recipe bugs or evaluator bugs — both valuable). Exit criteria before any Layer 2 work.
• Layer 2 — compiled skeleton rendering: deterministic renderer emitting the orchestrator-facing form — one line per step (intent + tool + routing arrows), notes dropped by default, markdown-ish imperative content, tabular where uniform. With routing removed per the maintainer direction, measured composition says 40%+ reduction before touching semantics; a skeleton-with-detail-on-demand form goes far lower (per-step detail already has a pull path, and Layer 1 adds a push path).
• Layer 3 — deterministic step direction: with the evaluator in hand, chains of mechanical steps are auto-executed server-side and only the landing step is reported — the orchestrator handles only the ~28 judgment steps. Moves execution authority; needs its own design round.
• Independent quick win: note discipline — a recipe-lint rule capping note: fields to operational content (24% of the file today), with rationale moved to issues/ADRs.

Open questions for human review

1. Delivery authority: MCP handler payload (typed/tested/budgeted) vs literal hook script (additionalContext/updatedToolOutput)? Evidence favors the handler; the hook channel remains for enforcement.
2. Echo format: what exactly must the orchestrator repeat, and does the delivered directive replace the recipe's routing text entirely (Layer 2) or coexist during transition?
3. Which steps qualify for full deterministic direction (Layer 3), and what audit trail replaces the orchestrator's narration for them?
4. Relationship to Unified hook checkpoint framework: declarative tool-call enforcement for L1 skill sessions #3758 (unified checkpoint framework): shared session-keyed state, division of enforcement vs delivery.
5. Measurement plan and success thresholds for the Layer 1 experiment.
6. Does routing removal from recipe text change validate_recipe/semantic-rule coverage (rules currently lint the routing that would move out of the agent-visible rendering — they keep linting the YAML source, which remains authoritative)?

Related

• Unified hook checkpoint framework: declarative tool-call enforcement for L1 skill sessions #3758 — unified hook checkpoint framework (enforcement-side constellation; dormant)
• Instruction rules are duplicated across surfaces with no ownership layer — copies drift independently #4372 — instruction ownership layer (governs derived renderings)
• Pipeline step completion tracker: hook-driven step validation and dependency enforcement #3266, Rectify: PreToolUse Recipe-Read Guard — Structural Enforcement Against Raw Recipe File Reads #3685, Hook-based recipe reload guard — block duplicate recipe loads within the same session #3911 — step-tracker / recipe-read hook guards
• Add SessionEnd hook for terminal cleanup as defense-in-depth #677 — no SessionEnd hook; Replace skill_guard flag files with transcript-as-oracle + write-through cache #2561 — transcript-as-oracle alternative to flag files
• Re-architect open_kitchen/load_recipe delivery: bounded envelope + pull access #4304 / Codex order aborts when required recipe skeleton exceeds conservative 10KB envelope #4362 / open_kitchen response exceeds Claude Code MCP output cap — step graph truncated from orchestrator context, causing out-of-order step execution (review_approach before rectify) #4253 (closed) — recipe delivery envelope/pagination work this builds on
• Assess repository-guidance redesign for Codex and Claude #4424 — repository-guidance redesign (adjacent context-reduction effort)
• Codex backend: scope-discipline injection, effort remap, and diff-size gates #4476 / PR feat: Codex scope discipline — digest injection, effort remap, diff-size gate (#4476) #4478 — the scope-discipline change whose recipe growth triggered this discussion

Source

Exploration report: .autoskillit/temp/recipe_context_reduction_exploration_2026-08-07.md (2026-08-07; measurements, agent traces, and research citations inlined above).

Sub-tickets

Investigation data (2026-08-13)

• Full payload: ~255KB (255,000 Codex tokens at 1:1) — exceeds all inline delivery ceilings
• Routing skeleton (step names + types + routing edges only): 28,351 bytes — fits all backends
• Flow records (~104KB) are 100% redundant — agent never reads them for routing
• On-demand step loading via get_recipe_section(step_name) already exists and works
• note fields: 30.4KB (27%), with params: 27.5KB (26%) — both step-local, JIT-deliverable
• 9 structural-clone families (55 steps, 39.7KB, 36%) — consolidation deferred to separate ticket
• Investigation report: .autoskillit/temp/investigate/investigation_recipe_delivery_bound_spill_2026-08-13_143500.md

Commissioning observation — 2026-10-16, MiniMax-M3

PR5255 implementation run38960061259, tested head 20ee39237e8e89a8d21f403c3a825273fcac6bf7, provides empirical context for this issue's additive routing-hint proposal. This is a bounded observation from one failed run, not a change to the maintainer's chosen scope or evidence that all orchestrators generally fail adherence.

Retained root transcript out/transcripts/-workspace-sandbox/5a5cdc35-9fdf-4480-92c2-78d5fbcb8592.jsonl: successful test record193 supplies I4 with five reset counters before commit_guard/main_repo_guard. The carrier identifies segmentI4 and source_step=test, but no selected entry target. The model calls those counters199/203/207/211/215. Later carrier216 supplies I8 beginning with remediate and containing its on_success=plan route, yet the model proceeds to prepare_pr222 and compose_pr295. Required integration/audit is absent. The later deviation is direct noncompliance with a delivered route.

Adversarial re-review corrected an earlier assumption: the actual test->commit_guard YAML body was not previously delivered. check_changes result188 has an empty success carrier because the unconditional on_result fallback is classified outside success delivery. That deterministic omission is tracked separately in #5286 as repair of existing automatic delivery from #4578. The full recipe remained pullable through get_recipe_section; no such pull occurred before test. Do not conflate an available pull path with already delivered content.

The optional graph-derived entry/next_step hint already proposed here could be evaluated against this mixed-segment case. Its absence is not independently a breach of a promised carrier field, and the transcript does not establish that adding it alone would prevent the full failed run. Keep the existing YAML graph authoritative and use measured model evaluation before claiming adherence improvement. Review and strict ownership gates correctly failed closed and must remain intact.

Two independent reviewers examined the candidate body and reran their source/transcript checks after context compaction. The parent personally verified their citations and reproduced the narrower fallback omission with production functions without a model call. No duplicate enhancement ticket was filed.

Related observation from #5286: mixed I4 entry hint

In commissioning run 38960061259, the test success carrier delivered I4 bodies beginning with the reset counters, ahead of commit_guard. Track this mixed entry-hint observation here alongside recipe presentation work. The #5286 delivery fix makes manual pull closures include every reached segment; it does not change inline body order. Post-integration commissioning remains required with the review and strict-cleanup gates intact.

Additional adherence evidence — 2026-10-17, MiniMax-M3

Implementation39082343853 and 39082347591 at28448805b provide bounded model-adherence evidence after #5286's delivery repair. In run1, actual check_changes result195 now delivers I3/test with its commit_guard route. The root still invokes reset counters after test before later recovering to guards/merge/auditGO. It asks a merge to target an implementation branch rather than the canonical feature branch, publishes that wrong base on precondition failure, then later merges/pushes correctly. It skips the mandatory review path after capture_review_repository. The previous assumption that test routing was absent is not applicable to this new run.

Run2 has no auditGO, compose or review worker. Root run_cmd335 creates PR18 directly while labeled step_name=escalate_stop, although that source step is action:stop. Both roots report implementation_complete despite these departures and PR metadata failures. The strict harness correctly rejects their results. Claim/body input loss is also recorded in existing #45. One actual conditional-write evidence mismatch is separately filed as #5289.

These are observed deviations from the delivered graph and argument contract, not proof of a new routing-graph defect or proof that an additive next_step hint alone would fix them. Keep YAML authoritative, preserve gates and measure any proposed adherence change on these retained cases. This adds empirical context to the earlier maintainer motivation discussion without expanding this issue's acceptance scope. Parent inspected source/transcripts and joined all evidence/review agents. No duplicate hint ticket was filed.

Recommissioning evidence — 2026-10-18 (e529c48)

Tested feature head e529c48a8d6576ca7bbff29a51c0d7ee12caa167, integrated develop a93c20037bddef0c534af3f7da44ca4310733b33 including #5290/#5289.

Both exact-head attempts 39177297225 / 39177304001 failed before prepare-pr. Root retry calls omitted the declared audit attempt ID despite the recipe and prior response providing it (#45). Run1 also repeatedly copied raw AUTOSKILLIT_TEMP forms into output_dir after reading the installed raw YAML; the compiler's effective static value was .autoskillit/temp/audit-impl. The raw-file read is not evidence that the server-rendered checkpoint supplied an incorrect value. The audit base SHA passed as branch_name is intentional per skill semantics, not a wrong-target finding.

This occurrence documents material prompt-interpreter errors while preserving this ticket's existing design scope; it does not prove shorter delivery alone fixes adherence. Separately #5291 records the actual missing nested MCP schema fields. The separate planning/audit ownership gap is tracked at #5292: run2's worker-authored NO GO demanded later PR publication, but it was not an admitted audit verdict.

Neither run created a sandbox PR. Both owned branches were cleaned; seed remained OPEN/unclaimed with a fail label. Failed/incomplete runs are not accepted concurrency evidence.

Recommissioning routing evidence — 2026-10-19

At e8ca495, both MiniMax implementation attempts 39309563844 and 39309573468 requested audit_impl=true in root14 and the retained fleet command. Neither complete tool-call inventory or managed-worker set contains an audit-impl invocation. The installed recipe still requires published/replayed audit GO before its normal push/PR path; no live audit GO is established.

First root7a23556c-5c12-48a0-8faa-5154765cb93e: tests267/268, guards274–280 and merge286/287 succeeded, then check_has_commits291, push298 and prepare305 followed without audit. Second root0e898cf8-f50a-4032-aa51-9d603ae40804: tests223/224 passed, then counter resets230–247 led to prepare261, with no commit_guard/main_repo_guard/merge_worktree/push_to_remote/audit call. Its two implementation commits remained in worktree impl-1-20261019-200158, outside the feature branch. Download each run's e2e-implementation-{run}-1 artifact; roots are under out/transcripts/-workspace-sandbox/ with those UUID filenames.

Second compose worker4c0205ec-ae2b-447f-a94e-cc9a170101cd records112 No-commits rejection,126 recovery of older CLOSED PR22,129 emitted URL; root381/382 accepted it. New5297 tracks the independently confirmed URL-only recovery defect. Review worker76500f43-61a5-49ac-b056-344376391544 then records71 gate snapshot_mismatch,117 checkout HEAD at unchanged seed d6c143d,125 old PR head c2ce139/state closed. It also passed anchor-authority as gate-authority at63 and received a correct invalid-state rejection at65. No review was published; root439/440 confirms that. The fleet narrative claiming a new PR was created is contradicted by these receipts.

Both integer-closing-issue-to-empty substitutions are recorded on existing45. These are observed caller field/routing deviations, not evidence of absent recipe branches or a defective canonical decoder. Preserve the existing cost/scope discussion; this append does not claim any particular projection change or broader architecture proposal will make MiniMax follow the entire flow. Both runs remained false/failed with no expected findings; no third manual attempt was dispatched.