Investigation: Server-side variable substitution via prehook tool for recipe with: fields #45

Summary

Recipe steps today use a ${{ inputs.X }} / ${{ context.X }} template syntax in their with: fields, but no server-side code resolves those context.* templates. The substitution remains a pure convention the orchestrating LLM agent interprets when it reads the YAML from load_recipe. The agent mentally assembles the resolved values and then passes literal strings to run_skill / run_skill_retry.

This works, but the entire correctness guarantee rests on the LLM never misreading a template or forgetting which context variables are in scope. The hypothesis is that a dedicated prehook tool — called by the agent immediately before each run_skill invocation — could take the step's with: dict plus the live pipeline context and return fully-resolved arguments, making substitution deterministic and observable on the server side.

PR #3216 (merged 2026-05-29) delivered a partial fix only: server-side substitution of ${{ inputs.* }} references in served recipe content, routed through _resolve_hidden_inputs_in_content() at recipe composition time (now at src/autoskillit/recipe/ingredients/_recipe_composition.py:562 via the post-#4671 subpackage shim). It does not address ${{ context.* }} references, which are still resolved entirely by the LLM. The bookkeeping on that PR is also slightly off — its body says Closes #3209, but #3209 is still OPEN at the time of this refresh, so the closure attribution should not be relied on as evidence that #3209 itself was resolved. Note that the _resolve_hidden_value() function cited below at _binding_input.py:135-158 was extracted into that module by PR #4854 on 2026-08-26 — it performs related but separate hidden-input substitution at a different code path.

Current Architecture (the gap)

Where templates appear

Every step in a recipe (e.g. recipes/implementation.yaml) uses templates freely. A direct count yields 301 ${{ context.* }} references in src/autoskillit/recipes/implementation.yaml alone (the bundled recipe that most heavily exercises the context.* namespace):

implement:
  tool: run_skill_retry
  with:
    skill_command: "/autoskillit:implement-worktree-no-merge ${{ context.plan_path }}"
    cwd: "${{ inputs.work_dir }}"

Where they are (and are not) resolved

The regex patterns for all three template namespaces now live in src/autoskillit/recipe/contracts/_contracts_types.py:17-20 (the recipe/contracts.py module was restructured into a subpackage during the recent recipe-layer decomposition):

_CONTEXT_REF_RE = re.compile(r"\$\{\{\s*context\.([A-Za-z_]\w*)\s*\}\}")  # line 17
INPUT_REF_RE    = re.compile(r"\$\{\{\s*inputs\.([A-Za-z_]\w*)\s*\}\}")   # line 18
_TEMPLATE_REF_RE = re.compile(r"\$\{\{[^}]+\}\}")                         # line 19
RESULT_CAPTURE_RE = re.compile(r"\$\{\{\s*result\.([\w-]+)\s*\}\}")        # line 20

These patterns are used only by the static validator (src/autoskillit/recipe/validator.py:323-342 — the loop that walks with_args and validates each inputs.* and context.* reference via iter_steps_with_context), never at runtime.

The actual execution path is unchanged in shape from the original issue:

1. load_recipe (src/autoskillit/server/tools/tools_recipe.py) returns raw YAML — no substitution.
2. The orchestrator agent reads it, builds a mental context dict, and calls run_skill with literal strings it computed itself.
3. run_skill (src/autoskillit/server/tools/tools_execution/) receives a plain skill_command: str. It passes that directly to run_headless_core (src/autoskillit/execution/headless/), which hands it verbatim to claude -p.

Note the three subpackage restructures since the issue was filed: server/tools_execution.py → server/tools/tools_execution/, execution/headless.py → execution/headless/, and recipe/contracts.py → recipe/contracts/.

Where partial server-side substitution exists today

Three narrow substitution paths have been added since filing; none of them resolve context.* references:

• src/autoskillit/recipe/_binding_input.py:135-158 — _resolve_hidden_value() resolves only hidden inputs.* references (those marked hidden from the orchestrator). Returns effective unchanged for non-strings or non-hidden references.
• src/autoskillit/recipe/helpers/_io_loading.py:194 — _substitute_recipe_values() is recursive but only substitutes temp, scripts, and recipe-note placeholders. It does not touch inputs.* or context.*.
• src/autoskillit/server/tools/tools_execution/_run_skill_prepare.py:458-461 — Server-side recipe parameter resolution explicitly skips values containing ${{ placeholders, with the comment: "Skip values containing unresolved template references — a finalized projection may retain ${{ context.* }} placeholders."

# src/autoskillit/server/tools/tools_execution/_run_skill_prepare.py:456-461
if not state.output_dir and "output_dir" in _recipe_step.with_args:
    _recipe_output_dir = _recipe_step.with_args["output_dir"]
    # Skip values containing unresolved template references —
    # a finalized projection may retain ${{ context.* }} placeholders.
    if isinstance(_recipe_output_dir, str) and "${{" not in _recipe_output_dir:
        state.output_dir = _recipe_output_dir

This skip guard is the strongest evidence that server-side resolution of ${{ context.* }} is intentionally deferred on the runtime side — the tool already knows a context.* reference can land in with_args and explicitly refuses to touch it.

Where the context dict lives

Currently: only in the LLM's conversation context. The orchestrator is told to maintain it. From the agent-instruction prompt at src/autoskillit/server/tools/tools_recipe.py:243-245:

Steps with capture: fields extract values from tool results into a pipeline context dict. Use captured values in subsequent steps via ${{ context.var_name }} in with: arguments.

There is no server-side store for the pipeline's current ingredient values or accumulated captures.

Related Work

PR #3216 — partial fix (hidden inputs.* in served content only)

PR #3216 ("Fix hidden ingredient ${{ inputs.* }} interpolation in served recipe content", merged 2026-05-29) added _resolve_hidden_inputs_in_content() to _recipe_composition.py (now at recipe/ingredients/_recipe_composition.py:562 via the post-#4671 subpackage shim). It closes the gap for hidden-ingredient substitution at recipe-composition time but does not implement any of the three approaches proposed here (prehook tool, inline context parameter on run_skill, or server-side context store), and it explicitly does not resolve ${{ context.* }} references.

Bookkeeping note: the PR body is annotated with Closes #3209, but #3209 is still OPEN. This is a minor bookkeeping discrepancy — the PR's actual code change does affect the area #3209 tracks, but the formal "Closes" linkage should not be treated as evidence that #3209 itself is resolved.

Issue #3065 — closed upstream, complementary direction

#3065 was CLOSED on 2026-07-23 (stateReason: COMPLETED) after adopting a Python-side evaluation approach for skip_when_false rather than agent-side substitution. Its body explicitly states: "This aligns with #45's direction of moving template/condition evaluation server-side." That complementary work is COMPLETED upstream; this issue tracks the parallel context.* substitution path that remains entirely LLM-mediated.

Proposed Investigation (still valid, with current file references)

Question 1 — Is a dedicated prehook tool the right shape?

The idea: a new MCP tool (resolve_step_args or apply_context) that the agent calls before each run_skill:

# Agent calls this first:
resolved = resolve_step_args(
    with_args={"skill_command": "/autoskillit:impl ${{ context.plan_path }}", "cwd": "${{ inputs.work_dir }}"},
    inputs={"work_dir": "/home/user/project"},
    context={"plan_path": "/home/user/project/temp/make-plan/plan.md"},
)
# resolved → {"skill_command": "...", "cwd": "/home/user/project"}

# Then calls run_skill with the resolved values.

Pros:

• Substitution is verifiable and logged server-side.
• Missing variables fail fast with a structured error instead of ${{ context.plan_path }} being passed literally to a headless session.
• Tool is stateless — no server-side context store needed.
• Composable: works for any tool, not just run_skill.

Cons:

• Adds one round-trip per step (latency + token cost).
• Agent might still pass pre-resolved strings and skip the prehook.
• Doesn't prevent the agent from calling run_skill directly with unresolved templates.

Question 2 — Should substitution happen inside run_skill directly?

Alternative: run_skill and run_skill_retry accept an optional context parameter, and run_headless_core would resolve all ${{ }} references in skill_command, cwd, and add_dir before launching the subprocess.

Pros:

• No extra round-trip.
• Substitution is guaranteed — no way to bypass it.
• The ${{ }} syntax in recipes becomes executable, not just documentation.

Cons:

• Requires schema changes to run_skill / run_skill_retry tool signatures.
• Adds implicit behavior to a currently transparent tool (conflicts with the explicit skip guard at _run_skill_prepare.py:460).
• Context dict still must be assembled and passed by the agent.

Question 3 — What about a server-side pipeline context store?

The most ambitious variant: run_skill results are automatically captured into a server-side context dict (keyed by the recipe's capture: declarations), and a set_pipeline_context tool lets the agent seed ingredient values at pipeline start. Subsequent run_skill calls look up their ${{ }} references from this store automatically.

Key open questions:

• Lifetime: is the store per-load_recipe call, per-session, or explicit?
• Multi-pipeline isolation: what if two pipelines run concurrently?
• This would be a significant architectural addition — warrants careful scoping.

Question 4 — Impact on existing recipes

All existing recipes (implementation.yaml, bugfix-loop.yaml, audit-and-fix.yaml, etc.) use ${{ }} templates extensively. The investigation should verify:

• Do any recipes pass pre-resolved strings today in a way that would conflict with server-side substitution?
• Are there edge cases where a template value is intentionally left literal (e.g. passed through to a downstream tool)?
• Would adding an optional context param to run_skill be backward-compatible with all current recipe YAML?

Files to Examine

Acceptance Criteria for the Investigation

• Evaluate all three approaches (prehook tool, inline run_skill param, server-side store) and recommend one.
• Prototype or spec the API surface for the recommended approach.
• Identify whether any existing test coverage would need to change.
• Confirm backward compatibility with all 5 bundled recipes.
• Document the inputs.* vs context.* namespace distinction — do they need to be unified?
• Assess whether the prehook should be gated (kitchen) or ungated.
• Reconcile with PR Fix hidden ingredient ${{ inputs.* }} interpolation in served recipe content (#3209) #3216: confirm scope boundary (hidden inputs.* only) and re-verify that Hidden ingredient interpolation fails in run_diagnostic recipe steps #3209's open status does not invalidate the closure attribution.

Labels

investigation, recipes, dx, enhancement

MiniMax execution evidence — 2026-10-17, head28448805b

Fresh implementation39082343853 and 39082347591, MiniMax-M3 through Claude Code2.1.280, expose the existing caller-side context-resolution risk. Both receive the canonical sandbox issue1 URL and successfully claim it, yielding integer issue_number1. Both already pass string "1" to create_and_publish_branch. Later they send integer1 to prepare-pr closing_issue, correctly receive an expects-string rejection, and then replace the still-available number with empty absence values instead of valid string "1". Both PR bodies lack the required canonical closing link.

Run1 rootd283a9e8 transcript records43–48 and444–468 preserve the sequence. Run2 rootfd37c73d records46–51,247–280,291–292 and323–338 preserve its repeated type/shape recovery and direct PR-create fallback. Download the runs' e2e-implementation-RUNID-1 artifacts; root transcripts are under out/transcripts/-workspace-sandbox/. The source recipe maps context.issue_number, while the typed skill slot requires a string; _prompts.py193 reserves absence values for unavailable context. Current runtime does not promise server-side context coercion.

Two independent adversarial reviewers and parent primary-source inspection rejected a new source-bug ticket for this blanking: valid string input is supported and the caller discarded available identity. This is empirical context for this existing issue, not a new enforcement guarantee or a change to its scope. A separate actual prep-output evidence mismatch is filed as #5289; it does not explain away the independent issue-identity loss. Both commissioning attempts remain failed in #5255.

Recommissioning evidence — 2026-10-18 (e529c48)

Tested feature head e529c48a8d6576ca7bbff29a51c0d7ee12caa167, integrated develop a93c20037bddef0c534af3f7da44ca4310733b33 including #5290/#5289.

Both MiniMax attempts failed at audit admission: 39177297225 and 39177304001. After SEMANTIC_REJECTED exposed an audit_attempt_id, the orchestrator omitted the recipe's top-level retry_after_audit_attempt_id on retry. Root JSONL call/result pairs: run1 533/534, run2 296/297. Both returned correction_token_required. The recipe already captures and forwards the value (implementation.yaml1226-1288); source admission passes it into ledger reservation. Do not diagnose missing recipe wiring or relax admission from this evidence.

This is another observed L2 data-plumbing failure relevant to deterministic context substitution. The distinct L1 ArtifactRef schema gap is #5291; its invalid numeric-string calls are not proof that valid integer submissions fail, and resolving this issue alone is not proof that the L1 provider behavior will comply.