Skip to content

Successful recipe checkpoint omits an unconditional on_result fallback segment #5286

Closed

Description

@Trecek

Successful recipe checkpoint omits an unconditional on_result fallback segment

Fix target and integration order

Repository: TalonT-Org/AutoSkillit. Fix PR base: develop, baseline 75141403a04ca98bb4c7e6c3af9ce03eaf838f03 or descendant. This code already exists on develop and does not depend on the unmerged collector. Merge the remediation into develop, then integrate into PR #5255. Do not target the feature branch just because its commissioning exposed the failure.

Confirmed defect

The implementation recipe's check_changes step has a conditional no-changes route and an unconditional fallback to test. Its successful automatic checkpoint omits the fallback's target segment. The parser retains the fallback as a result_condition edge with condition=None, but edge_routes_success rejects that condition before applying the automatic-delivery rule. _target_segment_indices(..., success=True) therefore returns no later segment for this boundary. The test body, including on_success: commit_guard, never reaches the consumer automatically.

run_python is an automatic-delivery tool. This is missing successor-body delivery, not a request for a new routing engine or explicit next-step hint. The route remains recoverable through manual get_recipe_section. The source omission is deterministic; its contribution to the later model confusion is plausible, not proven to explain or prevent the whole run failure.

Live evidence and artifact reproduction

Implementation run 38960061259, attempt1, MiniMax-M3, tested commit 20ee39237e8e89a8d21f403c3a825273fcac6bf7, contains the develop baseline.

gh run download 38960061259 --repo TalonT-Org/AutoSkillit --name e2e-implementation-38960061259-1 --dir .autoskillit/temp/investigate/fallback-delivery-38960061259

Read out/transcripts/-workspace-sandbox/5a5cdc35-9fdf-4480-92c2-78d5fbcb8592.jsonl with record-aware JSONL extraction:

  1. Startup record 15 provides a segment-name overview, not the test body. Records 23 and 74 deliver I1/I2. I2 includes check_changes and close_issue_no_changes, but no test body. No test YAML is delivered before record 193. The generic prompt requires YAML-only routing but does not contain that missing edge.
  2. Successful implement completion183 has an empty checkpoint carrier. check_changes187/result188 then reports has_changes=true, commit_count=1, has_uncommitted_changes=false. Its success carrier has source_step=check_changes, segments=[], bodies=[], pull_closures=[].
  3. No get_recipe_section call occurs before the root invokes test_check192. Success193 delivers I4, whose bodies begin with reset counters before commit_guard. The test->commit_guard edge was not previously delivered.
  4. The root executes counters199/203/207/211/215, then prepare_pr222 and compose_pr295, skipping required integration/audit. Later carrier216 does supply remediate->plan, which the model independently ignores. Do not attribute that separate noncompliance to the missing test body.

The completed run fails review and strict cleanup. Those gates correctly reject a wrong-checkout snapshot and separately unproven published branch. Preserve them. Sandbox PR12/CI do not establish accepted commissioning; the parent later established manual provenance and restored the sandbox.

Deterministic no-model reproduction

Save this under project temp and run with uv run --no-sync python in the prepared project environment. It calls production compilation/selection, starts no model and contacts no network:

from pathlib import Path
from uuid import uuid4
from autoskillit.recipe._api import load_and_validate
from autoskillit.server.recipe._recipe_segment_delivery import (
    _edge_routes_success, _target_segment_indices,
)

root = Path('.autoskillit/temp/investigate') / ('fallback-probe-' + uuid4().hex)
root.mkdir(parents=True)
r = load_and_validate(
    'implementation', project_dir=root.resolve(), temp_dir=root.resolve() / 'cache',
    ingredient_overrides={
        'task': 'Normalize whitespace in reusable fixture', 'source_dir': '.',
        'issue_url': 'https://github.com/TalonT-Org/autoskillit-e2e-sandbox/issues/1',
        'audit_impl': 'true', 'open_pr': 'true', 'auto_merge': 'false',
    }, defer_unresolved=True, include_finalized_projection=True,
)
assert r['valid'], r.get('errors')
p = r['_finalized_projection']
for e in p.ordered_flow_edges:
    if e.source == 'check_changes' and e.edge_type == 'result_condition':
        print(e.target, e.edge_type, e.condition, _edge_routes_success(p, e))
print('success target segments:', _target_segment_indices(p, 'check_changes', success=True))
print('test segment:', next(
    (i, s.name) for i, s in enumerate(p.delivery_segments)
    if 'test' in s.ordered_step_names
))

Observed:

close_issue_no_changes result_condition ${{ result.has_changes }} == false True
test result_condition None False
success target segments: ()
test segment: (3, 'I3')

The selector receives a success flag, not has_changes; it collects possible successor segments rather than evaluating the actual result branch. The defect is exclusion of the unconditional fallback from that delivery set. The explicit conditional target remains in I2 and is filtered out as same-segment.

Source mechanism and test gap

Links pin develop baseline75141403a:

  • Recipe routing: conditional false -> close_issue_no_changes; fallback -> test. Test success -> commit_guard at line 695.
  • Parser retains when=None. Graph extraction emits result_condition with that condition.
  • edge_routes_success splits (edge.condition or '') on == and returns False before return automatic for the fallback.
  • Selector filters that edge from success. Carrier consequently emits no I3 body.
  • Registry declares automatic/recovery delivery for run_python; test_check declares both too. This is not an intentional non-checkpoint boundary.

Existing checkpoint tests use a simple success edge and one-step target, not an automatic conditional-result step with a default fallback crossing into a later segment. Compiler validation and runtime share the classifier; accepting the recipe does not establish correct success delivery at this boundary.

Required behavior and validation

Repair successful automatic checkpoint delivery so this reachable unconditional fallback's target segment/body arrives. Use existing graph/delivery machinery. Preserve tool-specific success/failure semantics, including wait_for_ci and merge-queue result handling, recovery/pull-only paths, bounded carriers and manual fallback. Do not indiscriminately mark all fallbacks on every tool as success. Do not introduce a parallel route map, generic executor or new hint system.

Add a regression for conditional no-change/default-test routes and actual successful checkpoint delivery into I3, including the test body's commit_guard route. Exercise compiler acceptance and runtime carrier together. Preserve special-tool failure/recovery tests and a manual pull case. Run all-files pre-commit and configured task test-local-gate; never invoke pytest directly. After integration, rerun full commissioning with gates intact. Delivery correctness does not guarantee model compliance.

Related work and confidence

Open #4482 proposes context reduction and additive routing hints. Closed #4578 introduced progressive delivery with manual fallback and successful-checkpoint loading. This narrow repair of an existing automatic checkpoint is distinct from the optional explicit next-step-hint proposal. Record the mixed I4 entry-hint observation under #4482 separately. Closed #1083's prompt-only skipping history is not this source mechanism.

The YAML graph itself is correct, and manual recovery is available. Missing automatic delivery and the classifier mechanism are reproduced directly. The later model also ignored a delivered remediate route; do not claim this omission alone caused the full failed run. Classification: bug, recipe:remediation.

Actual repaired handoff verified — 2026-10-17

Fix #5287 is included in published28448805b through developf66c21cc6. A no-model production-function probe now selects I3/test after successful check_changes. Focused configured verification passed457 tests, all-files pre-commit passed and full Tests39082335995 passed.

More directly, implementation39082343853 rootd283a9e8 transcript result195 delivers source_step=check_changes, segmentI3 and the full test body including on_success=commit_guard. The previously omitted fallback body is actually in the model's context. The old omission did not recur. The model subsequently misroutes some calls despite available routing, which is separate execution evidence under #4482.

Both fresh implementation attempts fail later acceptance and do not establish commissioning success. Keep this fixed delivery issue closed; #5255 records final results and new prep-output issue #5289. No expected-failure acceptance or weakened metadata/ownership validation was added.

Activity

  1. added
    bugExisting behavior is broken
    recipe:remediationRoute: investigate/decompose before implementation
    on Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugExisting behavior is brokenrecipe:remediationRoute: investigate/decompose before implementation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions