Skip to content

fix(backend): version corrections and recover finalize retries - #315

Draft
SamGu-NRX wants to merge 15 commits into
obv/burst14-bs-correction-finalize-racesfrom
gpt-burst/1022w2-bs-races
Draft

SamGu-NRX wants to merge 15 commits into
obv/burst14-bs-correction-finalize-racesfrom
gpt-burst/1022w2-bs-races

Conversation

@SamGu-NRX

@SamGu-NRX SamGu-NRX commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

Fixes the correction/completion races by making accepted confirmations and published results immutable. The lease remains an advisory duplicate-work guard because the shared StateStore protocol has atomic insert-if-absent but no compare-and-delete operation.

What changed

  • Acceptance journals an immutable answer without an owner, after the retry slot exists and before acknowledgment. The hook only nudges processing afterward. A run selection that races creation or failure cannot permanently bind the accepted answer.
  • Runs claim answers through claim:<capture>:<confirmation>:<generation> using put_if_absent. Only a failed owner permits the next generation. Competing claimants cannot overwrite each other or delete claim history.
  • Creation publishes the pending run before seeding. The newest eligible run claims available answers during seeding, nudges, snapshots, or sweeps. A delivered run with a newer run never adopts new answers, even if that newer run failed.
  • Unowned answers and failed-owned answers remain available for adoption. Live owners recompose until their published version covers their claimed answers. A late takeover preserves acceptance order when replaying fields, so an older answer cannot replace a newer one.
  • Before skipping legacy apply_pending, the real and stub reads stages idempotently import old workers' confirmation slots. Run creation also imports them. An interrupted import can resume without duplicating an answer.
  • Criteria and result fold each run's correction sequence onto private input copies. The confirmation hook never changes the shared base reads/reads.json.
  • Each criteria/result composition gets a private work directory and new storage keys. The result stage uploads those bytes before inserting result_version:<run>:<sequence>. A version records the correction IDs it incorporated. Older versions are never overwritten.
  • Result reads choose the latest immutable publication. Terminal events carry internal version metadata, while the existing iOS event types and payloads stay unchanged. A sweep can emit a final event after a crash between publication and event append.
  • Completion and confirmation races now leave accepted corrections pending by coverage, not by a separately deletable marker. A stale lease release can still delete a successor's advisory token, but it can cause only duplicate recomposition. It cannot erase a correction or overwrite a published result.

Why

Lease ownership checks cannot fence a later write or deletion across separate StateStore calls. Immutable records keep correction recovery independent of lease ownership. The earlier finalize retry fix also preserves queued work when a worker finishes or fails.

Round 4 ownership recovery

Acceptance review of 3d6d2a18 found two remaining gaps: an answer could arrive after seeding but before the new run became eligible, or its selected owner could fail during acceptance. Ownerless acceptance and failed-owner claim generations replace that fixed-target design.

Seven deterministic regressions fail on 3d6d2a18 and pass here. They pause creation after seeding and during the pending state, fail a run during journal insertion or after acknowledgment, race two takeover claimants, and check that a delivered predecessor's result and events do not change. Events and barriers assert that each boundary was reached; no sleeps synchronize the tests.

Claim unit tests cover generation contention on memory and local Modal backends, failed-owner takeover, delivered-owner protection, and acceptance-order replay after late adoption. Versioned publication, advisory leases, legacy import, and iOS-facing payloads remain in place.

Round 3 regression evidence at 3d6d2a1

The independent review of dd6e645e found historical-run contamination, answers stranded on failed runs, missing legacy imports, and inactive test barriers. Round 3 addressed those cases; acceptance review then found the creation and failure windows covered below.

  • Run-scope tests preserve A's served result and public events after correcting B. They also check failed-run isolation, one-time adoption by the next run, and retry targeting.
  • Legacy-import tests cover the real reads entry point, the stub, pre-creation import, interruption recovery, and modern-answer deduplication.
  • Race tests now force done-run lease contention and competing immutable insertions. Each barrier or injected failure asserts it was reached. HTTP tests inject failures before and after journal insertion and immediately before acknowledgment.
  • The acknowledgment test caught a duplicate-ID regression in the first Round 3 checkpoint. Keeping the retry slot durable before journaling fixes that ordering gap. Round 4 removes target reservation entirely.

Finalize overlap

This PR is stacked on obv/burst14-bs-correction-finalize-races, PR #295. Its earlier finalize retry fix remains in backend/server/api/finalize.py: _clear_slot at lines 145-153, _guard at 501-548, and _finish_loop at 551-562. Rounds 2 through 4 made no further edits to that file.

At #327 head 7d3a996e, Zephyrus adds _non_finite_pointer at lines 77-98, validation at 240-242, and the packet acceptance claim with storage-failure cleanup at 281-297. The inspected diffs change the same file but do not directly intersect. No merge or conflict-resolution check ran.

Verification

At pushed head 1b666409, from backend:

  • OMP_NUM_THREADS=1 uv run pytest -q: 1568 passed, 16 skipped, 4 failed, 14 errors, in 378.31 seconds.
  • The full worktree baseline at 3d6d2a18 had 1557 passed, 16 skipped, 4 failed, 14 errors. Both runs have the same four evaluation-gate failures and the missing load_receipt import behind all 14 cold-restart errors. No new full-suite failure appeared.
  • OMP_NUM_THREADS=1 uv run pytest -q server/orchestrator/tests server/api/tests/test_hooks_orchestrator.py tests/test_runtime_join_journey.py tests/test_result_published_inputs.py: 251 passed.
  • All seven new ownership regressions fail on 3d6d2a18 with lost-answer or missing-owner assertions and pass unchanged here.
  • uv run ruff check . passes. uv run ruff format --check . passes for 517 files.
  • Earlier Round 3 checks passed 22 focused tests. Its eight new scope/import regressions failed on dd6e645e. Mutations that skipped the busy path, swallowed storage errors, or dropped a contending answer each failed the strengthened tests.
Earlier Round 2 evidence

At dd6e645e, the four then-current portable races failed on 2b696e4a and passed on the fix. The broader publication, orchestration, result, and runtime-join selection passed 262 tests. The archived 2b696e4a full run had 1513 passes, 16 skips, six failures, and 14 errors. Its two additional failures involved CLI replay and SSR checks affected by missing Git metadata, so that archive was not a fully equivalent checkout.

Risk

No production or remote Modal run occurred. Legacy compatibility uses injected old-format rows in local tests, not a live mixed-version deployment. Acceptance review covered 3d6d2a18; the Round 4 head still needs independent acceptance. Context7 remained unavailable because its monthly quota was exhausted.

Checklist

  • The scope is focused; related changes belong together.
  • I explained what changed and why.
  • N/A: no visible UI changes, so no before-and-after images.
  • N/A: no UI motion changes, so no video.

Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Oct 22, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the size:L 100-499 effective changed lines. label Oct 22, 2026
@SamGu-NRX

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-22T15:27:59.176141Z 2b696e4 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

SamGu-NRX and others added 5 commits October 22, 2026 11:41
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep this checkpoint as WIP: the new deterministic regression is red.
Ownership checks do not atomically fence writes or deletes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Checkpoint the isolated store operations before wiring stage publication. Eight memory and Modal-adapter tests pass; the previous lease-based integration remains incomplete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added size:XL 500-999 effective changed lines. and removed size:L 100-499 effective changed lines. labels Oct 22, 2026
SamGu-NRX and others added 4 commits October 22, 2026 12:59
Keep base reads unchanged, hydrate matching criteria/result inputs, upload each composition to new keys, then append its immutable version and final event. Recover uncovered answers on terminal runs. Focused tests pass; forced dual-composer coverage and full verification remain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All four portable regressions fail on 2b696e4 and pass here. Cover stale lease release with two composers and immutable result objects, concurrent stale reads, busy acceptance, and a storage-to-marker crash. Broader orchestrator/result verification passes 242 tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record an internal event watermark on each immutable publication, filter late old emitters and duplicates, and retain historical finals. Recover a publication interrupted before its event. Keep confirmation retries idempotent when clientAt changes. Thirteen event, API, and runtime join tests pass; full Ruff check and format check pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Equal-coverage versions can publish changed verdicts without overwriting prior objects. Keep the existing final-event retry contract, reject missing result artifacts before publication, and retain criteria input isolation. Publication and orchestration integration: 262 passed; journal and fault tests: 25 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@SamGu-NRX SamGu-NRX changed the title fix(backend): serialize correction completion and finalize retries fix(backend): version corrections and recover finalize retries Oct 22, 2026
SamGu-NRX and others added 5 commits October 22, 2026 13:58
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep the retry slot durable before journaling the answer so an interrupted POST retains its confirmation ID.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Publish pending runs before seeding; allow failed claim owners to be superseded without deleting history. Replay adopted answers in acceptance order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL 500-999 effective changed lines.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant