[MuncleUscles]

Problem and outcome

Finalizing transaction A currently copies the contract's latest accepted head. If B has already been accepted on the same contract while its appeal window is still open, finalizing A incorrectly publishes B's state as finalized.

Persist each successful decision's own post-state and finalize only that state, independently of younger accepted transactions.

Follow-up commits implement the maintainer plan from review: contract upgrades close the earlier lifecycle, legacy rows recover from successor pre-states, and backed-off finalization heads no longer starve other contracts.

Delivery context

• Target: v0.123-dev; merged with v0.123-dev at 9e907cd2 (includes fix(genvm): serve latest-finalized storage reads from finalized state 🐛 #1792 and the chore(release): v0.123 RC fixes (#1765, #1715, #1773, #1789) #1797 RC fixes) in d3e52332, no conflicts.
• Studio-only change; no cross-repository Depends-On requirement.
• Complements fix(genvm): serve latest-finalized storage reads from finalized state 🐛 #1792: that PR routes LATEST_FINALIZED reads to finalized storage; this PR makes the contents of finalized storage correct.
• Owner: @danieljrc888.

Implementation

Transaction-local finalization state (original change)

• Preserve rollback pre-state and transaction-local finalization_state together in the existing transaction snapshot JSON.
• Refresh post-state on a successful recomputation without overwriting rollback state; failed appeals preserve the original decision's post-state.
• Finalization reads only its saved post-state when the worker claim omits the full snapshot, and verifies the receipt hash when available.
• Validate state before message dispatch, status updates or fee settlement.
• Keep unverifiable legacy decisions accepted and retryable instead of incorrectly finalizing or canceling them after generic retry exhaustion.
• Treat an empty post-state as valid, distinct from missing state.

Backoff without starvation (84dea930)

• claim_next_finalization skips contracts whose head is inside a retry backoff window instead of re-claiming the same blocked oldest candidate. Same-contract order is preserved: a backed-off row holds back itself and every younger row on its contract.
• Retry bookkeeping is cleared when a transaction finalizes.

Legacy recovery from successor pre-states (7a394045)

• resolve_finalization_state keeps the existing hash-verified fallbacks and additionally tries the pre-state snapshots of up to 3 younger same-contract transactions, accepting one only when its hash matches the receipt's contract_state_hash. No unchecked latest-head fallback and no automatic cancellation.
• New read-only report to run before rollout: python -m backend.database_handler.finalization_legacy_report [--limit N] lists decided rows and how each would recover, or why it is UNRECOVERABLE (no hash, no matching snapshot, contract missing, interleaved upgrade).

Upgrades close the earlier lifecycle (4713ec3b, b2fc9eea)

Scheduling is unchanged (upgrades keep priority and wait for current same-contract work). Processing an UPGRADE_CONTRACT transaction now:

1. Validates input; invalid upgrades are canceled with no side effects.
2. Defers (keeps nothing, retries later) while any other same-contract row is leased or mid-consensus.
3. Preflights every decided predecessor's finalization state with the same resolver finalization uses; if any cannot be resolved, the upgrade is canceled (predecessor_finalization_state) before anything is touched.
4. Force-finalizes each decided predecessor in queue order through the normal finalization path (fees, value refunds, internal messages, appeal bond settlement). A pending paid appeal is refunded first (abort_pending_appeal_admission, guarded so resolved bonds are never refunded), atomically with a progress record and lease renewal on the upgrade row, so retries never refund or record twice.
5. Applies the code to both state trees, clears stale appeal recovery snapshots on never-started rows, and marks the upgrade FINALIZED in one commit. The receipt records closed_transactions, refunded_appeals and applied_at.

Never-started transactions stay queued and run on the new code. Failures after the preflight are transient and retried with backoff; predecessors already finalized stay final.

Appeals and rollback:

• Appeal admission returns CanNotAppeal for a decision with a PENDING or FINALIZED upgrade queued after it (RPC precheck and again under the row lock).
• Rewinds (get_rewindable_newer_transactions, rollback_transactions) never reset FINALIZED, CANCELED or upgrade rows, so an older appeal can no longer restore old code or reopen an upgrade.
• Legacy interleaving: when a decision overtaken by a pre-change FINALIZED upgrade finalizes or restores its pre-state, the upgrade's code slot is kept.

Validation

Executed locally on the review head:

• Backend unit suite: 1,731 passed, 7 skipped; 1 failure (test_studio_fees.py::test_leader_timeout_appeal_uses_next_normal_fee_schedule) that also fails on the base in this environment and passes in a clean export.
• PostgreSQL suite: 268 passed, 1 existing xfailed.

New regression coverage includes: backed-off heads not starving other contracts while preserving same-contract order; successor pre-state recovery and the legacy report; upgrade force-finalizing ACCEPTED, UNDETERMINED and timeout predecessors with fee settlement; never-started rows running on the new code; paid appeal refunded (nothing forfeited); appeal admission blocked by queued and applied upgrades; validator appeal after an upgrade not restoring old code; CANCELED rows not resurrected; unresolvable predecessor canceling the upgrade untouched; crash and retry without double refund or settlement; deferral while a stale-leased row exists; legacy interleaving; upgrade claim priority.

Full E2E on this head is requested separately.

Upgrade and rollback caveats

No schema migration, SDK interface change or dependency pin change. Additional state lives in existing transaction JSON.

Rollout:

• Do not run old and new workers concurrently.
• Before deploying, run finalization_legacy_report against the target database. Resolve any UNRECOVERABLE rows explicitly: they deliberately remain blocked, with no finalization side effects.
• Rolling back to the old implementation reintroduces incorrect-head finalization and upgrade replay.

Known limits:

• A deferring upgrade re-occupies a worker for about 5 s per attempt until a stale lease is recovered.
• If an upgrade is canceled after some predecessors were force-finalized, those stay final and the code is not applied.
• Legacy interleaving restores only the code slot; storage written by the old code is not migrated.
• Decisions queued before an upgrade can no longer be appealed.

User-facing release note

Finalized contract reads no longer expose the state of a newer transaction whose finalization/appeal period has not completed. Contract upgrades now finalize earlier decided transactions (refunding pending appeals) before the new code applies, and can no longer be undone by an older appeal.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 26024a8f-d41d-4b7e-97c1-975f22d5a64b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[danieljrc888]

Reviewed the upgrade caveat. The fix itself looks correct: I found no path that writes a wrong finalized state, and the new unit tests pass (tests/unit/: 1683 passed on this head; it also composes cleanly with the other v0.123-dev candidates: 1717 passed). The concern is liveness during the upgrade, and it is broader than the description suggests.

Finding: one unrecoverable legacy row can stall finalization on every contract

A transaction accepted by the old code has no finalization_state, and its stored receipt is stripped, so the only recovery is hash(live accepted head) == receipt.contract_state_hash (base.py:3907-3913). That fails exactly in the overlap case this PR fixes: legacy tx A is still in its appeal window and any younger tx B on the same contract gets accepted (before or after the deploy). From then on:

1. A never finishes. _transaction_finalization_state raises FinalizationStateError on every attempt, and worker.py:977 excludes that error from cancel_on_exhaustion, so A retries forever.
2. That contract freezes. claim_next_finalization requires every older tx on the contract to be terminal, so B and everything after it never finalize. claim_next_transaction also defers new PENDING/ACTIVATED work queued behind a past-window ACCEPTED tx, so new writes stall too.
3. Finalization stalls globally. claim_next_finalization returns a single global candidate (ORDER BY queue_order ASC LIMIT 1 over rn = 1 rows). When that candidate is in the worker's backoff, _try_claim_work releases it and skips finalization for the cycle without trying the next one. A stuck legacy row that is the oldest eligible row therefore blocks finalization for every contract, and only slips past it while another worker holds it. Before this PR a generic error was cancelled after 3 retries, which ended the stall; now it is unbounded.
4. Rolling deploys keep creating legacy rows. Old consensus-worker pods keep accepting without finalization_state and keep finalizing from the moving head until they are gone.

No wrong state is written (fail-closed), but on stg/prd one busy contract with two writes near the deploy is enough, and recovery needs manual DB work.

Suggested fixes

1. Successor pre-state fallback (exact, cheap). When B executed on top of A, B's rollback pre-state equals A's post-state. Before raising, look for a younger transaction on the same contract whose saved pre-state hashes to the receipt hash:

# transactions_processor.py
def find_successor_pre_state(
    self, to_address: str, queue_order: int, expected_hash: str
) -> dict | None:
    rows = self.session.execute(
        text(
            "SELECT contract_snapshot->'states'->'accepted' FROM transactions "
            "WHERE to_address = :addr AND queue_order > :q "
            "AND contract_snapshot IS NOT NULL ORDER BY queue_order"
        ),
        {"addr": to_address, "q": queue_order},
    ).scalars()
    for state in rows:
        if state is not None and _compute_contract_state_hash(state) == expected_hash:
            return state
    return None

and in _transaction_finalization_state, after the live-head check and before the final raise:

        state = context.transactions_processor.find_successor_pre_state(
            context.transaction.to_address,
            context.transaction.queue_order,
            expected_hash,
        )
        if state is not None:
            return state

The hash check means any match is exactly A's result, so this cannot introduce wrong state. A DB test with legacy A + accepted B (no finalization_state on A) covers it.

2. Bound whatever is still unrecoverable. Pick one:

• keep FinalizationStateError cancellable after N attempts, or fall back to the previous head-copy finalization behind a loud warning / env flag (no worse than before this PR, and B's own finalization overwrites it shortly after); or
• make claim_next_finalization skip rows that are in backoff or flagged as blocked, so one row cannot cause head-of-line starvation for other contracts.

Also log the missing-state case once (or rate-limited) instead of a stack trace roughly every 40s per worker.

3. Deploy plan, needed with or without the code changes.

• Recreate the consensus workers (scale to 0, then up) instead of a rolling update, so old and new finalizers never overlap.
• Before deploying, check for rows still exposed:

  SELECT hash, to_address, status FROM transactions
  WHERE status IN ('ACCEPTED','UNDETERMINED','LEADER_TIMEOUT','VALIDATORS_TIMEOUT')
    AND contract_snapshot->'finalization_state' IS NULL;

• After deploying, watch for Missing transaction-local finalization state and have the manual recovery query ready.

Minor: every successful accept now stores both the pre- and post-state in contract_snapshot, roughly doubling its size (it has been observed at tens of MB), and appeal claims load it. Pruning still covers terminal rows, so this is just a heads-up.

With (1) plus either (2) or the deploy plan in (3), I'm happy for this to go into the v0.123 release candidate.