Skip to content

fix(runnerhub): ERRORED cleanup releases only the binding it saw (RIG-4742) - #1810

Open
rigel-mintaka wants to merge 7 commits into
compass-runner/4452-errored-terminal-guardfrom
compass-runner/4742-binding-lifetime-token
Open

rigel-mintaka wants to merge 7 commits into
compass-runner/4452-errored-terminal-guardfrom
compass-runner/4742-binding-lifetime-token

Conversation

@rigel-mintaka

@rigel-mintaka rigel-mintaka commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

This PR is part of a stack containing 2 PRs:

  1. main
  2. fix(runnerhub): ignore lifecycle frames that arrive after ERRORED (RIG-4452) #1774
  3. "fix(runnerhub): ERRORED cleanup releases only the binding it saw (RIG-4742)" (this PR)

Summary

The ERRORED lost-session cleanup runs detached. Before this change, it unbound whatever row held the session id when it ran. A resume that re-bound the same session id in the same enrollment could therefore be unbound and archived by the old lifetime's cleanup. Implements RIG-4742 option A (binding lifetime token).

  • Binding identity: each sessionBinding now carries:
    • version: a new session_bindings.binding_version column, set to a fresh UUID on every RecordSessionBinding upsert and returned by Record and Resolve. A UUID, not xmin, because transaction ids wrap around.
    • lifetime: a per-insert counter for the cache, which stores without a durable row rely on.
  • Cleanup: the ERRORED path passes the binding it saw. dropLostSession skips if the current binding differs. releaseSession re-checks the cache before and after the durable delete.
  • Conditional durable delete: a new DeleteSessionBindingVersion(ctx, sessionID, version) deletes only that write and reports whether it removed a row. If a peer Server re-bound the row, it removes nothing and the cleanup stops. A cached binding whose durable write failed has version "". It never deletes a row; if a row exists, the session was re-bound elsewhere, so the stale cache entry is dropped and no loss is reported. DeleteSessionBinding is unchanged and still serves Stop and the deliver-refusal loss.
  • Stale reverse entry: when a cleanup finds its row re-bound, it drops both the stale session entry and the account entry that still names it.
  • Reverse read-through: SessionForAccount now returns the row's version, so a binding cached from a delivery's reverse read carries it. Its ERRORED cleanup then deletes that row and reports the loss.
  • Ordering with enroll (rebased onto fix(runnerhub): serialize enroll reaps with session promotion #1756): releaseSession takes bindingWriteMu like promoteSession, so a Stop's delete cannot land between an enroll's map-clear and its reap.
  • Migration 0003_session_binding_version.sql: adds the column with a constant default (no table rewrite), then backfills existing rows with a UUID.
    • Renumbered to 0003 after main took 0002_forge_scopes.sql. Any open PR that lands first with a 0003 forces another renumber.

Internal token, no API change. This was decided by the compass lane on RIG-4742; Matt can overrule it there.

Verification

  • Red→green regression tests. Each one fails on main's unconditional cleanup:
    • TestErroredCleanupKeepsSameEnrollmentRebind: pauses the old cleanup, re-promotes without re-enroll, releases. The cache and durable bindings survive and no loss is reported.
    • TestErroredCleanupKeepsRebindDuringDurableDelete: re-bind lands while the durable delete is paused. Fails if the delete is unconditional.
    • TestErroredCleanupKeepsPeerRebind: a peer re-binds the row; the cache is stale. Fails if a not-removed delete is ignored.
    • TestErroredCleanupWithoutDurableWriteKeepsPeerRow: the hub's durable write failed and a peer wrote the row. The row is kept, no loss is reported, and the peer's binding resolves. Fails if a version-less release ignores the competing row.
    • TestErroredCleanupWithoutDurableWriteKeepsLegacyRow: a version-less release meets a row with the default '' version. Fails if it calls the versioned delete.
    • TestErroredCleanupDropsStaleAccountForPeerRebind: a peer re-binds the session id to another account mid-cleanup. Fails if the old account still resolves to it.
    • TestErroredCleanupReleasesReverseReadThroughBinding: a binding learned by reverse read goes ERRORED. Fails if the read-through caches no version.
    • TestStopUnbindWaitsForEnrollReap: a Stop races an in-flight reap. Fails if the unbind does not wait for the enroll.
    • TestDeleteSessionBindingByVersionSkipsRebind (pgtest, real Postgres): a stale version deletes nothing and records no usage event; the current version deletes and ends its interval. Fails if the SQL condition or the end event is dropped.
  • go test -race passes for ./server/, ./internal/runner/..., ./internal/runnerhub/ and ./internal/store/.
  • pgtest suites for ./internal/store/, ./internal/runnerhub/ and ./server/ pass against local Postgres 17.
  • golangci-lint reports 0 issues untagged, and nilaway is clean (the pre-push moon ci caught one nil-flow, fixed). With pgtest, none of its findings are in touched lines.
  • squawk and sqruff pass on the migrations; sqlc generate output is committed.

Spec-impact: none (server-internal binding release; no API or documented state change).
Ledger-impact: none

@linear-code

linear-code Bot commented Oct 6, 2026

Copy link
Copy Markdown

RIG-4742

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Compass engineering docs preview: https://compass-runner-4742-binding.compass-eng-docs.pages.dev

Deployed from compass-runner/4742-binding-lifetime-token at 260e55e.

@mattwilkinsonn
mattwilkinsonn added this pull request to stack #1824 October 7, 2026 00:25
@rigel-mintaka
rigel-mintaka force-pushed the compass-runner/4742-binding-lifetime-token branch from 9078fe7 to 6427ae0 Compare October 7, 2026 03:00
rigel-mintaka and others added 7 commits October 6, 2026 23:50
…-4742)

The detached ERRORED cleanup unbound whatever row held the session id when
it ran, so a resume that re-bound the same id in the same enrollment could
be unbound and archived by the old lifetime's cleanup.

A binding now carries the durable row version (xmin) and a cache lifetime.
The cleanup passes the binding it saw; dropLostSession skips a different
one, and releaseSession re-checks the cache around a delete conditioned on
that version. DeleteSessionBinding takes a version and reports a removal,
so a peer's re-bind is also left alone.

Co-authored-by: Matt Wilkinson <matt@rigel.build>
…G-4742)

xmin is a 32-bit transaction id and repeats after wraparound, so an old
cached version could match a later re-bind. binding_version is a fresh UUID
per upsert. The conditional release is a separate DeleteSessionBindingVersion,
and DeleteSessionBinding is back to its unconditional form for Stop. A cached
binding whose durable write failed has an empty version, which matches no
row, so its cleanup cannot delete a peer's row.

Co-authored-by: Matt Wilkinson <matt@rigel.build>
A cached binding whose durable write failed has an empty version. A legacy
row also has the column default '', so a versioned delete with '' could
remove it. The limited release now skips the store for an empty version.
The removed check no longer reads only, which nilaway could not prove
non-nil.

Co-authored-by: Matt Wilkinson <matt@rigel.build>
…ll versions (RIG-4742)

A cached binding with no durable version now checks for a competing row
before releasing. If one exists, the session was re-bound elsewhere: the
stale cache entry is dropped and no loss is reported. The migration also
gives pre-existing rows a real version, so no row keeps the '' default.

Co-authored-by: Matt Wilkinson <matt@rigel.build>
…sion (RIG-4742)

A cleanup that found its row re-bound evicted only the session's cache
entry. The reverse account entry still named the session, so delivery for
the old account could reach the new owner's session. Evict both.

Co-authored-by: Matt Wilkinson <matt@rigel.build>
A stale-version delete records nothing; the current-version delete ends
its open interval.

Co-authored-by: Matt Wilkinson <matt@rigel.build>
…h enroll (RIG-4742)

A binding learned through SessionForAccount had no version or lifetime, so
its ERRORED cleanup took the version-less path, found its own row, and
never released or reported it. The reverse read now returns the version.

A Stop's release now takes bindingWriteMu, as promotion does since the
rebase onto the enroll serialization, so its delete cannot land between
an enroll's map-clear and its reap and hide the session from the sweep.

Co-authored-by: Matt Wilkinson <matt@rigel.build>
@rigel-mintaka
rigel-mintaka force-pushed the compass-runner/4742-binding-lifetime-token branch from 6427ae0 to 260e55e Compare October 7, 2026 03:56
@rigel-mintaka
rigel-mintaka marked this pull request as ready for review October 7, 2026 05:13

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant