Repository navigation
fix(runnerhub): ignore lifecycle frames that arrive after ERRORED (RIG-4452) - #1774
Open
rigel-mintaka wants to merge 9 commits into
Open
rigel-mintaka wants to merge 9 commits into
rigel-mintaka wants to merge 9 commits into
Conversation
|
❌ This stack could not start testing because there was a merge conflict. See more details here.
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
|
Compass engineering docs preview: https://compass-runner-4452-errored.compass-eng-docs.pages.dev Deployed from |
rigel-mintaka
force-pushed
the
compass-runner/4452-errored-terminal-guard
branch
from
October 6, 2026 20:02
cb0334a to
52241ed
Compare
rigel-mintaka
marked this pull request as ready for review
October 6, 2026 22:35
mattwilkinsonn
added this pull request to stack #1824
October 7, 2026 00:25
mattwilkinsonn
approved these changes
Oct 7, 2026
rigel-mintaka
force-pushed
the
compass-runner/4452-errored-terminal-guard
branch
from
October 7, 2026 03:00
73feb00 to
eb4718e
Compare
…G-4452) When the Runner's shared-stream ERRORED send stalls, it falls back to a one-shot stream. Frames still buffered on the cancelled shared stream can then be delivered after ERRORED and republish the session as live. The hub now marks a session ERRORED and drops its later lifecycle frames. A lifecycle lock serializes the guard with the status publish, so an in-flight frame cannot publish after ERRORED. A recovery command (resume Start, Reload) clears the mark under the same lock as the command's queue admission, so a command that never reached the Runner leaves it set. A re-enroll clears all marks. Trace frames still relay. Co-authored-by: Matt Wilkinson <matt@rigel.build>
…ssion (RIG-4673) A recovery command clearing the guard at admission left a window before the Runner ran it, in which a dead-lifetime frame could still publish. The hub now records ERRORED's RunnerSeq per session and drops lifecycle frames at or below it. RunnerSeq is Runner-wide and monotonic, so new-lifetime frames pass on their own; re-enroll resets the counter and clears the map. The recovery- admission path (relayRecovery, router admit) is removed. Co-authored-by: Matt Wilkinson <matt@rigel.build>
…ollment (RIG-4673) Each container socket built its own Gateway counter, so a resumed session's new socket restarted RunnerSeq at 1 and fell under the hub's ERRORED boundary. agentHost now passes one SeqCounter to every Gateway, matching the proto's per-Runner contract. The hub also stamps an enrollment generation, so an ERRORED delivery paused across a re-enroll cannot reinstall a boundary the re-enroll cleared. The stale-frame test now covers settle and presence edges. Co-authored-by: Matt Wilkinson <matt@rigel.build>
…eqs close gaps (RIG-4673) PublishEvents now stamps the hub's enrollment generation when the stream opens. A session frame from an older stream is dropped before the tail relay, lifecycle edges, and ERRORED's lost-session cleanup, so a resumed session cannot be unbound by its dead process's late ERRORED. Container Gateways share one RunnerSeq counter on separate streams, so a lower seq can arrive after a higher one. The hub now tracks skipped seqs (bounded) and SeenGap reports only those still unseen. Co-authored-by: Matt Wilkinson <matt@rigel.build>
…cking per enrollment (RIG-4673) An RWMutex now fences session-frame delivery (read) against enroll (write), so the generation check, tail relay, and lifecycle edges cannot straddle a re-enroll. ERRORED's detached cleanup carries its enrollment generation and skips if a re-enroll has happened, so it cannot unbind a re-bound session. Re-enroll resets the RunnerSeq gap tracker, and stale-stream events no longer feed it. Co-authored-by: Matt Wilkinson <matt@rigel.build>
…est (RIG-4673) Separate streams can deliver one session's frames out of seq order, so an ERRORED at seq 5 could land after the resumed READY at 6 and retire it. The hub now keeps the highest accepted lifecycle seq per session and drops any older lifecycle frame, ERRORED included, before the tail relay. Deliver holds the enrollment read lock for the whole event, so a re-enroll cannot reset sequence state between the generation check and its use. Co-authored-by: Matt Wilkinson <matt@rigel.build>
Co-authored-by: Matt Wilkinson <matt@rigel.build>
…generation (RIG-4673) A lifecycle frame now takes lifecycleMu before its seq is recorded, so a lower seq cannot overtake a higher one that is still mid-delivery. Sessions reads its router and enrollment generation as one pair, and enroll holds the write lock until the new router is installed. Generations start at 1, so a PublishEvents stream opened before the first Enroll is fenced too. lifecycleSeqs becomes a bounded LRU. ERRORED cleanup's generation check is a synchronous helper with its own test. Co-authored-by: Matt Wilkinson <matt@rigel.build>
…(RIG-4673) A cold binding lookup ran under the hub-wide lifecycleMu, so one slow store read stalled lifecycle delivery for every session. Each session now has its own refcounted lock held from seq record through publication. lifecycleMu only guards the seq LRU and the lock map. Co-authored-by: Matt Wilkinson <matt@rigel.build>
rigel-mintaka
force-pushed
the
compass-runner/4452-errored-terminal-guard
branch
from
October 7, 2026 03:56
eb4718e to
ca21ef3
Compare
mattwilkinsonn
approved these changes
Oct 8, 2026
|
Stacked PR 1810 failed testing in the merge queue. Please investigate the failure and re-submit the stack. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is part of a stack containing 2 PRs:
mainSummary
Server-side terminal guard for RIG-4452 (option B), keyed on RunnerSeq per RIG-4673 (option A).
Hub.deliverSessionkeeps the highest accepted lifecycle RunnerSeq for each session, in a bounded LRU purged on re-enroll. Any lifecycle frame at or below it is dropped, ERRORED included. A frame buffered on a cancelled shared PublishEvents stream cannot republish an ERRORED session as live. A delayed old ERRORED cannot kill a resumed lifetime. Trace frames still relay.recordSeqthrough publication. A lower seq therefore cannot overtake a higher one that is still mid-delivery. The lock is per session, so a slow binding lookup stalls only that session.gateway.SeqCounter(Deps.Seq, owned byagentHost). Before this, a resumed session's new socket restarted at 1, which broke the proto's per-Runner contract.Hub.enrollMuserializes delivery againstenroll, and generations start at 1. PublishEvents and Sessions stamp the generation at stream open, and Sessions reads its router and generation as one pair. Frames from an older or pre-enroll stream are dropped. ERRORED's lost-session cleanup skips if a re-enroll has happened since.recordSeqtracks skipped seqs (bounded; overflow stays a gap until re-enroll). A late arrival closes its gap.Rebase onto #1756
Hub.enrollnow takesenrollMu, thenbindingWriteMu(from #1756), thenmu.enrollMuis released once the router is installed;bindingWriteMustays held through the durable reap. Lock order:enrollMu,bindingWriteMu, a session lock,lifecycleMu,mu. Promotion never runs under a session lock, so the order holds.Follow-up
Verification
TestStaleStateAfterErroredIsIgnored, which also checks the settle and presence edges.TestNewLifetimeStateAfterErroredPublishesTestErroredOlderThanNewLifetimeIsIgnoredTestLowerSeqCannotOvertakeHigherSeqMidDeliveryTestSlowBindingLookupStallsOnlyItsSessionTestReenrollClearsErroredBoundaryTestReenrollWaitsForInFlightErroredTestLostSessionCleanupFromOldEnrollmentKeepsRebindingTestFramesFromStreamBeforeReenrollAreDroppedTestSeamPublishEventsOpenedBeforeEnrollIsFencedTestDeliverSequenceGapDetection(late-arrival, re-enroll and overflow subtests)TestSequenceSharedAcrossGateways-count=100.go test -racepasses for./server/,./internal/runner/...and./internal/runnerhub/.internal/runnerhubandinternal/runner/....Spec-impact: none (server-internal ordering and diagnostics; no API or documented state change).
Ledger-impact: none