You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Worker tests race under load: TestAuthorizationRevoked and TestCrashBeforeReleaseRecorded #1448
Two internal/worker tests on integration/galactic-worker (at a639f63) fail intermittently when the machine is loaded. The failures reproduce on a639f63 itself, so later PRs did not introduce them.
TestAuthorizationRevoked can hang until the package timeout. Its Observe stream can subscribe after the revoked claim is already gone, so the withdrawal it waits for never arrives.
TestCrashBeforeReleaseRecorded can find the releasing record already deleted before the simulated restart, so the restart path it means to test does not run.
Both were seen during local task test:unit runs while delivering #1434. They will intermittently fail CI for every later PR in this epic.
Expected
Both tests pass deterministically under load, and each still checks the behavior it names: withdrawal on revocation, and recovery when a crash happens before the release is recorded.
Acceptance
Each test synchronizes on the event it waits for instead of on timing. For example, Observe returns the current state when it subscribes, or the test waits until the subscription exists before revoking. The release path gets a hook that stops it before the record is removed.
go test -race -count=50 ./internal/worker/... passes while stress (or an equivalent CPU load) runs. The PR includes the command and its output.
Reverting the fix makes the tests fail again under that load. The PR records the result.
The delivering PR targets integration/galactic-worker, carries this evidence, and says Fixes datum-cloud/galactic#1448. GitHub does not auto-close from a non-default base, so close this issue explicitly when the PR merges, and record the PR and merged SHA.
Triage on integration/galactic-worker at a639f63: both races reproduce, and both are in the tests' synchronization, not in Worker.
Reproduction
Built with go test -race -c ./internal/worker/ and run with 24 busy-loop CPU burners on a 12-core machine:
Test
Runs
Result
TestAuthorizationRevoked
40 processes, 20 s timeout each
16 hung in stream.Recv() (recovery_test.go:260)
TestCrashBeforeReleaseRecorded
-count=200
21 failed: recovery_test.go:120: restored status = <nil>, want Withdrawing
Root cause
TestAuthorizationRevoked: the client's Observe call returns before the server handler registers its subscriber. The test revokes at once. When the withdrawal finishes first, the handler's initial snapshot no longer contains the claim, so the stream never sends the release and Recv blocks. Observe already sends the current state when it subscribes; there is just nothing to send for a claim that is gone. docs/worker/contract.md promises complete snapshots and nothing about claims already deleted, so Worker's behavior is correct and unchanged.
TestCrashBeforeReleaseRecorded: this differs from the description above. The releasing record survives the crash. It is removed after the restart: the new incarnation's first reconcile confirms the rows are gone and deletes the record before the test reads the status, which is then nil. Separately, e.idle() checks only the queue length, which is zero while a pass is still running, so it did not prove the failing Delete had run.
Fix
Test-only, in internal/worker/recovery_test.go:
TestAuthorizationRevoked receives the first Observe snapshot, which shows the claim Ready, before revoking. The server sends it only after subscribing.
TestCrashBeforeReleaseRecorded waits until the injected Delete failure is used. It then restarts with the fake Local module's existing Plan pause, which holds the release before the record is removed. While paused, it checks Withdrawing and the restored releasing record, then resumes and waits for the release to finish.
With the fix applied, under the same load: 40/40 and 200/200 passed. The PR will carry a go test -race -count=50 ./internal/worker/... run under load, and the same load with the fix reverted.
Scope change before the PR. Running the acceptance command, go test -race -count=50 ./internal/worker/... under load, found two more races that block it, and one of them is in Worker itself.
Observe could block forever before its first snapshot (product bug)
Observe registers its subscriber, releases the lock, and then sends to its own notify channel, which has room for one signal, to queue the first snapshot. If a status is published in between, fanout fills the channel first and the handler blocks on a send that only it could receive. The stream sends nothing, ignores cancellation, and the subscriber is never removed. A goroutine dump of a hung TestObserveDeliversCompleteSnapshots showed the handler waiting on that send for 27 minutes. One of the 17 TestAuthorizationRevoked hangs in the earlier reproduction was the same deadlock, so the test fix alone would not have cleared it.
The fix makes the first send non-blocking. The pending map already holds the latest snapshot, so one wake-up is enough. This restores the documented behavior (Observe streams status snapshots, contract.md) and changes no contract text. A new test publishes a status in that window through a nil-by-default hook on Worker. It fails with DeadlineExceeded without the fix and passes with it.
Two more test races
TestCrashBeforeProgramming: crash() cancels the context and then opens the fake module's Plan gate. If the reconciler reaches its select after both are ready, Go picks either case, so Plan could continue and Apply could write rows after the crash. The fake's Plan and Apply now check the context again after the gate opens. Under load, it failed 27 of 1000 runs before the fix and 0 of 1000 after.
TestObserveDeliversCompleteSnapshots: when the other caller's 300 ms stream deadline passes, the server ends that stream cleanly. Under load, that end can reach the client before its own deadline error, so Recv returns EOF instead of DeadlineExceeded. The test checks that nothing is delivered, so it now accepts either ending as long as no message arrived. It failed 2 of 318 completed runs before the fix.
The PR carries all of this with the stress and revert evidence.
Fixed by #1454, merged into integration/galactic-worker as ecfb4ba.
The Worker tests now pass 50 repetitions under heavy CPU load, and the same run hangs with the change reverted. The commands and counts are in the PR's evidence comment. Beyond the two tests named here, the fix covers the Observe first-snapshot deadlock and two more racy tests described in the comment above.
Activity
privateip commentedon Nov 19, 2026
Triage on
integration/galactic-workerat a639f63: both races reproduce, and both are in the tests' synchronization, not in Worker.Reproduction
Built with
go test -race -c ./internal/worker/and run with 24 busy-loop CPU burners on a 12-core machine:TestAuthorizationRevokedstream.Recv()(recovery_test.go:260)TestCrashBeforeReleaseRecorded-count=200recovery_test.go:120: restored status = <nil>, want WithdrawingRoot cause
TestAuthorizationRevoked: the client'sObservecall returns before the server handler registers its subscriber. The test revokes at once. When the withdrawal finishes first, the handler's initial snapshot no longer contains the claim, so the stream never sends the release andRecvblocks.Observealready sends the current state when it subscribes; there is just nothing to send for a claim that is gone. docs/worker/contract.md promises complete snapshots and nothing about claims already deleted, so Worker's behavior is correct and unchanged.TestCrashBeforeReleaseRecorded: this differs from the description above. The releasing record survives the crash. It is removed after the restart: the new incarnation's first reconcile confirms the rows are gone and deletes the record before the test reads the status, which is then nil. Separately,e.idle()checks only the queue length, which is zero while a pass is still running, so it did not prove the failingDeletehad run.Fix
Test-only, in
internal/worker/recovery_test.go:TestAuthorizationRevokedreceives the firstObservesnapshot, which shows the claim Ready, before revoking. The server sends it only after subscribing.TestCrashBeforeReleaseRecordedwaits until the injectedDeletefailure is used. It then restarts with the fake Local module's existing Plan pause, which holds the release before the record is removed. While paused, it checksWithdrawingand the restored releasing record, then resumes and waits for the release to finish.With the fix applied, under the same load: 40/40 and 200/200 passed. The PR will carry a
go test -race -count=50 ./internal/worker/...run under load, and the same load with the fix reverted.privateip commentedon Nov 19, 2026
Scope change before the PR. Running the acceptance command,
go test -race -count=50 ./internal/worker/...under load, found two more races that block it, and one of them is in Worker itself.Observe could block forever before its first snapshot (product bug)
Observeregisters its subscriber, releases the lock, and then sends to its ownnotifychannel, which has room for one signal, to queue the first snapshot. If a status is published in between,fanoutfills the channel first and the handler blocks on a send that only it could receive. The stream sends nothing, ignores cancellation, and the subscriber is never removed. A goroutine dump of a hungTestObserveDeliversCompleteSnapshotsshowed the handler waiting on that send for 27 minutes. One of the 17TestAuthorizationRevokedhangs in the earlier reproduction was the same deadlock, so the test fix alone would not have cleared it.The fix makes the first send non-blocking. The pending map already holds the latest snapshot, so one wake-up is enough. This restores the documented behavior (
Observestreams status snapshots, contract.md) and changes no contract text. A new test publishes a status in that window through a nil-by-default hook onWorker. It fails withDeadlineExceededwithout the fix and passes with it.Two more test races
TestCrashBeforeProgramming:crash()cancels the context and then opens the fake module's Plan gate. If the reconciler reaches itsselectafter both are ready, Go picks either case, so Plan could continue and Apply could write rows after the crash. The fake's Plan and Apply now check the context again after the gate opens. Under load, it failed 27 of 1000 runs before the fix and 0 of 1000 after.TestObserveDeliversCompleteSnapshots: when the other caller's 300 ms stream deadline passes, the server ends that stream cleanly. Under load, that end can reach the client before its own deadline error, soRecvreturnsEOFinstead ofDeadlineExceeded. The test checks that nothing is delivered, so it now accepts either ending as long as no message arrived. It failed 2 of 318 completed runs before the fix.The PR carries all of this with the stress and revert evidence.
privateip commentedon Nov 19, 2026
Fixed by #1454, merged into
integration/galactic-workeras ecfb4ba.The Worker tests now pass 50 repetitions under heavy CPU load, and the same run hangs with the change reverted. The commands and counts are in the PR's evidence comment. Beyond the two tests named here, the fix covers the
Observefirst-snapshot deadlock and two more racy tests described in the comment above.