Skip to content

fix(worker): Deliver the first status snapshot and stabilize tests - #1454

Merged
privateip merged 1 commit into
integration/galactic-workerfrom
fix/issue-1448
Nov 19, 2026
Merged

privateip merged 1 commit into
integration/galactic-workerfrom
fix/issue-1448

Conversation

@privateip

@privateip privateip commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1448

A Worker status stream could hang forever before sending its first update, when a status change arrived while the stream was starting. The stream now always sends its first update. Four Worker tests that relied on timing now wait for the event they check. Under heavy CPU load the Worker tests pass 50 repetitions in a row, and with this change reverted the same run hangs.

Test plan

  • Worker tests, 50 repetitions under 24 busy CPU loops on 12 cores: two runs pass at this head.
  • Same run with this change reverted: hangs until the 10-minute timeout.
  • Each racy test, repeated under the same load: between 2 and 30 failures or hangs per test before the fix, none after. The counts are in the first comment.
  • New stream test fails every time without the fix and passes with it. Unit tests and lint on the changed code pass.

Observe registered its subscriber, released the lock, then sent to its
own notify channel, which holds one signal. A status published in that
window filled the channel first, so the handler blocked on a send only
it could receive: the stream sent nothing, ignored cancellation and
never removed its subscriber. The first send is now non-blocking, since
pending already holds the latest snapshot. A nil-by-default hook lets
TestObservePublishBeforeFirstSnapshot publish in that window; it fails
with DeadlineExceeded without the fix.

Four tests raced under CPU load:

- TestAuthorizationRevoked revoked the claim as soon as the client's
  Observe call returned, before the server subscribed. It now receives
  the first snapshot before revoking.
- TestCrashBeforeReleaseRecorded lost its record after the restart,
  when the new incarnation's first reconcile finished the release
  before the test read the status. The restart now holds that reconcile
  at Plan with the fake module's pause, checks the restored status and
  record, then resumes. It also waits for the injected Delete failure
  to be used instead of for an empty queue.
- TestCrashBeforeProgramming: crash() cancels the context and then
  opens the fake's Plan gate, and select could pick the gate, letting
  Apply write rows after the crash. The fake's Plan and Apply now check
  the context again after the gate opens.
- TestObserveDeliversCompleteSnapshots: the server ends a stream
  cleanly at its deadline, which can reach the client as EOF before
  the client's DeadlineExceeded. The test now accepts a clean end once
  the deadline has passed, when no message arrived.

Fixes #1448
@privateip privateip self-assigned this Nov 19, 2026
@privateip

Copy link
Copy Markdown
Collaborator Author

Details and evidence for this PR.

go test -race ./internal/worker/... failed intermittently under CPU load. Four tests raced, and one of the hangs was a real deadlock in Observe.

Observe deadlock (product fix)

Observe registered its subscriber, released w.mu, then sent to its own notify channel (capacity 1) to queue the first snapshot. A status published in that window filled the channel first, so the handler blocked on a send that only it could receive. The stream sent nothing, ignored cancellation and kept its subscriber forever. A goroutine dump of a hung test showed the handler at that send for 27 minutes.

The first send is now non-blocking. pending already holds the latest snapshot, so one wake-up is enough. This restores what docs/worker/contract.md documents for Observe and changes no contract text. Worker.subscribed, nil outside tests, lets the new TestObservePublishBeforeFirstSnapshot publish inside that window.

Test races

Test Race Fix
TestAuthorizationRevoked The client's Observe returns before the server subscribes. A fast withdrawal left the stream nothing to send. Receive the first snapshot, sent only after subscribing, before revoking.
TestCrashBeforeReleaseRecorded The record survived the crash, but the restart's first reconcile finished the release before the test read Withdrawing. Hold the restart's reconcile at Plan with the fake's pause, check the restored status and record, then resume. Wait for the injected Delete failure, not an empty queue.
TestCrashBeforeProgramming crash() cancels the context and then opens the fake's Plan gate. select could take the gate, so Apply wrote rows after the crash. The fake's Plan and Apply check the context again after the gate opens.
TestObserveDeliversCompleteSnapshots At its deadline the server ends the other caller's stream cleanly, which can reach the client as EOF before its own DeadlineExceeded. Accept DeadlineExceeded, or EOF within 50 ms of the deadline, when no message arrived. An earlier end still fails.

Evidence

Load: 24 busy-loop CPU burners on a 12-core machine (stress is not installed). Tests were built with go test -race -c ./internal/worker/.

The acceptance command, go test -race -count=50 -timeout=30m ./internal/worker/..., under load at this PR's head:

ok  	go.datum.net/galactic/internal/worker	293.065s
ok  	go.datum.net/galactic/internal/worker/auth	2.955s
ok  	go.datum.net/galactic/internal/worker/sources	10.196s
ok  	go.datum.net/galactic/internal/worker/store	1.307s

A second run also passed (internal/worker in 287.435s), as did three runs of an earlier revision that differed only in the EOF check below.

The same command with the commit reverted (git revert --no-commit), with -timeout=10m:

panic: test timed out after 10m0s
	running tests:
		TestAuthorizationRevoked (9m50s)
FAIL	go.datum.net/galactic/internal/worker	600.333s

Targeted runs under the same load:

Test Reverted With this change
TestAuthorizationRevoked (40 processes, 20 s timeout each) 14 hung 40/40 pass
TestCrashBeforeReleaseRecorded (-count=200) 30 failed: restored status = <nil>, want Withdrawing 200/200 pass
TestCrashBeforeProgramming (-count=1000) 26 failed: rows before restart = [...] 1000/1000 pass
TestObserveDeliversCompleteSnapshots (-count=300 reverted, -count=600 fixed) 1 failed (EOF), then hung at the timeout after 117 runs 600/600 pass

TestObservePublishBeforeFirstSnapshot fails deterministically when the blocking send is restored (first snapshot: rpc error: code = DeadlineExceeded) and passes with the fix.

task test:unit passes. golangci-lint run --new-from-rev=origin/integration/galactic-worker ./internal/worker/... reports 0 issues. The full task lint reports 201 goconst/lll findings, none in files this PR changes.

@privateip privateip changed the title fix(worker): Status streams no longer hang, and Worker tests stop racing fix(worker): Deliver the first status snapshot and stabilize tests Nov 19, 2026
@privateip
privateip marked this pull request as ready for review November 19, 2026 04:50
@privateip
privateip requested a review from a team as a code owner November 19, 2026 04:50
@privateip
privateip merged commit ecfb4ba into integration/galactic-worker Nov 19, 2026
28 checks passed
@privateip
privateip deleted the fix/issue-1448 branch November 19, 2026 04:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant