Repository navigation
fix(worker): Deliver the first status snapshot and stabilize tests - #1454
Conversation
Observe registered its subscriber, released the lock, then sent to its own notify channel, which holds one signal. A status published in that window filled the channel first, so the handler blocked on a send only it could receive: the stream sent nothing, ignored cancellation and never removed its subscriber. The first send is now non-blocking, since pending already holds the latest snapshot. A nil-by-default hook lets TestObservePublishBeforeFirstSnapshot publish in that window; it fails with DeadlineExceeded without the fix. Four tests raced under CPU load: - TestAuthorizationRevoked revoked the claim as soon as the client's Observe call returned, before the server subscribed. It now receives the first snapshot before revoking. - TestCrashBeforeReleaseRecorded lost its record after the restart, when the new incarnation's first reconcile finished the release before the test read the status. The restart now holds that reconcile at Plan with the fake module's pause, checks the restored status and record, then resumes. It also waits for the injected Delete failure to be used instead of for an empty queue. - TestCrashBeforeProgramming: crash() cancels the context and then opens the fake's Plan gate, and select could pick the gate, letting Apply write rows after the crash. The fake's Plan and Apply now check the context again after the gate opens. - TestObserveDeliversCompleteSnapshots: the server ends a stream cleanly at its deadline, which can reach the client as EOF before the client's DeadlineExceeded. The test now accepts a clean end once the deadline has passed, when no message arrived. Fixes #1448
|
Details and evidence for this PR.
Observe deadlock (product fix)
The first send is now non-blocking. Test races
EvidenceLoad: 24 busy-loop CPU burners on a 12-core machine ( The acceptance command, A second run also passed ( The same command with the commit reverted ( Targeted runs under the same load:
|
Fixes #1448
A Worker status stream could hang forever before sending its first update, when a status change arrived while the stream was starting. The stream now always sends its first update. Four Worker tests that relied on timing now wait for the event they check. Under heavy CPU load the Worker tests pass 50 repetitions in a row, and with this change reverted the same run hangs.
Test plan