Skip to content

Worker tests race under load: TestAuthorizationRevoked and TestCrashBeforeReleaseRecorded #1448

Closed

Description

@privateip

Parent: Deliver Galactic Worker with Local and Fabric delivery for tenant services

Observed

Two internal/worker tests on integration/galactic-worker (at a639f63) fail intermittently when the machine is loaded. The failures reproduce on a639f63 itself, so later PRs did not introduce them.

  • TestAuthorizationRevoked can hang until the package timeout. Its Observe stream can subscribe after the revoked claim is already gone, so the withdrawal it waits for never arrives.
  • TestCrashBeforeReleaseRecorded can find the releasing record already deleted before the simulated restart, so the restart path it means to test does not run.

Both were seen during local task test:unit runs while delivering #1434. They will intermittently fail CI for every later PR in this epic.

Expected

Both tests pass deterministically under load, and each still checks the behavior it names: withdrawal on revocation, and recovery when a crash happens before the release is recorded.

Acceptance

  • Each test synchronizes on the event it waits for instead of on timing. For example, Observe returns the current state when it subscribes, or the test waits until the subscription exists before revoking. The release path gets a hook that stops it before the record is removed.
  • go test -race -count=50 ./internal/worker/... passes while stress (or an equivalent CPU load) runs. The PR includes the command and its output.
  • Reverting the fix makes the tests fail again under that load. The PR records the result.
  • The delivering PR targets integration/galactic-worker, carries this evidence, and says Fixes datum-cloud/galactic#1448. GitHub does not auto-close from a non-default base, so close this issue explicitly when the PR merges, and record the PR and merged SHA.

Activity

  1. added theissue type on Nov 19, 2026
  2. privateip commented on Nov 19, 2026

    @privateip
    CollaboratorAuthor

    Triage on integration/galactic-worker at a639f63: both races reproduce, and both are in the tests' synchronization, not in Worker.

    Reproduction

    Built with go test -race -c ./internal/worker/ and run with 24 busy-loop CPU burners on a 12-core machine:

    Test Runs Result
    TestAuthorizationRevoked 40 processes, 20 s timeout each 16 hung in stream.Recv() (recovery_test.go:260)
    TestCrashBeforeReleaseRecorded -count=200 21 failed: recovery_test.go:120: restored status = <nil>, want Withdrawing

    Root cause

    • TestAuthorizationRevoked: the client's Observe call returns before the server handler registers its subscriber. The test revokes at once. When the withdrawal finishes first, the handler's initial snapshot no longer contains the claim, so the stream never sends the release and Recv blocks. Observe already sends the current state when it subscribes; there is just nothing to send for a claim that is gone. docs/worker/contract.md promises complete snapshots and nothing about claims already deleted, so Worker's behavior is correct and unchanged.
    • TestCrashBeforeReleaseRecorded: this differs from the description above. The releasing record survives the crash. It is removed after the restart: the new incarnation's first reconcile confirms the rows are gone and deletes the record before the test reads the status, which is then nil. Separately, e.idle() checks only the queue length, which is zero while a pass is still running, so it did not prove the failing Delete had run.

    Fix

    Test-only, in internal/worker/recovery_test.go:

    • TestAuthorizationRevoked receives the first Observe snapshot, which shows the claim Ready, before revoking. The server sends it only after subscribing.
    • TestCrashBeforeReleaseRecorded waits until the injected Delete failure is used. It then restarts with the fake Local module's existing Plan pause, which holds the release before the record is removed. While paused, it checks Withdrawing and the restored releasing record, then resumes and waits for the release to finish.

    With the fix applied, under the same load: 40/40 and 200/200 passed. The PR will carry a go test -race -count=50 ./internal/worker/... run under load, and the same load with the fix reverted.

  3. privateip commented on Nov 19, 2026

    @privateip
    CollaboratorAuthor

    Scope change before the PR. Running the acceptance command, go test -race -count=50 ./internal/worker/... under load, found two more races that block it, and one of them is in Worker itself.

    Observe could block forever before its first snapshot (product bug)

    Observe registers its subscriber, releases the lock, and then sends to its own notify channel, which has room for one signal, to queue the first snapshot. If a status is published in between, fanout fills the channel first and the handler blocks on a send that only it could receive. The stream sends nothing, ignores cancellation, and the subscriber is never removed. A goroutine dump of a hung TestObserveDeliversCompleteSnapshots showed the handler waiting on that send for 27 minutes. One of the 17 TestAuthorizationRevoked hangs in the earlier reproduction was the same deadlock, so the test fix alone would not have cleared it.

    The fix makes the first send non-blocking. The pending map already holds the latest snapshot, so one wake-up is enough. This restores the documented behavior (Observe streams status snapshots, contract.md) and changes no contract text. A new test publishes a status in that window through a nil-by-default hook on Worker. It fails with DeadlineExceeded without the fix and passes with it.

    Two more test races

    • TestCrashBeforeProgramming: crash() cancels the context and then opens the fake module's Plan gate. If the reconciler reaches its select after both are ready, Go picks either case, so Plan could continue and Apply could write rows after the crash. The fake's Plan and Apply now check the context again after the gate opens. Under load, it failed 27 of 1000 runs before the fix and 0 of 1000 after.
    • TestObserveDeliversCompleteSnapshots: when the other caller's 300 ms stream deadline passes, the server ends that stream cleanly. Under load, that end can reach the client before its own deadline error, so Recv returns EOF instead of DeadlineExceeded. The test checks that nothing is delivered, so it now accepts either ending as long as no message arrived. It failed 2 of 318 completed runs before the fix.

    The PR carries all of this with the stress and revert evidence.

  4. privateip commented on Nov 19, 2026

    @privateip
    CollaboratorAuthor

    Fixed by #1454, merged into integration/galactic-worker as ecfb4ba.

    The Worker tests now pass 50 repetitions under heavy CPU load, and the same run hangs with the change reverted. The commands and counts are in the PR's evidence comment. Beyond the two tests named here, the fix covers the Observe first-snapshot deadlock and two more racy tests described in the comment above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions