Skip to content

test: full-suite subprocess-spawn contention causes 12 intermittent failures across 7 files #332

Description

@dean0x

Problem

A full npm test (4103 tests / 114 files) produced 12 failures across 7 files. All 7 files pass 3/3 in isolation — 21 isolated runs, zero failures. The failures are load-induced subprocess-spawn contention, not logic bugs.

Affected Files and Symptoms

File Failures Type
tests/redact-secrets.test.ts 3 assertion failures (spawned script exit 1 instead of 0) + 1 timeout Hard assertion + timeout
tests/decisions/ledger-ops.test.ts 2 × 5s timeouts Timeout
tests/shell-hooks.test.ts 2 × 5s timeouts (json-helper.js operations describe) Timeout
tests/eager-memory-refresh.test.ts 2 × 5s timeouts (this file spawns real claude) Timeout
tests/decisions/decisions-usage-scan.test.ts timing violation: 11088ms against 8000ms limit Timing
tests/build-mds.test.ts beforeAll hook timeout at 10000ms Timeout
tests/compliance-e2e.test.ts beforeEach hook timeout at 10000ms Timeout

Open Thread

The redact-secrets exit-1 failures were not root-caused — they were dismissed on the strength of 3/3 clean isolated runs (inference from absence). If that file ever fails in CI, run the underlying redaction script by hand with the same fixture and capture stderr before assuming contention.

Evidence

All 7 files pass in isolation (21 clean runs). The pattern is consistent with vitest's default unlimited concurrency saturating the subprocess pool when the full suite runs.

Filing this so a future CI failure is not re-diagnosed from scratch.

Fix Directions

  • Raise per-test timeouts on spawn-heavy files (ledger-ops, shell-hooks, redact-secrets, eager-memory-refresh, build-mds, compliance-e2e)
  • Cap vitest concurrency for spawn-heavy files via pool: 'forks' + poolOptions.forks.maxForks in their vitest config, or use --reporter=verbose --sequence.concurrent=false for those files
  • Separate spawn-heavy tests into their own vitest project with a lower concurrency cap

Found during review of PR #327.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    tech-debtTechnical debt items to address

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions