79 co-located test files are typechecked by nothing — `typecheck:tests` is a glob over `packages/*/tsconfig.test.json`, and `packages/web-search` shipped 16 files outside it

[sroussey]

What

package.json:32:

"typecheck:tests": "for f in packages/*/tsconfig.test.json; do echo \"typecheck $f\" && tsc -p \"$f\" || exit 1; done"

That glob is the only thing that typechecks a co-located test. A workspace with co-located tests and no tsconfig.test.json is silently outside it — and packages/web-search, the window's new package, is exactly that. Its own tsconfig.json excludes them:

$ tail -2 packages/web-search/tsconfig.json
  "exclude": ["dist", "node_modules", "src/**/__tests__/**", "src/**/*.test.ts"]
}
$ ls packages/web-search/tsconfig.test.json
ls: cannot access 'packages/web-search/tsconfig.test.json': No such file or directory

So its 16 test files are excluded from the build project and invisible to the test typecheck. Measured across the tree:

$ for d in packages/* providers/* examples/*; do [ -d "$d/src" ] || continue
    n=$(find $d/src -name '*.test.ts' | wc -l); [ "$n" -eq 0 ] && continue
    t=no; [ -f "$d/tsconfig.test.json" ] && t=YES
    printf "%-42s %4d  %s\n" "$d" "$n" "$t"; done

packages/ai                                  40  YES
packages/job-queue                           14  YES
packages/knowledge-base                       3  YES
packages/mcp                                 11  YES
packages/storage                             31  YES
packages/task-graph                         110  YES
packages/test                               441  no   (its own tsconfig includes src/**/*)
packages/util                                41  YES
packages/web-search                          16  no
providers/anthropic                           1  no
providers/google-gemini                       1  no
providers/openai                              1  no
providers/openrouter                          1  no
examples/cli                                 44  no
examples/eval                                15  no
examples/web                                  3  no

79 files (16 + 4 + 44 + 15 + 3) are typechecked by no gate. examples/* is doubly invisible: it is also absent from scripts/typecheck-budget.json, which prints no examples/ row at all.

Proof — planted, then measured

$ printf '\nconst PLANTED_TYPE_ERROR: number = "not a number";\nvoid PLANTED_TYPE_ERROR;\n' \
    >> packages/web-search/src/__tests__/limitResults.test.ts

$ bun run typecheck:tests
typecheck packages/ai/tsconfig.test.json
typecheck packages/job-queue/tsconfig.test.json
typecheck packages/knowledge-base/tsconfig.test.json
typecheck packages/mcp/tsconfig.test.json
typecheck packages/storage/tsconfig.test.json
typecheck packages/task-graph/tsconfig.test.json
typecheck packages/util/tsconfig.test.json
EXIT=0

$ bunx tsc -p packages/web-search/tsconfig.json --noEmit ; echo $?
0

$ bunx vitest run packages/web-search/src/__tests__/limitResults.test.ts
 Test Files  1 passed (1)
      Tests  4 passed (4)

const x: number = "not a number" in a shipped package's test file survives typecheck:tests, tsc -p on the package, and the test run. vitest transpiles without typechecking, so nothing anywhere sees it.

Why it matters

This is the same class 6ddf7d41 found and fixed in the 2026-08-10 window — "tsconfig.test.json was referenced by nothing, leaving ~330 relocated files type-checked nowhere" — reintroduced by the next new package, because the fix was a glob rather than a guard. The failure mode is quiet: build passes (tests are excluded from the build project), test passes (vitest transpiles), lint passes (oxlint reports rules, not TS diagnostics), and only a human reading the file notices.

3c0480d91's own follow-up commit is the live example of the cost when the gate does exist: "typecheck-budget runs typecheck:tests, which I had not run before pushing — lint and the suites were green, and vitest transpiles without typechecking, so nothing local caught it." For packages/web-search there is no equivalent backstop at all.

Proposed fix

1. Add packages/web-search/tsconfig.test.json, copying packages/mcp/tsconfig.test.json verbatim (it is 12 lines and already parameterless).
2. Replace the shell glob with a guard that derives the set the way scripts/lib/testDiscovery.ts already derives test sections: for every workspace under packages/, providers/ and examples/ that contains a *.test.ts under src/, require a tsconfig.test.json and run it. A workspace with co-located tests and no project file should fail the job, not be skipped by it — the current glob cannot tell "no tests" from "no project".
3. That guard belongs in the existing typecheck-budget job (.github/workflows/test.yml:31-54), which already calls typecheck:tests.

Measured on origin/main @ 2d36880 (0.6.0). bun run typecheck:tests takes 21.6s across the seven projects it currently covers, so adding six more is cheap.

Found during the 2026-09-14 review of packages/.

[sroussey]

Fixed on claude/upbeat-feynman-59uare @ 68ef6e5a8.

What landed

scripts/typecheck-tests.ts replaces the shell glob (package.json:29 is now "typecheck:tests": "bun ./scripts/typecheck-tests.ts"). It derives the set rather than globbing it:

• walks each workspace's src for .test.ts and .test.tsx, reusing walkFiles from scripts/lib/testDiscovery.ts (generalized to a suffix list — the one-word change A .test.tsx is invisible to test discovery *and* to the independent oracle that exists to catch discovery holes — 6th cycle, and the coverage rewrite now globs {ts,tsx} while discovery does not #950 asks for in this walk; discovery's own walks and its oracle are untouched, so A .test.tsx is invisible to test discovery *and* to the independent oracle that exists to catch discovery holes — 6th cycle, and the coverage rewrite now globs {ts,tsx} while discovery does not #950 stands);
• reads every tsc program's resolved file list from tsc --showConfig, so include/exclude are expanded by the compiler rather than re-implemented by a guard that could then agree with itself and disagree with tsc;
• fails, naming the workspace and a count, on a test file that is in no program anything runs.

Two shapes satisfy the rule, and the difference is which gate runs the program:

1. a tsconfig.test.json — this gate runs it;
2. the workspace's own tsconfig.json where it includes the tests and the workspace declares build-types, which is what runs that program.

scripts/lib/typecheckTests.ts holds the pure decision (planTypecheckTests, formatGaps), with scripts/typecheckTests.test.ts covering it — same shape as require-green-ci.ts / lib/cleanTree.ts / cleanTree.test.ts.

New project files: packages/web-search, providers/{anthropic,google-gemini,openai,openrouter} (all five byte-identical to packages/mcp/tsconfig.test.json, as proposed), plus scripts/tsconfig.test.json.

Coverage: 13 test projects over 17 workspaces, against the 7 the glob reached.

One correction to the measurement

The 79 is 20, not 79 — the other 59 (and packages/test's 441) are checked today, by their own build program. examples/{cli,eval,web} do not exclude tests from tsconfig.json ("exclude": ["dist"]), and each declares build-types, which bun run build:types runs in both the lint and build jobs. Measured rather than assumed, per workspace:

$ printf '\nconst PLANTED_TYPE_ERROR: number = "not a number";\nvoid PLANTED_TYPE_ERROR;\n' \
    >> examples/cli/src/ui/model/progressBar.test.ts
$ rm -f examples/cli/tsconfig.tsbuildinfo && tsc -p examples/cli/tsconfig.json
examples/cli/src/ui/model/progressBar.test.ts(33,7): error TS2322: Type 'string' is not assignable to type 'number'.
EXIT=2

Same result for examples/eval, examples/web and packages/test (441 files, encode.test.ts planted, error TS2322). So they get no second project file — a duplicate program would pay for packages/test's 2.4M instantiations twice in the same CI job that already measures them — but the guard now verifies that coverage per file instead of taking it on trust: flip examples/cli's exclude to hide its tests, or drop its build-types, and the job goes red.

The genuinely unchecked set was packages/web-search 16 + one each in the four providers = 20.

One workspace the issue did not count

scripts/ has 10 test files (now 11) and no program at all — its own tests, including this guard's. scripts/tsconfig.test.json brings them in, and doing so turned up the only real type error in the newly-checked files:

scripts/testDiscovery.test.ts(101,7): error TS2345: Argument of type '(p: { test: { root: string; }; }) => string'
  is not assignable to parameter of type '(value: TestProjectConfiguration, index: number, array: TestProjectConfiguration[]) => string'.

A vitest project may be written as a path string, and the roots were read as if the entry were always an object. Fixed by narrowing to the inline shape the config actually builds, the way the sibling test in the same file already does — no any, no @ts-expect-error. The other 19 newly-checked files are clean, and nothing is excepted.

Fail-proof 1 — the issue's exact planted error now goes red

Where the issue measured EXIT=0:

$ printf '\nconst PLANTED_TYPE_ERROR: number = "not a number";\nvoid PLANTED_TYPE_ERROR;\n' \
    >> packages/web-search/src/__tests__/limitResults.test.ts
$ bun run typecheck:tests
...
✔ typecheck packages/util/tsconfig.test.json (0.7s)
packages/web-search/src/__tests__/limitResults.test.ts(37,7): error TS2322: Type 'string' is not assignable to type 'number'.
✖ typecheck packages/web-search/tsconfig.test.json (0.7s)
✔ typecheck providers/anthropic/tsconfig.test.json (0.7s)
✔ typecheck providers/google-gemini/tsconfig.test.json (0.7s)
✔ typecheck providers/openai/tsconfig.test.json (0.7s)
✔ typecheck providers/openrouter/tsconfig.test.json (0.7s)
✔ typecheck scripts/tsconfig.test.json (1.4s)

✖ typecheck failed for:
    packages/web-search/tsconfig.test.json

EXIT=1

Every project runs before the verdict, so a second error does not cost another CI round trip.

Fail-proof 2 — a deleted project file fails naming the workspace, it is not skipped

This is the half the glob could not do at all:

$ mv packages/web-search/tsconfig.test.json /tmp/
$ bun run typecheck:tests
  providers/openrouter                 1 test file(s) ← providers/openrouter/tsconfig.test.json
  scripts                             10 test file(s) ← scripts/tsconfig.test.json

✖ 16 co-located test file(s) in 1 workspace(s) are typechecked by nothing:

  packages/web-search: 16 test file(s) typechecked by nothing
    no typecheck program covers them — add packages/web-search/tsconfig.test.json
      packages/web-search/src/__tests__/BraveWebSearchProvider.test.ts
      packages/web-search/src/__tests__/SearxngWebSearchProvider.integration.test.ts
      packages/web-search/src/__tests__/SearxngWebSearchProvider.test.ts
      … and 13 more

  Tests are excluded from each package's build program, vitest transpiles
  without typechecking and oxlint reports rules rather than TS diagnostics,
  so a type error in these files is invisible until someone reads it.
  tsconfig.test.json is the program that checks them — copy one from a
  package that has it, e.g. packages/mcp/tsconfig.test.json.

EXIT=1

Both restored afterwards; the gate is green on the committed tree.

Verification (Node 24.20.0, Bun 1.4.2)

command	exit	time
bun run lint	0	10s (32s cold)
bun run build:types	0	0.2s (turbo cached)
bun run format-check	0	0.3s
bun run typecheck:tests	0	24s cold / 8s warm
bun run typecheck:budget	0	43s, 40 packages within budget
bun scripts/test.ts --check-sections	0	<1s
bun scripts/test.ts scripts (vitest)	0	23s — 9 files, 81 tests
bun scripts/test.ts scripts bun	0	7s — 9 files, 50 tests

24s for 13 projects against the issue's 21.6s for 7.

CI needs no new step: typecheck-budget already runs typecheck:tests, and its comment in .github/workflows/test.yml now says the step also fails on an uncovered workspace rather than skipping it.

---

Generated by Claude Code