Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,13 +36,13 @@ jobs:
node-version: ${{ matrix.node }}
cache: npm
- run: npm ci
- name: Run categorized tests
- name: Run Vitest
run: npm run test:ci
- name: Upload JUnit test results
if: ${{ always() }}
uses: actions/upload-artifact@v4
with:
name: test-results-${{ matrix.os }}-node-${{ matrix.node }}
path: test-results/*.xml
path: test-results/vitest.junit.xml
if-no-files-found: error
- run: npm run pack:verify
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ Adding support for a new Coding Agent host starts with

- All relative `.md` references across `skills/`, `references/`, `templates/`, `models/`, `docs/`, and `case-studies/`
must resolve; `test/skills-docs/doc-link-graph.test.mjs` enforces this in `npm test`.
- After adding, moving, or renaming markdown docs, run `node --test test/skills-docs/doc-link-graph.test.mjs` before committing,
- After adding, moving, or renaming markdown docs, run `npx vitest run test/skills-docs/doc-link-graph.test.mjs` before committing,
and regenerate the routing graph with `node scripts/doc-link-graph/cli.mjs skills/better-harness`
(it rewrites `docs/better-harness-doc-links.mmd`, which the test checks for staleness).
- Every reference doc shipped under `skills/better-harness/references/` must stay reachable from `SKILL.md` routing,
Expand Down
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Useful focused checks:

```bash
# Markdown links and the generated Harness routing graph
node --test test/skills-docs/doc-link-graph.test.mjs
npx vitest run test/skills-docs/doc-link-graph.test.mjs

# Package and runtime-bundle boundaries
npm run pack:verify
Expand Down Expand Up @@ -95,8 +95,8 @@ unrelated cleanup, dependency changes, or local host state into the same change.
Run the smallest relevant tests while developing, then the broader gate justified
by the risk:

- Markdown moves or links: `node --test test/skills-docs/doc-link-graph.test.mjs`.
- Runtime logic: focused `node --test test/<area>.test.mjs`, then `npm test`.
- Markdown moves or links: `npx vitest run test/skills-docs/doc-link-graph.test.mjs`.
- Runtime logic: focused `npx vitest run test/<area>.test.mjs`, then `npm test`.
- Package/runtime roots: `npm run pack:verify`.
- Visual output: preview smoke, console inspection, and screenshot review.
- Cross-platform command changes: verify argv-based execution and avoid
Expand Down
2 changes: 1 addition & 1 deletion docs/adapters/contributing-new-coding-agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,7 @@ moved, regenerate and verify the link graph:

```bash
node scripts/doc-link-graph/cli.mjs skills/better-harness
node --test test/skills-docs/doc-link-graph.test.mjs
npx vitest run test/skills-docs/doc-link-graph.test.mjs
npm test
npm run pack:verify
git diff --check
Expand Down
2 changes: 1 addition & 1 deletion docs/community.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ This is the complete reference. For the common cases, see Start Here above.
| Skill detail and reusable guidance | Yes | `skills/<skill>/references/` or shared `references/` | Topic-scoped Markdown with stable headings and source boundaries | Loaded only when the skill or agent task needs it | Link/path check; consumer grep for any 2+ consumer claim |
| Maturity model | Yes, additive only | `models/<model>.md` plus `models/routing.md` | Stable `model_id`, aliases, audience, levels, dimensions, evidence sources, scoring and confidence rules | Selectable through model routing after routing-file registration | Model-routing checks, fixture/report sample, no mutation of built-in defaults unless intentional |
| Detector or analysis signal | Yes | owning `models/<model>.md`, `scripts/<business-capability>/`, or `skills/<skill>/references/` | Detector or signal id when stable, source evidence, emitted fields, false-positive and downgrade rules | Selected through model, capability, or skill routing; shared prose needs two visible workflow consumers | Fixtures or sample evidence ledger plus behavior, report-quality, or model-routing checks |
| Executable analysis capability | Yes, but only with tests | `scripts/<business-capability>/cli.mjs` | CLI contract, JSON output shape, fixtures, scoped helpers | Called by skills, hooks, reports, or host adapters | `node --test`, fixture assertions, cross-platform command construction |
| Executable analysis capability | Yes, but only with tests | `scripts/<business-capability>/cli.mjs` | CLI contract, JSON output shape, fixtures, scoped helpers | Called by skills, hooks, reports, or host adapters | `npm test` or focused `npx vitest run`, fixture assertions, cross-platform command construction |
| Host evidence adapter | Yes | `scripts/<capability>/platforms/<host>.mjs` and `docs/adapters/README.md` | Matrix row with discovery paths, normalized evidence shape, smoke command, and split trigger if needed | Capability-specific host collection; not packaging | Host matrix entry, smoke command or explicit unavailable note |
| Lifecycle enforcement | Yes, narrowly | `hooks/hooks.json.template` and `hooks/git-scripts/<hook>/` | Hook event, mode, command contract, expected failure behavior | Installed from the template into host lifecycle points | Hook fixture/test or dry-run output; no shell-specific assumptions |
| Reporting template family | Yes, with routing and runtime tests | `templates/reporting/` | `report-structure.md` owns Markdown structure; mode files own metadata, companion rules, imports/props, and validation | Selected by report generation and output mode | Template tests, parser checks, path checks, and visual preview/smoke when relevant |
Expand Down
97 changes: 97 additions & 0 deletions docs/specs/2026-08-13-vitest-migration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Make test failures directly visible

## Traceability

- Spec ID: vitest-migration
- Status: Implemented
- Supersedes the CI runner/reporting slice of
[Make test failures concise and actionable](2026-08-13-test-ci-diagnostics.md).

## Intent

Replace the repository's custom `node:test` CI orchestration with Vitest so a
failed GitHub Actions step names the failing file and test, prints the useful
assertion stack in the main log, links an annotation to the source, and retains
downloadable JUnit evidence. Remove the custom reporting lifecycle that reduced
successful output but hid the identity of failures behind dots and artifacts.

## Acceptance Scenarios

- AC-1: `npm test` runs the complete `test/**/*.test.mjs` population through
Vitest on the declared Node.js versions without requiring tests to adopt
Vitest assertions or mocks.
- AC-2: An intentionally failing isolated test produces a main-log summary
containing its relative file, full test name, assertion message, and source
location; diagnosing the failure does not require opening an artifact.
- AC-3: GitHub Actions uses Vitest's built-in default and `github-actions`
reporters plus its JUnit reporter, uploads the JUnit file on success or
failure, and publishes a test Job Summary.
- AC-4: The repository no longer owns a custom test scheduler or reporter for
ordinary CI execution; `test/support/run-ci.mjs` and
`test/support/github-actions-reporter.mjs` are removed.
- AC-5: Existing Node assertion behavior, dynamic skip behavior, per-test
cleanup, suite setup/teardown, filename filtering, and exit codes remain
observable after migration.
- AC-6: CI passes on Linux Node 22.20.0, macOS Node 22.20.0, Windows Node
22.20.0, and Linux Node 24.x from the exact pushed head.
- AC-7: `npm run pack:verify` continues to exclude test infrastructure and
development dependencies from the published runtime artifact.

## Non-goals

- Rewriting Node `assert` calls to Vitest `expect` matchers.
- Adding coverage thresholds, browser tests, snapshots, or TypeScript test
compilation.
- Changing product behavior or weakening platform-specific tests.
- Keeping capability-by-capability subprocess scheduling when Vitest can report
file and suite ownership directly.

## Plan and Tasks

1. Add the exact stable Vitest version as a development dependency and add
local and CI configurations for the existing `.test.mjs` population.
2. Mechanically replace `node:test` imports with Vitest imports. Map per-test
cleanup from `t.after` to `t.onTestFinished`, retain context `skip`, and map
suite hooks to `beforeAll` and `afterAll`.
3. Replace `npm test` and `npm run test:ci` with Vitest run commands. Configure
the CI route with `default`, `github-actions`, and JUnit reporters so failures
stay human-readable while machine artifacts remain available.
4. Update the GitHub Actions upload boundary for the single Vitest JUnit file
and remove the custom runner and reporter.
5. Validate one isolated intentional failure, focused migrated tests, the full
suite, package verification, documentation links, and the exact-head CI
matrix before release.

Decision rationale: Vitest 4.1.10 supports the repository's Node.js range and
ships the three required reporting surfaces. The test corpus primarily uses
`node:assert/strict`, so the migration changes lifecycle registration while
leaving nearly all assertions and fixtures unchanged.

## Test and Review Evidence

- AC-1/AC-5: `npm test` passes all 91 files and 1,305 Vitest cases. The five
result-count difference from the 1,310 Node baseline is accounted for by
removing four custom-runner component tests and representing the two adapter
cases directly instead of counting their former parent subtest. All 91 test
files import Vitest, and no `node:test`, `t.after`, or `context.test` route
remains. Focused lifecycle coverage passes 79/79 across cleanup, dynamic
skip, and filesystem-heavy suites.
- AC-2: the reporting contract launches Vitest against an intentionally failing
temporary fixture and passes only when the captured main log contains the
relative file, full test name, Node assertion message, and line 3 location.
- AC-3: a focused `npm run test:ci` writes and parses
`test-results/vitest.junit.xml`. A GitHub Actions environment simulation also
writes a non-empty Job Summary with the one-test pass result.
- AC-4: both custom support modules are absent, and no non-historical reference
to either module remains.
- AC-6: the exact-head four-platform PR matrix remains the merge gate; local
evidence does not claim remote Windows, macOS, or Linux status.
- AC-7: `npm run pack:verify` passes with 508 npm entries and 530 runtime zip
entries. Test files, Vitest configuration, and installed development modules
remain outside both artifacts.
- Documentation and installation: `npm ci` succeeds from the lockfile; the
regenerated documentation graph contains 39 files and 56 links, and its
focused suite passes 6/6. `git diff --check` is clean.
- Risk: Vitest file parallelism and cleanup timing can differ from `node:test`.
The fork pool preserves process isolation, cleanup is migrated explicitly,
and the full cross-platform suite remains the release gate.
Loading
Loading