diff --git a/CLAUDE.md b/CLAUDE.md index 4122301..ab98c63 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -24,7 +24,7 @@ Help is resolved **first**, before either parser runs and before the dynamic `im **`src/changedTests.js`**: `resolveChangedTitles(ref, cwd)` shells out to git (`execFileSync` with an argument array, never a string — a ref is user input) and returns the `it()` titles this branch added or changed, to feed the same filter path `--test` uses. `extractTitles(source)` is the pure half. Test files are identified by the **suffix** `*.twd.test.*`, never by directory: the examples use `src/twd-tests`, `app/twd-tests` and `src/twd-test` between them, and the suffix also keeps a project's Vitest suite — which uses `it()` too — from contributing titles. Diffs to the **working tree** (`git diff `, no second ref) and adds untracked test files, so uncommitted work counts; in CI the tree is clean and this is identical to ` HEAD`. -**`src/parseArgs.js`**: `parseRunArgs(argv)` returns `{ testFilters, changedSince, record, shard, reportDir, noReport, updateSnapshots, ci, cpuThrottle }`. Supports `--test` (repeatable substring filter), `--changed-since`, `--report-dir`, `--no-report`, `--cpu-throttle`, and the recording flags `--record`, `--record-dir`, `--record-speed`, `--record-pace`. Each accepts both `--flag value` and `--flag=value`; a value starting with `--` is refused, so `--test --record` cannot swallow the flag after it. The returned `record` object is passed to `runTests()` as `recordOverrides` and wins over the config file. `parseReportArgs(argv)` parses `twd-cli report [] [--format ]` into `{ input, format }`, defaulting `format` to `markdown` and rejecting anything outside `REPORT_FORMAT_CHOICES`. +**`src/parseArgs.js`**: `parseRunArgs(argv)` returns `{ testFilters, changedSince, record, shard, reportDir, noReport, updateSnapshots, ci, cpuThrottle, repeat }`. Supports `--test` (repeatable substring filter), `--changed-since`, `--report-dir`, `--no-report`, `--cpu-throttle`, `--repeat`, and the recording flags `--record`, `--record-dir`, `--record-speed`, `--record-pace`. `repeat` is `null` when the flag is absent. `--repeat ` accepts a positive integer in either value form and throws otherwise, the way `--cpu-throttle` does. `--repeat` together with `--record`, `--shard`, `--update-snapshots` or `--ci` is refused in the parser, before launch and before the report folder is cleaned; `record.enabled: true` in the config is refused in `runTests` above `cleanReportDir()` with a message naming that config key. While repeating, the page reloads between rounds (ids are random per load, so rounds join on registration index), `retryCount` is forced to 1, `maxFailures` counts distinct tests that have failed rather than failed rounds, and coverage, contract validation and layout snapshots are skipped (the snapshot injection sets `__TWD_SNAPSHOTS__` to false). A test that failed any round fails the run. `run.json` keeps one row per test and adds `runs` and `passes`, plus `repeat` on the run header; the console counts tests, not rounds, and the console block and both report views call out tests that both passed and failed. Each accepts both `--flag value` and `--flag=value`; a value starting with `--` is refused, so `--test --record` cannot swallow the flag after it. The returned `record` object is passed to `runTests()` as `recordOverrides` and wins over the config file. `parseReportArgs(argv)` parses `twd-cli report [] [--format ]` into `{ input, format }`, defaulting `format` to `markdown` and rejecting anything outside `REPORT_FORMAT_CHOICES`. `RUN_FLAGS`, `MERGE_FLAGS` and `REPORT_FLAGS` are the exported list of what each parser recognises, and they do two jobs: `tests/usage.test.js` derives the expected `--help` contents from them, and the "Did you mean" suggestion picks from them. A flag added to a parser has to be added to its list and given a help line in `src/usage.js`, or the suite fails. diff --git a/bin/twd-cli.js b/bin/twd-cli.js index c09567c..6b1afcc 100755 --- a/bin/twd-cli.js +++ b/bin/twd-cli.js @@ -24,7 +24,7 @@ if (command === undefined || command === 'help' || isHelp(command)) { console.log(USAGE[command]()); } else if (command === 'run') { try { - const { testFilters, changedSince, record, shard, reportDir, noReport, updateSnapshots, ci, cpuThrottle } = + const { testFilters, changedSince, record, shard, reportDir, noReport, updateSnapshots, ci, cpuThrottle, repeat } = parseRunArgs(args); const { runTests } = await import('../src/index.js'); const hasFailures = await runTests({ @@ -37,6 +37,7 @@ if (command === undefined || command === 'help' || isHelp(command)) { updateSnapshots, ci, cpuThrottle, + repeat, }); process.exit(hasFailures ? 1 : 0); } catch (error) { diff --git a/docs/cpu-throttling.md b/docs/cpu-throttling.md index 718d78b..dcabba6 100644 --- a/docs/cpu-throttling.md +++ b/docs/cpu-throttling.md @@ -26,16 +26,20 @@ CPU throttling: 6x (the browser runs 6 times slower; the dev server does not). Notes: - **It raises the odds of hitting a race; it does not reproduce one every - time.** Run it in a loop before concluding anything, in either direction: + time.** Repeat the selection in one run before concluding anything, in + either direction. `--repeat` reloads the page between rounds, keeps the + browser open, and names the test that did not pass every time: ```bash - for i in $(seq 10); do - npx twd-cli run --cpu-throttle 6 --test "checkout flow" --no-report > /dev/null && echo pass || echo FAIL - done + npx twd-cli run --repeat 10 --cpu-throttle 6 --test "checkout flow" ``` -- **Turn retries off while you hunt.** The default `retryCount` of `2` gives a - failed test a second attempt, and a flake that passes on the second attempt - is exactly what you are looking for. `retryCount` counts attempts, so - `"retryCount": 1` means no retry. A test that only passed on a retry is + A test that passed 8 of those 10 rounds is listed under `Flaky` with + `(8/10 passed)`. The run exits 1 if any round failed. While repeating, + `--repeat` skips coverage, contracts and layout snapshots. +- **Retries are off while `--repeat` is running.** The default `retryCount` + of `2` gives a failed test a second attempt, and a flake that passes on + the second attempt is exactly what you are looking for, so `--repeat` + forces a single attempt. `retryCount` counts attempts: `"retryCount": 1` + means no retry on an ordinary run. A test that only passed on a retry is listed under `Retried` in the run-complete block. - **Only the browser is slowed.** The dev server, and anything else outside the page, runs at full speed. diff --git a/src/index.js b/src/index.js index d8b0562..59ccefe 100644 --- a/src/index.js +++ b/src/index.js @@ -15,6 +15,7 @@ import { resolveRecordFilename } from './recordFilename.js'; import { resolvePerTestRecording } from './perTestRecording.js'; import { selectShardIds } from './shard.js'; import { parseCpuThrottle } from './cpuThrottle.js'; +import { recordRound, failedTestCount, tallyToStatus } from './repeat.js'; import { buildRunReport } from './runReport.js'; import { cleanReportDir, copySnapshotCaptures, writeReportFolder, @@ -57,6 +58,20 @@ function recordedFileSize(absPath) { } } +// Passed to page.evaluate, so it runs in the page: `window` is the app, not Node. +// Reloading between --repeat rounds mints new test ids, and both loads have to +// read the registry the same way. +function readRegisteredHandlers() { + const state = window.__TWD_STATE__; + if (!state || !state.handlers) return []; + return Array.from(state.handlers.values()).map((h) => ({ + id: h.id, + name: h.name, + parent: h.parent, + type: h.type, + })); +} + export async function runTests(options = {}) { const { testFilters = [], @@ -68,7 +83,10 @@ export async function runTests(options = {}) { updateSnapshots = false, ci = false, cpuThrottle = null, + repeat = null, } = options; + const repeatCount = repeat ?? 1; + const repeating = repeatCount > 1; const sharded = Boolean(shard); let browser; let config; @@ -140,7 +158,8 @@ export async function runTests(options = {}) { executed, notRun: 0, stoppedEarly, - retryCount: config.retryCount, + retryCount: repeating ? 1 : config.retryCount, + repeat: repeating ? repeatCount : null, version: cliVersion(), url: config.url, snapshots, @@ -183,6 +202,30 @@ export async function runTests(options = {}) { if (reportOptions?.unknownFormats.length) { console.warn(`Warning: unknown report format(s) ignored: ${reportOptions.unknownFormats.join(', ')}`); } + // config.record can be a shared default object; copy instead of mutating it. + // Resolved before cleanReportDir: a refused --repeat run must not delete + // the previous report. Flag combinations are refused in parseRunArgs; + // this guards direct runTests callers and the config-only case. + const record = { ...(config.record || {}), ...recordOverrides }; + const recording = Boolean(record.enabled); + if (!record.dir) record.dir = path.join(reportOptions?.dir ?? DEFAULT_REPORT_DIR, RECORDINGS_DIR); + if (repeating) { + const flagConflict = [ + sharded ? '--shard' : null, + recordOverrides?.enabled ? '--record' : null, + ].filter(Boolean); + if (flagConflict.length) { + const error = new Error(`twd-cli run: --repeat cannot be combined with ${flagConflict.join(' and ')}.`); + error.usage = true; + throw error; + } + if (config.record?.enabled) { + const error = new Error('twd-cli run: --repeat cannot run with "record.enabled": true in twd.config.json.'); + error.usage = true; + throw error; + } + } + if (reportOptions) { try { cleanReportDir(path.resolve(workingDir, reportOptions.dir)); @@ -220,19 +263,21 @@ export async function runTests(options = {}) { `Changed since ${changedSince}: ${changedTitles.length} test title(s) to match.` ); } - // config.record can be a shared default object; copy instead of mutating it. - const record = { ...(config.record || {}), ...recordOverrides }; - const recording = Boolean(record.enabled); - if (!record.dir) record.dir = path.join(reportOptions?.dir ?? DEFAULT_REPORT_DIR, RECORDINGS_DIR); - if (recording) { assertFfmpegCapable(record.ffmpegPath, record.format); ffmpegLog = createFfmpegLog(); } - // Load contract validators if configured + // Load contract validators if configured. --repeat skips them: a contract + // checked across n reloads is a different question, and out of scope here. let contractValidators = []; - if (config.contracts && config.contracts.length > 0) { + const contractsConfigured = Boolean(config.contracts && config.contracts.length > 0); + if (repeating) { + console.log('Skipping layout snapshots (--repeat).'); + } + if (repeating && contractsConfigured) { + console.log('Skipping contract validation (--repeat).'); + } else if (contractsConfigured) { contractValidators = await loadContracts(config.contracts, workingDir); } @@ -266,7 +311,7 @@ export async function runTests(options = {}) { // Register mock collector for contract validation const collectedMocks = new Map(); const occurrenceCounters = new Map(); - if (config.contracts && config.contracts.length > 0) { + if (!repeating && config.contracts && config.contracts.length > 0) { await page.exposeFunction('__twdCollectMock', (mock) => { const occKey = `${mock.alias}:${mock.testId}`; const count = (occurrenceCounters.get(occKey) || 0) + 1; @@ -280,13 +325,15 @@ export async function runTests(options = {}) { // evaluateOnNewDocument, never evaluate: this runs before any script on the // page, so the flags are already set by the time matchLayout reads them. // The twdSnapshot vite plugin sets its own flag with ??= precisely so this - // injection wins. __TWD_SNAPSHOTS__ is always on here because twd-cli is - // where a layout snapshot is actually decided. + // injection wins. __TWD_SNAPSHOTS__ is on here (off under --repeat) + // because twd-cli is where a layout snapshot is actually decided. await page.evaluateOnNewDocument((flags) => { - window.__TWD_SNAPSHOTS__ = true; + // Explicit false under --repeat: twd-js defaults this with ??= true, + // so leaving it unset would turn snapshots back on. + window.__TWD_SNAPSHOTS__ = flags.snapshots ?? true; if (flags.update) window.__TWD_UPDATE_SNAPSHOTS__ = true; if (flags.ci) window.__TWD_SNAPSHOT_CI__ = true; - }, { update: updateSnapshots, ci }); + }, { update: repeating ? false : updateSnapshots, ci: repeating ? false : ci, snapshots: !repeating }); // Navigate to your development server startedAt = Date.now(); @@ -301,16 +348,7 @@ export async function runTests(options = {}) { } // Enumerate registered handlers (for the count line and --test filtering) - const registeredHandlers = await page.evaluate(() => { - const state = window.__TWD_STATE__; - if (!state || !state.handlers) return []; - return Array.from(state.handlers.values()).map((h) => ({ - id: h.id, - name: h.name, - parent: h.parent, - type: h.type, - })); - }); + const registeredHandlers = await page.evaluate(readRegisteredHandlers); partialHandlers = registeredHandlers; // The full discovered order, before any filtering. Computed here, ahead of @@ -491,81 +529,135 @@ export async function runTests(options = {}) { partialStatus = []; let recordingFailed = false; let recordingAbandoned = false; - const seenIds = new Set(); - - for (const ids of chunks) { - let chunkClip = null; - if (perTest && !recordingFailed) { - const testNames = ids.map((id) => buildTestPath(id, registeredHandlers)).filter(Boolean); - chunkClip = await startClip(testNames); - } else if (perTest && !recordingAbandoned) { - // Every further clip would stop against the same dead encoder, each one - // racing the 30s deadline, so maxClips stalls replace one. The tests keep - // running: a broken encoder must not swallow their results. - recordingAbandoned = true; - console.error('Recording stopped for the rest of the run. The remaining tests still run, unrecorded.'); - } - const chunkStatus = await page.evaluate(async (retryCount, chunkIds) => { - const TestRunner = window.__testRunner; - const testStatus = []; - const runner = new TestRunner({ - onStart: (test) => { - test.status = "running"; - }, - onPass: (test, retryAttempt) => { - test.status = "done"; - const entry = { id: test.id, status: "pass" }; - if (retryAttempt !== undefined) entry.retryAttempt = retryAttempt; - testStatus.push(entry); - }, - onFail: (test, err) => { - test.status = "done"; - // The raw snapshot travels out; src/failureDiagnostics.js renders - // it in Node. This callback is serialised into the page and has no - // module scope, so it cannot reach a formatter, and duplicating one - // here would be a copy that drifts. `undefined` on a twd-js without - // diagnostics support, and dropped by serialisation. - testStatus.push({ - id: test.id, - status: "fail", - diagnostics: test.diagnostics, - error: `${err.message} (at ${window.location.href})`, - }); - }, - onSkip: (test) => { - test.status = "done"; - testStatus.push({ id: test.id, status: "skip" }); - }, - }, { retryCount }); - await runner.runByIds(chunkIds); - return testStatus; - }, config.retryCount, ids); - - for (const entry of chunkStatus) { - if (seenIds.has(entry.id)) continue; - seenIds.add(entry.id); - partialStatus.push(entry); - } - executed += ids.length; - - let bail = false; - if (config.maxFailures > 0) { - const failed = partialStatus.filter((t) => t.status === 'fail').length; - if (failed >= config.maxFailures) { - stoppedEarly = true; - bail = true; + // One in-page callback for every chunk. --repeat passes attemptCount 1: + // retryCount counts attempts, and a second attempt is the flake this flag + // exists to see. 0 is not "no retries" — the in-page onFail then reads + // err.message off null and the chunk throws. + const runChunk = (ids, attemptCount) => page.evaluate(async (retryCount, chunkIds) => { + const TestRunner = window.__testRunner; + const testStatus = []; + const runner = new TestRunner({ + onStart: (test) => { + test.status = "running"; + }, + onPass: (test, retryAttempt) => { + test.status = "done"; + const entry = { id: test.id, status: "pass" }; + if (retryAttempt !== undefined) entry.retryAttempt = retryAttempt; + testStatus.push(entry); + }, + onFail: (test, err) => { + test.status = "done"; + // The raw snapshot travels out; src/failureDiagnostics.js renders + // it in Node. This callback is serialised into the page and has no + // module scope, so it cannot reach a formatter, and duplicating one + // here would be a copy that drifts. `undefined` on a twd-js without + // diagnostics support, and dropped by serialisation. + testStatus.push({ + id: test.id, + status: "fail", + diagnostics: test.diagnostics, + error: `${err.message} (at ${window.location.href})`, + }); + }, + onSkip: (test) => { + test.status = "done"; + testStatus.push({ id: test.id, status: "skip" }); + }, + }, { retryCount }); + await runner.runByIds(chunkIds); + return testStatus; + }, attemptCount, ids); + + if (repeating) { + // Ids are random per page load, so each round reloads, re-enumerates and + // joins results by registration index rather than by id. + console.log(`Repeating each test ${repeatCount} times; retries are off for this run.`); + const tally = new Map(); + let handlersNow = registeredHandlers; + let orderedNow = allTestIds; + let selection = baseIds; + + for (let round = 1; round <= repeatCount && !stoppedEarly; round++) { + if (round > 1) { + console.log(`Repeat ${round}/${repeatCount}: reloading ${config.url} ...`); + await page.goto(config.url); + await page.waitForSelector('#twd-sidebar-root', { timeout: config.timeout }); + handlersNow = await page.evaluate(readRegisteredHandlers); + partialHandlers = handlersNow; + const activeFilters = changedTitles ? [...testFilters, ...changedTitles] : testFilters; + orderedNow = orderedTestIds(handlersNow); + allTestIds = orderedNow; + if (activeFilters.length > 0) { + const { ids } = selectTestIds(handlersNow, activeFilters); + if (ids.length === 0) { + throw new Error(`Repeat ${round}/${repeatCount}: no tests matched after reload.`); + } + selection = ids; + filteredIds = ids; + } else { + selection = orderedNow; + filteredIds = orderedNow; + } } - } - if (perTest && chunkClip) { - const result = await finishClip(chunkClip, ids.length); - if (result.failed) recordingFailed = true; - if (result.info) recordingInfos.push(result.info); - if (result.info) for (const entry of partialStatus) if (ids.includes(entry.id)) entry.recording = result.info.file; + for (const ids of chunk(selection, config.chunkSize)) { + const chunkStatus = await runChunk(ids, 1); + recordRound(tally, orderedNow, chunkStatus, handlersNow); + partialStatus = tallyToStatus(tally); + executed = tally.size; + // Distinct tests, not failed rounds: one test failing every round + // must not trip the default limit of 10 and cut the measurement short. + if (config.maxFailures > 0 && failedTestCount(tally) >= config.maxFailures) { + stoppedEarly = true; + break; + } + } } + } else { + const seenIds = new Set(); + + for (const ids of chunks) { + let chunkClip = null; + if (perTest && !recordingFailed) { + const testNames = ids.map((id) => buildTestPath(id, registeredHandlers)).filter(Boolean); + chunkClip = await startClip(testNames); + } else if (perTest && !recordingAbandoned) { + // Every further clip would stop against the same dead encoder, each one + // racing the 30s deadline, so maxClips stalls replace one. The tests keep + // running: a broken encoder must not swallow their results. + recordingAbandoned = true; + console.error('Recording stopped for the rest of the run. The remaining tests still run, unrecorded.'); + } - if (bail) break; + const chunkStatus = await runChunk(ids, config.retryCount); + + for (const entry of chunkStatus) { + if (seenIds.has(entry.id)) continue; + seenIds.add(entry.id); + partialStatus.push(entry); + } + executed += ids.length; + + let bail = false; + if (config.maxFailures > 0) { + const failed = partialStatus.filter((t) => t.status === 'fail').length; + if (failed >= config.maxFailures) { + stoppedEarly = true; + bail = true; + } + } + + if (perTest && chunkClip) { + const result = await finishClip(chunkClip, ids.length); + if (result.failed) recordingFailed = true; + if (result.info) recordingInfos.push(result.info); + if (result.info) for (const entry of partialStatus) if (ids.includes(entry.id)) entry.recording = result.info.file; + } + + if (bail) break; + } } if (recording && !perTest) { @@ -599,16 +691,16 @@ export async function runTests(options = {}) { // Contract validation. A sharded run validates even after an early stop and // flags the result partial, so merge can say exactly what is missing rather // than silently dropping a shard's worth of mocks. A non-sharded run keeps - // skipping, exactly as before. - const contractsConfigured = Boolean(config.contracts && config.contracts.length > 0); + // skipping, exactly as before. --repeat never validates: the skip was logged + // before launch, and the report must not claim a check that did not happen. let contractsBlock = { - configured: contractsConfigured, + configured: repeating ? false : contractsConfigured, partial: false, results: [], skipped: [], }; - if (contractsConfigured && (sharded || !stoppedEarly)) { + if (!repeating && contractsConfigured && (sharded || !stoppedEarly)) { // Never under sharding. A shard whose slice exercised no mocks — and any // shard with an empty slice — collects nothing, which is normal, so this // would advertise a twd-js version problem that does not exist on the @@ -663,12 +755,14 @@ export async function runTests(options = {}) { // while a red fourth writes none — a merged report that looks complete but // is missing a quarter of the code paths. `merge` applies the gate to the // true global result instead. - if (selectedIds && config.coverage) { + if (repeating && config.coverage) { + console.log('Skipping coverage collection (--repeat).'); + } else if (selectedIds && config.coverage) { console.log('Skipping coverage collection (test filter active).'); } let coverageData = null; - if (config.coverage && !selectedIds && (sharded || !hasFailures)) { + if (config.coverage && !repeating && !selectedIds && (sharded || !hasFailures)) { coverageData = await page.evaluate(() => window.__coverage__); if (!coverageData) { console.log('No code coverage data found.'); @@ -718,11 +812,13 @@ export async function runTests(options = {}) { maxFailures: config.maxFailures, reportPath, cpuThrottle: throttleRate, + repeat: repeating ? repeatCount : null, })); return hasFailures; } catch (error) { + if (error?.usage) throw error; const message = error && error.message ? error.message : String(error); const diagnostic = config ? explainError(error, config) : null; const reportPath = emitReport({ error: { message, diagnostic: diagnostic ?? null } }); @@ -735,6 +831,7 @@ export async function runTests(options = {}) { durationMs, reportPath, cpuThrottle: throttleRate, + repeat: repeating ? repeatCount : null, })); console.log('\nRun interrupted before completion — results above are partial.'); } diff --git a/src/needsAttention.js b/src/needsAttention.js index 27b4613..040dbbd 100644 --- a/src/needsAttention.js +++ b/src/needsAttention.js @@ -1,3 +1,5 @@ +import { isFlaky } from './repeat.js'; + function byRunOrder(a, b) { return (a.index ?? Infinity) - (b.index ?? Infinity); } @@ -25,6 +27,7 @@ export function needsAttention(report) { // so match that exact form instead. const at = snapshots.findIndex((s) => String(test.error ?? '').includes(`${s.name}.failed.png`)); const snapshot = at === -1 ? null : snapshots.splice(at, 1)[0]; + const repeated = typeof test.runs === 'number'; return { kind: 'test', path: test.path ?? test.id, @@ -32,6 +35,7 @@ export function needsAttention(report) { attempts: test.attempts ?? 1, recording: test.recording ?? null, snapshot, + ...(repeated ? { runs: test.runs, passes: test.passes ?? 0, flaky: isFlaky(test) } : {}), }; }); diff --git a/src/parseArgs.js b/src/parseArgs.js index fe259ab..5ec04a7 100644 --- a/src/parseArgs.js +++ b/src/parseArgs.js @@ -1,5 +1,6 @@ import { parseShardSpec } from './shard.js'; import { parseCpuThrottle } from './cpuThrottle.js'; +import { parseRepeat } from './repeat.js'; // Every flag each parser recognises. src/usage.js has to describe all of // them, and tests/usage.test.js checks that it does. @@ -16,6 +17,7 @@ export const RUN_FLAGS = [ '--record-speed', '--record-pace', '--cpu-throttle', + '--repeat', ]; export const MERGE_FLAGS = ['--out']; @@ -104,6 +106,9 @@ export function parseRunArgs(argv) { // null, not 1, when absent: the flag overrides cpuThrottle in the config, so // a default here would shadow it. let cpuThrottle = null; + // null when absent, so a single run does not grow a repeat field. 1 is a + // legal value and means one pass, the same as leaving the flag off. + let repeat = null; // Two separate flags on purpose, the way Jest separates them. They close two // different holes: --update-snapshots rewrites references that already exist, // --ci forbids creating one that does not. The precedence between them is @@ -170,6 +175,12 @@ export function parseRunArgs(argv) { // was throttled. cpuThrottle = parseCpuThrottle(value, '--cpu-throttle'); i += consumed - 1; + } else if (token === '--repeat' || token.startsWith('--repeat=')) { + const { value, consumed } = readValue(argv, token, '--repeat', i); + // Throws, like --shard and --cpu-throttle. A silently dropped repeat is + // one run the caller believes was repeated. + repeat = parseRepeat(value); + i += consumed - 1; } else if (token.startsWith('--')) { unknown.push(token); } @@ -177,7 +188,20 @@ export function parseRunArgs(argv) { if (unknown.length) throw unknownOptionsError('run', unknown, RUN_FLAGS); - return { testFilters, changedSince, record, shard, reportDir, noReport, updateSnapshots, ci, cpuThrottle }; + // Refused here, before runTests can clean the report folder or launch. + if (repeat > 1) { + const conflicts = [ + shard ? '--shard' : null, + record.enabled ? '--record' : null, + updateSnapshots ? '--update-snapshots' : null, + ci ? '--ci' : null, + ].filter(Boolean); + if (conflicts.length) { + throw new Error(`twd-cli run: --repeat cannot be combined with ${conflicts.join(' and ')}.`); + } + } + + return { testFilters, changedSince, record, shard, reportDir, noReport, updateSnapshots, ci, cpuThrottle, repeat }; } // `twd-cli report [] [--format ]`. diff --git a/src/repeat.js b/src/repeat.js new file mode 100644 index 0000000..e627249 --- /dev/null +++ b/src/repeat.js @@ -0,0 +1,89 @@ +import { buildTestPath } from './buildTestPath.js'; + +// A positive integer, the same rule for `--repeat 5` and `--repeat=5`. +// Throws, like --shard and --cpu-throttle: a dropped value would be a single +// run the caller believes was repeated. +export function parseRepeat(value) { + if (typeof value === 'string' && /^[1-9]\d*$/.test(value)) return Number(value); + + const got = value === undefined ? 'nothing' : JSON.stringify(value); + throw new Error(`Invalid --repeat: expected a positive integer, got ${got}.`); +} + +// A test that failed at least one round and passed at least one. A test that +// failed every round is a failure, not a flake. +export function isFlaky(test) { + const runs = test.runs ?? 0; + const passes = test.passes ?? 0; + const skips = test.skips ?? 0; + return passes > 0 && runs - passes - skips > 0; +} + +// Fold one round into tallies keyed by registration index. +// +// twd-js mints test ids with Math.random() at registration, so a reload cannot +// be joined on id. Registration order is stable across loads — the same +// assumption sharding's fingerprint makes — so the index in this round's +// ordered id list is the identity. `handlers` are this round's, and the path +// is stored now because a later round's handler map will not contain this id. +export function recordRound(tally, orderedIds, results, handlers) { + const indexById = new Map(orderedIds.map((id, index) => [id, index])); + const seen = new Set(); + for (const result of results) { + if (seen.has(result.id)) continue; + seen.add(result.id); + const index = indexById.get(result.id); + if (index === undefined) continue; + + const row = tally.get(index) ?? { + index, + id: result.id, + runs: 0, + passes: 0, + skips: 0, + path: null, + }; + row.id = result.id; + row.runs += 1; + if (result.status === 'pass') row.passes += 1; + else if (result.status === 'skip') row.skips += 1; + else if (result.status === 'fail') { + row.error = result.error; + if (result.diagnostics !== undefined) row.diagnostics = result.diagnostics; + } + const failures = row.runs - row.passes - row.skips; + if (failures > 0) row.status = 'fail'; + else if (row.passes > 0) row.status = 'pass'; + else row.status = 'skip'; + row.path = buildTestPath(result.id, handlers) ?? row.path; + tally.set(index, row); + } +} + +// Tests that have failed at least once. A further failed round of the same +// test does not add another, which is what keeps --repeat from tripping +// maxFailures on the test it is measuring. +export function failedTestCount(tally) { + let count = 0; + for (const row of tally.values()) if (row.status === 'fail') count += 1; + return count; +} + +export function tallyToStatus(tally) { + return [...tally.values()] + .sort((a, b) => a.index - b.index) + .map((row) => { + const entry = { + id: row.id, + status: row.status, + runs: row.runs, + passes: row.passes, + skips: row.skips, + path: row.path, + index: row.index, + }; + if (row.error !== undefined) entry.error = row.error; + if (row.diagnostics !== undefined) entry.diagnostics = row.diagnostics; + return entry; + }); +} diff --git a/src/reportHtml.js b/src/reportHtml.js index deeadfd..27dd7fd 100644 --- a/src/reportHtml.js +++ b/src/reportHtml.js @@ -46,6 +46,7 @@ function verdict(report) { `${esc(summary.skipped)} skipped`, ]; if (summary.notRun) stats.push(`${esc(summary.notRun)} not run`); + if (run.repeat > 1) stats.push(`${esc(run.repeat)}×`); if (report.contracts.configured) stats.push(`${esc(summary.contracts.errors)} contract errors`); stats.push(`${esc((run.durationMs / 1000).toFixed(1))}s · ${esc(run.url)} · ${esc(run.startedAt)}`); return `
${esc(String(outcome).toUpperCase())}
${stats.join('')}
`; @@ -60,7 +61,12 @@ function image(file, images) { function item(entry, images) { if (entry.kind === 'test') { const chips = []; - if (entry.attempts > 1) chips.push(`${entry.attempts} attempts`); + if (typeof entry.runs === 'number') { + const count = `${entry.passes}/${entry.runs} passed`; + chips.push(`${esc(entry.flaky ? `flaky · ${count}` : count)}`); + } else if (entry.attempts > 1) { + chips.push(`${entry.attempts} attempts`); + } return `
${esc(entry.path)}
` + (entry.error ? `
${esc(entry.error)}
` : '') + (chips.length ? `
${chips.join('')}
` : '') diff --git a/src/reportMarkdown.js b/src/reportMarkdown.js index eda3a7a..dfd6779 100644 --- a/src/reportMarkdown.js +++ b/src/reportMarkdown.js @@ -48,10 +48,17 @@ function table(report) { return [`| ${cols.join(' | ')} |`, `|${cols.map(() => '---').join('|')}|`, `| ${cells.join(' | ')} |`]; } +function resultNote(entry) { + if (typeof entry.runs === 'number') { + const count = `${entry.passes}/${entry.runs} passed`; + return ` _(${entry.flaky ? `flaky, ${count}` : count})_`; + } + return entry.attempts > 1 ? ` _(${entry.attempts} attempts)_` : ''; +} + function item(entry) { if (entry.kind === 'test') { - const retried = entry.attempts > 1 ? ` _(${entry.attempts} attempts)_` : ''; - const lines = [`- ❌ **${escapeMd(pretty(entry.path))}**${retried}`]; + const lines = [`- ❌ **${escapeMd(pretty(entry.path))}**${resultNote(entry)}`]; if (entry.error) lines.push(` > ${code(firstLine(entry.error))}`); if (entry.snapshot) lines.push(` > Layout snapshot ${code(entry.snapshot.name)} differs, diff in the report`); return lines; @@ -83,6 +90,8 @@ export function renderMarkdown(report) { lines.push(...table(report), ''); + if (report.run?.repeat > 1) lines.push(`Repeated ${report.run.repeat}×.`, ''); + const items = needsAttention(report); if (items.length) { lines.push('#### Needs attention'); diff --git a/src/runReport.js b/src/runReport.js index 48e81a3..2d7967e 100644 --- a/src/runReport.js +++ b/src/runReport.js @@ -69,6 +69,7 @@ export function buildRunReport({ snapshots = [], recordingFailed = false, coverage = null, + repeat = null, }) { // Resolved here, in the shard that ran the tests, because this is the only // place the handler map is valid: ids are per-page-load random, so shard 2's @@ -80,7 +81,7 @@ export function buildRunReport({ return finalizeReport({ schemaVersion: REPORT_SCHEMA_VERSION, - run: { twdCliVersion: version, url }, + run: { twdCliVersion: version, url, ...(repeat > 1 ? { repeat } : {}) }, error, snapshots, coverage, @@ -116,12 +117,15 @@ export function buildRunReport({ ...test, // For display. May be null if the handler somehow went missing, so every // renderer has to tolerate that. - path: buildTestPath(test.id, handlers), + // A repeat reloads the page, so an id from an earlier round is absent from + // the last handler map. The tally stored the path and index while that + // round's map was still valid; keep them when the lookup misses. + path: buildTestPath(test.id, handlers) ?? test.path ?? null, // Identity. Position in the shard-independent ordered list is stable // across shards because registration order is deterministic, unlike the // random id. The path cannot serve as the key: two tests may share one // (duplicate names) and can legally land in different shards. - index: positions.has(test.id) ? positions.get(test.id) : null, + index: positions.has(test.id) ? positions.get(test.id) : (test.index ?? null), attempts: attemptsFor(test, retryCount), })), contracts: contracts ?? { diff --git a/src/testSummary.js b/src/testSummary.js index 92c021f..12635c8 100644 --- a/src/testSummary.js +++ b/src/testSummary.js @@ -1,5 +1,6 @@ import { buildTestPath } from './buildTestPath.js'; import { formatFailureDiagnostics } from './failureDiagnostics.js'; +import { isFlaky } from './repeat.js'; /** * Display name for one test result. @@ -16,6 +17,28 @@ function resolvePath(entry, handlers) { return entry.path ?? buildTestPath(entry.id, handlers) ?? entry.id; } +// Count tests, not rounds, so the console matches run.json and the reports. +// A test that failed any round has status "fail" and counts once; the +// per-test (passes/runs passed) suffix carries the round detail. +function outcomeCounts(testStatus) { + return { + passed: testStatus.filter((t) => t.status === 'pass').length, + failed: testStatus.filter((t) => t.status === 'fail').length, + skipped: testStatus.filter((t) => t.status === 'skip').length, + }; +} + +function pushResultLines(lines, test, handlers, marker, suffix) { + const testPath = resolvePath(test, handlers); + lines.push(` ${marker} ${testPath}${suffix}`); + for (const row of formatFailureDiagnostics(test.diagnostics)) { + lines.push(` ${row}`); + } + if (test.error) { + lines.push(` ${String(test.error).replace(/\n/g, '\n ')}`); + } +} + export function formatRunComplete({ testStatus, handlers, @@ -27,10 +50,9 @@ export function formatRunComplete({ computeMs = null, reportPath = null, cpuThrottle = 1, + repeat = null, }) { - const passed = testStatus.filter((t) => t.status === 'pass').length; - const failed = testStatus.filter((t) => t.status === 'fail').length; - const skipped = testStatus.filter((t) => t.status === 'skip').length; + const { passed, failed, skipped } = outcomeCounts(testStatus); const duration = (durationMs / 1000).toFixed(1); const lines = [ @@ -54,21 +76,26 @@ export function formatRunComplete({ // Right under the duration it inflates. Only when active, so an unthrottled // block stays exactly what it has always been. if (cpuThrottle > 1) lines.push(` CPU throttle: ${cpuThrottle}x`); + if (repeat > 1) lines.push(` Repeat: ${repeat}x`); - const failures = testStatus.filter((t) => t.status === 'fail'); + const repeating = repeat > 1; + const flaky = repeating ? testStatus.filter((t) => isFlaky(t)) : []; + const failures = testStatus.filter((t) => t.status === 'fail' && !(repeating && isFlaky(t))); if (failures.length > 0) { lines.push('', ` Failed tests (${failures.length}):`); for (const failure of failures) { - const testPath = resolvePath(failure, handlers); - lines.push(` × ${testPath}`); - // Above the error, not below: a twd-js failure message can carry a full - // accessible-roles dump, which would bury the block underneath it. - for (const row of formatFailureDiagnostics(failure.diagnostics)) { - lines.push(` ${row}`); - } - if (failure.error) { - lines.push(` ${String(failure.error).replace(/\n/g, '\n ')}`); - } + const suffix = repeating ? ` (${failure.passes ?? 0}/${failure.runs} passed)` : ''; + // The marker line, then diagnostics, then the error. A twd-js failure + // message can carry a full accessible-roles dump, which would bury the + // block if the error came first. + pushResultLines(lines, failure, handlers, '×', suffix); + } + } + + if (flaky.length > 0) { + lines.push('', ` Flaky (${flaky.length}):`); + for (const test of flaky) { + pushResultLines(lines, test, handlers, '~', ` (${test.passes}/${test.runs} passed)`); } } diff --git a/src/usage.js b/src/usage.js index c859643..fb3a1f4 100644 --- a/src/usage.js +++ b/src/usage.js @@ -63,6 +63,18 @@ CPU throttling: shows. Overrides cpuThrottle in twd.config.json (default 1, full speed). Below 1 is refused. +Repeating: + --repeat Run the selected tests times in this browser, + reloading the page between rounds, and list each test + that failed a round with how often it passed. is + a positive integer. Retries are off for the run. + maxFailures counts tests that failed, not each failed + round. The console counts tests, not rounds. Cannot + be combined with --record, --shard, + --update-snapshots or --ci. Skips coverage, + contracts and layout snapshots. + Exits 1 if any round of any test failed. + Sharding (beta): --shard / Run slice i of n. Each shard discovers the whole suite and takes every nth test, so the count never @@ -94,6 +106,7 @@ Examples: npx twd-cli run --record --changed-since origin/main npx twd-cli run --shard 2/4 npx twd-cli run --cpu-throttle 6 --test "checkout flow" + npx twd-cli run --repeat 10 --cpu-throttle 6 --test "checkout flow" Create a twd.config.json file in your project root to customize settings. `; diff --git a/tests/cli.test.js b/tests/cli.test.js index 274fb0c..84c05d3 100644 --- a/tests/cli.test.js +++ b/tests/cli.test.js @@ -83,6 +83,20 @@ describe("twd-cli help", () => { expect(stderr).toMatch(/Did you mean --test\?/); }); + it("run with a bad --repeat refuses to run: stderr names it, exit 1", async () => { + const { code, stdout, stderr } = await cli("run", "--repeat", "0"); + expect(code).toBe(1); + expect(stdout).toBe(""); + expect(stderr).toMatch(/Invalid --repeat: expected a positive integer, got "0"/); + }); + + it("run with --repeat and --ci refuses to run: stderr names it, exit 1, nothing on stdout", async () => { + const { code, stdout, stderr } = await cli("run", "--repeat", "2", "--ci"); + expect(code).toBe(1); + expect(stdout).toBe(""); + expect(stderr).toMatch(/--repeat cannot be combined with --ci/); + }); + it("run with a --cpu-throttle below 1 refuses to run: stderr names it, exit 1", async () => { const { code, stdout, stderr } = await cli("run", "--cpu-throttle", "0"); expect(code).toBe(1); diff --git a/tests/needsAttention.test.js b/tests/needsAttention.test.js index 7c0a15a..66be726 100644 --- a/tests/needsAttention.test.js +++ b/tests/needsAttention.test.js @@ -7,6 +7,22 @@ describe('needsAttention', () => { expect(needsAttention(report())).toEqual([]); }); + it('marks a test that both passed and failed as flaky and keeps its counts', () => { + const [item] = needsAttention(report({ + tests: [{ id: 't1', status: 'fail', error: 'boom', runs: 10, passes: 8, skips: 0 }], + })); + expect(item).toMatchObject({ kind: 'test', flaky: true, runs: 10, passes: 8, error: 'boom' }); + }); + + it('does not mark a test that failed every round as flaky', () => { + const [item] = needsAttention(report({ + tests: [{ id: 't1', status: 'fail', error: 'boom', runs: 10, passes: 0, skips: 0 }], + })); + expect(item.flaky).toBe(false); + expect(item.runs).toBe(10); + expect(item.passes).toBe(0); + }); + it('lists failed tests in run order with their error and attempts', () => { const items = needsAttention(report({ tests: [ diff --git a/tests/parseArgs.test.js b/tests/parseArgs.test.js index 13a994d..9e2336b 100644 --- a/tests/parseArgs.test.js +++ b/tests/parseArgs.test.js @@ -3,7 +3,7 @@ import { parseRunArgs, parseMergeArgs, parseReportArgs } from "../src/parseArgs. describe("parseRunArgs", () => { it("returns empty filters when no args", () => { - expect(parseRunArgs([])).toEqual({ testFilters: [], changedSince: null, record: {}, shard: null, reportDir: null, noReport: false, updateSnapshots: false, ci: false, cpuThrottle: null }); + expect(parseRunArgs([])).toEqual({ testFilters: [], changedSince: null, record: {}, shard: null, reportDir: null, noReport: false, updateSnapshots: false, ci: false, cpuThrottle: null, repeat: null }); }); it("parses a single --test ", () => { @@ -17,6 +17,7 @@ describe("parseRunArgs", () => { updateSnapshots: false, ci: false, cpuThrottle: null, + repeat: null, }); }); @@ -31,6 +32,7 @@ describe("parseRunArgs", () => { updateSnapshots: false, ci: false, cpuThrottle: null, + repeat: null, }); }); @@ -45,11 +47,12 @@ describe("parseRunArgs", () => { updateSnapshots: false, ci: false, cpuThrottle: null, + repeat: null, }); }); it("ignores a trailing --test with no value", () => { - expect(parseRunArgs(['--test'])).toEqual({ testFilters: [], changedSince: null, record: {}, shard: null, reportDir: null, noReport: false, updateSnapshots: false, ci: false, cpuThrottle: null }); + expect(parseRunArgs(['--test'])).toEqual({ testFilters: [], changedSince: null, record: {}, shard: null, reportDir: null, noReport: false, updateSnapshots: false, ci: false, cpuThrottle: null, repeat: null }); }); it("ignores positional tokens", () => { @@ -100,6 +103,7 @@ describe("parseRunArgs", () => { updateSnapshots: false, ci: false, cpuThrottle: null, + repeat: null, }); }); @@ -137,6 +141,7 @@ describe("parseRunArgs", () => { updateSnapshots: false, ci: false, cpuThrottle: null, + repeat: null, }); }); @@ -177,6 +182,7 @@ describe('parseRunArgs shard and report flags', () => { updateSnapshots: false, ci: false, cpuThrottle: null, + repeat: null, }); }); @@ -304,6 +310,61 @@ describe('parseRunArgs --cpu-throttle', () => { }); }); +describe('parseRunArgs --repeat', () => { + it('parses --repeat in both forms', () => { + expect(parseRunArgs(['--repeat', '5']).repeat).toBe(5); + expect(parseRunArgs(['--repeat=10']).repeat).toBe(10); + }); + + it('is null when the flag is absent', () => { + expect(parseRunArgs([]).repeat).toBeNull(); + }); + + it('accepts 1, a single run', () => { + expect(parseRunArgs(['--repeat', '1']).repeat).toBe(1); + }); + + it('throws on zero, fractions, and non-numbers', () => { + expect(() => parseRunArgs(['--repeat', '0'])).toThrow(/Invalid --repeat/); + expect(() => parseRunArgs(['--repeat=1.5'])).toThrow(/Invalid --repeat/); + expect(() => parseRunArgs(['--repeat', 'nope'])).toThrow(/got "nope"/); + expect(() => parseRunArgs(['--repeat', '-3'])).toThrow(/got "-3"/); + }); + + it('throws on a trailing --repeat with no value', () => { + expect(() => parseRunArgs(['--repeat'])).toThrow(/got nothing/); + }); + + it('throws rather than swallowing the flag that follows a valueless --repeat', () => { + expect(() => parseRunArgs(['--repeat', '--test', 'Login'])).toThrow(/got nothing/); + }); + + it('combines with --test and --cpu-throttle', () => { + const result = parseRunArgs(['--repeat', '10', '--cpu-throttle', '6', '--test', 'checkout']); + expect(result.repeat).toBe(10); + expect(result.cpuThrottle).toBe(6); + expect(result.testFilters).toEqual(['checkout']); + }); + + it.each([ + [['--repeat', '2', '--shard', '1/2'], /--repeat cannot be combined with --shard\./], + [['--repeat', '2', '--record'], /--repeat cannot be combined with --record\./], + [['--repeat', '2', '--update-snapshots'], /--repeat cannot be combined with --update-snapshots\./], + [['--repeat', '2', '--ci'], /--repeat cannot be combined with --ci\./], + [['--repeat', '2', '--shard', '1/2', '--ci'], /--repeat cannot be combined with --shard and --ci\./], + ])('refuses %j in the parser', (argv, message) => { + expect(() => parseRunArgs(argv)).toThrow(message); + }); + + it('allows --repeat 1 with those flags', () => { + expect(parseRunArgs(['--repeat', '1', '--ci']).repeat).toBe(1); + }); + + it('suggests --repeat for a near miss', () => { + expect(() => parseRunArgs(['--repeats', '2'])).toThrow(/Did you mean --repeat\?/); + }); +}); + describe('parseRunArgs unknown options', () => { it('refuses an unknown --flag instead of dropping it', () => { // The whole reason `run --help` ran the suite: a token no branch claimed diff --git a/tests/repeat.test.js b/tests/repeat.test.js new file mode 100644 index 0000000..42c0793 --- /dev/null +++ b/tests/repeat.test.js @@ -0,0 +1,95 @@ +import { describe, it, expect } from 'vitest'; +import { parseRepeat, recordRound, failedTestCount, tallyToStatus, isFlaky } from '../src/repeat.js'; + +describe('parseRepeat', () => { + it('accepts a positive integer string', () => { + expect(parseRepeat('5')).toBe(5); + expect(parseRepeat('1')).toBe(1); + }); + + it('rejects zero, fractions, and junk', () => { + expect(() => parseRepeat('0')).toThrow(/Invalid --repeat: expected a positive integer, got "0"/); + expect(() => parseRepeat('1.5')).toThrow(/got "1.5"/); + expect(() => parseRepeat('-2')).toThrow(/got "-2"/); + expect(() => parseRepeat('nope')).toThrow(/got "nope"/); + expect(() => parseRepeat(undefined)).toThrow(/got nothing/); + }); +}); + +describe('recordRound', () => { + const handlersFor = (id, suiteId = 's') => [ + { id: suiteId, name: 'Checkout', type: 'suite' }, + { id, name: 'checkout flow', parent: suiteId, type: 'test' }, + ]; + + it('counts passes and failures of the same test across reloads', () => { + const tally = new Map(); + recordRound(tally, ['a'], [{ id: 'a', status: 'pass' }], handlersFor('a', 's1')); + recordRound( + tally, + ['b'], + [{ id: 'b', status: 'fail', error: 'total mismatch', diagnostics: { mock: true } }], + handlersFor('b', 's2'), + ); + + const [row] = tallyToStatus(tally); + expect(row).toMatchObject({ + id: 'b', + status: 'fail', + runs: 2, + passes: 1, + skips: 0, + error: 'total mismatch', + path: 'Checkout > checkout flow', + index: 0, + }); + expect(row.diagnostics).toEqual({ mock: true }); + expect(isFlaky(row)).toBe(true); + expect(failedTestCount(tally)).toBe(1); + }); + + it('does not count a second result for the same id in one round', () => { + const tally = new Map(); + recordRound(tally, ['a'], [ + { id: 'a', status: 'pass' }, + { id: 'a', status: 'fail', error: 'dup' }, + ], handlersFor('a')); + expect(tallyToStatus(tally)[0]).toMatchObject({ runs: 1, passes: 1, status: 'pass' }); + }); + + it('keeps a test that failed every round as a failure with no passes', () => { + const tally = new Map(); + const handlers = [{ id: 'a', name: 't', type: 'test' }]; + recordRound(tally, ['a'], [{ id: 'a', status: 'fail', error: 'x' }], handlers); + recordRound(tally, ['a'], [{ id: 'a', status: 'fail', error: 'y' }], handlers); + const [row] = tallyToStatus(tally); + expect(row).toMatchObject({ runs: 2, passes: 0, status: 'fail', error: 'y' }); + expect(isFlaky(row)).toBe(false); + }); + + it('counts a skip separately from a failure', () => { + const tally = new Map(); + const handlers = [{ id: 'a', name: 't', type: 'test' }]; + recordRound(tally, ['a'], [{ id: 'a', status: 'skip' }], handlers); + recordRound(tally, ['a'], [{ id: 'a', status: 'pass' }], handlers); + expect(tallyToStatus(tally)[0]).toMatchObject({ runs: 2, passes: 1, skips: 1, status: 'pass' }); + expect(failedTestCount(tally)).toBe(0); + }); + + it('counts distinct failed tests, not failed rounds', () => { + const tally = new Map(); + const handlers = [ + { id: 'a', name: 'one', type: 'test' }, + { id: 'b', name: 'two', type: 'test' }, + ]; + recordRound(tally, ['a', 'b'], [ + { id: 'a', status: 'fail', error: 'x' }, + { id: 'b', status: 'pass' }, + ], handlers); + recordRound(tally, ['a', 'b'], [ + { id: 'a', status: 'fail', error: 'x' }, + { id: 'b', status: 'pass' }, + ], handlers); + expect(failedTestCount(tally)).toBe(1); + }); +}); diff --git a/tests/reportHtml.test.js b/tests/reportHtml.test.js index 874e9b4..3d2bbd8 100644 --- a/tests/reportHtml.test.js +++ b/tests/reportHtml.test.js @@ -26,6 +26,24 @@ describe('renderHtml', () => { expect(html).not.toContain('Needs attention'); }); + it('shows a flaky test under needs attention with its pass count', () => { + const html = renderHtml(report({ + repeat: 10, + tests: [{ id: 't1', status: 'fail', error: 'boom', runs: 10, passes: 8, skips: 0 }], + })); + expect(html).toContain('Needs attention (1)'); + expect(html).toContain('flaky · 8/10 passed'); + expect(html).toContain('>10<'); + }); + + it('shows a pass count for a test that failed every round without calling it flaky', () => { + const html = renderHtml(report({ + tests: [{ id: 't1', status: 'fail', error: 'boom', runs: 10, passes: 0, skips: 0 }], + })); + expect(html).toContain('0/10 passed'); + expect(html).not.toContain('flaky'); + }); + it('puts a contract error next to test failures', () => { const html = renderHtml(report({ contracts: { results: [contractResult({ validation: invalid() })] } })); expect(html).toContain('Needs attention (1)'); diff --git a/tests/reportMarkdown.test.js b/tests/reportMarkdown.test.js index 641a6e3..334fec7 100644 --- a/tests/reportMarkdown.test.js +++ b/tests/reportMarkdown.test.js @@ -25,6 +25,25 @@ describe('renderMarkdown', () => { expect(md).toContain('1 ✓ · 0 ✕ · 0 ⚠'); }); + it('labels a flaky test under needs attention and notes the repeat', () => { + const md = renderMarkdown(report({ + repeat: 10, + tests: [{ id: 't1', status: 'fail', error: 'Expected visible\nroles dump...', runs: 10, passes: 8, skips: 0 }], + })); + expect(md).toContain('- ❌ **Invoices › shows empty state** _(flaky, 8/10 passed)_'); + expect(md).toContain('Repeated 10×'); + expect(md).toContain(' > `Expected visible`'); + }); + + it('shows a pass count for a test that failed every round without calling it flaky', () => { + const md = renderMarkdown(report({ + repeat: 10, + tests: [{ id: 't1', status: 'fail', error: 'boom', runs: 10, passes: 0, skips: 0 }], + })); + expect(md).toContain('_(0/10 passed)_'); + expect(md).not.toContain('flaky'); + }); + it('lists each failure with its first error line', () => { const md = renderMarkdown(report({ tests: [{ id: 't1', status: 'fail', error: 'Expected visible\nroles dump...' }], diff --git a/tests/runReport.test.js b/tests/runReport.test.js index cd313ef..fa6a1ef 100644 --- a/tests/runReport.test.js +++ b/tests/runReport.test.js @@ -205,6 +205,29 @@ describe('schema v3', () => { expect(build().schemaVersion).toBe(3); }); + it('carries repeat on the run and per-test counts without changing the outcome', () => { + const report = build({ + repeat: 10, + tests: [{ id: 't1', status: 'fail', error: 'total', runs: 10, passes: 8, skips: 0 }], + }); + expect(report.run.repeat).toBe(10); + expect(report.tests[0]).toMatchObject({ status: 'fail', runs: 10, passes: 8, skips: 0 }); + expect(report.outcome).toBe('failed'); + expect(report.schemaVersion).toBe(3); + }); + + it('omits repeat from the header when the run was not repeated', () => { + expect(build().run.repeat).toBeUndefined(); + }); + + it('keeps a path recorded before the test id went stale across a reload', () => { + const report = build({ + tests: [{ id: 'stale', status: 'fail', error: 'e', path: 'Checkout > checkout flow', index: 4, runs: 2, passes: 1 }], + }); + expect(report.tests[0].path).toBe('Checkout > checkout flow'); + expect(report.tests[0].index).toBe(4); + }); + it('records the run header', () => { const report = build({ version: '1.10.0', url: 'http://localhost:5173' }); expect(report.run).toEqual({ diff --git a/tests/runTests.test.js b/tests/runTests.test.js index dc2501a..898f651 100644 --- a/tests/runTests.test.js +++ b/tests/runTests.test.js @@ -812,7 +812,7 @@ describe("runTests recording", () => { await runTests(); const [inject, flags] = page.evaluateOnNewDocument.mock.calls[0]; - expect(flags).toEqual({ update: false, ci: false }); + expect(flags).toEqual({ update: false, ci: false, snapshots: true }); // The injected function is serialised into the browser, so run it here // against a stand-in global to see what it actually sets. @@ -836,7 +836,7 @@ describe("runTests recording", () => { await runTests({ updateSnapshots: true, ci: true }); - expect(page.evaluateOnNewDocument.mock.calls[0][1]).toEqual({ update: true, ci: true }); + expect(page.evaluateOnNewDocument.mock.calls[0][1]).toEqual({ update: true, ci: true, snapshots: true }); }); it("does not touch any recording API when recording is disabled", async () => { @@ -2693,3 +2693,249 @@ describe("runTests CPU throttling", () => { expect(logs.some((l) => /CPU throttl/.test(l))).toBe(false); }); }); + +describe('runTests --repeat', () => { + let logSpy; + + function writtenReport() { + const call = vi.mocked(fs.writeFileSync).mock.calls + .find(([file]) => String(file).endsWith('run.json')); + return call ? JSON.parse(call[1]) : null; + } + + function completeBlock() { + const logs = logSpy.mock.calls.map((c) => String(c[0])); + return logs.find((l) => l.startsWith('--- Run complete ---')); + } + + beforeEach(() => { + vi.clearAllMocks(); + vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig }); + vi.mocked(fs.writeFileSync).mockReset(); + // An earlier test installs an rmSync that throws EPERM. clearAllMocks does + // not drop that implementation, and a leaked throw becomes a clean warning + // on every later run. + vi.mocked(fs.rmSync).mockReset(); + logSpy = vi.spyOn(console, 'log').mockImplementation(() => {}); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('reloads between rounds, forces a single attempt, and fails the run when any round failed', async () => { + const rounds = [ + { + handlers: [ + { id: 's1', name: 'Checkout', type: 'suite' }, + { id: 'a', name: 'checkout flow', parent: 's1', type: 'test' }, + { id: 'b', name: 'receipt', parent: 's1', type: 'test' }, + ], + status: [ + { id: 'a', status: 'pass' }, + { id: 'b', status: 'pass' }, + ], + }, + { + handlers: [ + { id: 's2', name: 'Checkout', type: 'suite' }, + { id: 'c', name: 'checkout flow', parent: 's2', type: 'test' }, + { id: 'd', name: 'receipt', parent: 's2', type: 'test' }, + ], + status: [ + { id: 'c', status: 'fail', error: 'total mismatch' }, + { id: 'd', status: 'pass' }, + ], + }, + ]; + let enumerated = 0; + const page = createMockPage(); + page.evaluate = vi.fn(async (_fn, retryCount, ids) => { + if (ids === undefined) { + const round = rounds[enumerated]; + enumerated += 1; + return round.handlers; + } + expect(retryCount).toBe(1); + return rounds[enumerated - 1].status; + }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, retryCount: 3, coverage: true, + }); + + const failed = await runTests({ repeat: 2 }); + + expect(failed).toBe(true); + expect(page.goto).toHaveBeenCalledTimes(2); + expect(page.waitForSelector).toHaveBeenCalledTimes(2); + expect(puppeteer.launch).toHaveBeenCalledTimes(1); + const block = completeBlock(); + expect(block).toContain('Passed: 1 | Failed: 1 | Skipped: 0'); + expect(block).toContain('Repeat: 2x'); + expect(block).toContain('Flaky (1):'); + expect(block).toContain('~ Checkout > checkout flow (1/2 passed)'); + expect(block).toContain('total mismatch'); + expect(block).not.toContain('Retried'); + const logs = logSpy.mock.calls.map((c) => String(c[0])); + expect(logs.some((l) => l.includes('retries are off'))).toBe(true); + expect(logs.some((l) => l.includes('Skipping coverage collection (--repeat)'))).toBe(true); + + const report = writtenReport(); + expect(report.run.repeat).toBe(2); + expect(report.outcome).toBe('failed'); + expect(report.tests.find((t) => t.passes === 1)).toMatchObject({ + status: 'fail', runs: 2, passes: 1, + }); + const files = vi.mocked(fs.writeFileSync).mock.calls.map(([f]) => String(f)); + expect(files.some((f) => f.includes('.nyc_output'))).toBe(false); + }); + + it('exits 0 when every round passed', async () => { + const handlers = [{ id: 't', name: 'stable', type: 'test' }]; + let enumerated = 0; + const page = createMockPage(); + page.evaluate = vi.fn(async (_fn, _retryCount, ids) => { + if (ids === undefined) { + enumerated += 1; + return handlers; + } + return [{ id: 't', status: 'pass' }]; + }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + + const failed = await runTests({ repeat: 3 }); + + expect(failed).toBe(false); + expect(page.goto).toHaveBeenCalledTimes(3); + expect(enumerated).toBe(3); + expect(completeBlock()).toContain('Passed: 1 | Failed: 0 | Skipped: 0'); + expect(completeBlock()).not.toContain('Flaky'); + expect(writtenReport().outcome).toBe('passed'); + expect(writtenReport().run.repeat).toBe(3); + }); + + it('does not bail when the same test fails every round under the default failure limit', async () => { + const handlers = [{ id: 't', name: 'flaky', type: 'test' }]; + const page = createMockPage(); + page.evaluate = vi.fn(async (_fn, _retryCount, ids) => { + if (ids === undefined) return handlers; + return [{ id: 't', status: 'fail', error: 'x' }]; + }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig, maxFailures: 2 }); + + const failed = await runTests({ repeat: 4 }); + + expect(failed).toBe(true); + expect(page.goto).toHaveBeenCalledTimes(4); + const chunks = page.evaluate.mock.calls.filter((call) => call.length === 3); + expect(chunks).toHaveLength(4); + expect(completeBlock()).toContain('(0/4 passed)'); + expect(completeBlock()).not.toContain('Flaky'); + expect(completeBlock()).not.toContain('Stopped early'); + }); + + it('stops once enough distinct tests have failed, and does not start another round', async () => { + const handlers = [ + { id: 't1', name: 'one', type: 'test' }, + { id: 't2', name: 'two', type: 'test' }, + { id: 't3', name: 'three', type: 'test' }, + ]; + const page = createMockPage(); + page.evaluate = vi.fn(async (_fn, _retryCount, ids) => { + if (ids === undefined) return handlers; + return ids.map((id) => ({ id, status: 'fail', error: 'x' })); + }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig, maxFailures: 1, chunkSize: 1 }); + + await runTests({ repeat: 3 }); + + expect(page.goto).toHaveBeenCalledTimes(1); + const chunks = page.evaluate.mock.calls.filter((call) => call.length === 3); + expect(chunks).toHaveLength(1); + expect(chunks[0][2]).toEqual(['t1']); + expect(completeBlock()).toContain('Stopped early'); + expect(writtenReport().run.repeat).toBe(3); + expect(writtenReport().tests).toEqual([ + expect.objectContaining({ runs: 1, passes: 0, status: 'fail' }), + ]); + }); + + it('refuses --repeat together with --shard before launching a browser', async () => { + await expect(runTests({ repeat: 2, shard: { index: 1, total: 2 } })) + .rejects.toThrow(/--repeat cannot be combined with --shard/); + expect(puppeteer.launch).not.toHaveBeenCalled(); + }); + + it('refuses --repeat together with recording before launching a browser', async () => { + await expect(runTests({ repeat: 2, recordOverrides: { enabled: true } })) + .rejects.toThrow(/--repeat cannot be combined with --record/); + expect(puppeteer.launch).not.toHaveBeenCalled(); + expect(assertFfmpegCapable).not.toHaveBeenCalled(); + }); + + it('refuses --repeat when recording is enabled in the config', async () => { + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, record: { enabled: true }, + }); + await expect(runTests({ repeat: 2 })).rejects.toThrow( + /--repeat cannot run with "record\.enabled": true in twd\.config\.json\./, + ); + expect(puppeteer.launch).not.toHaveBeenCalled(); + }); + + it('does not clean the report folder when the config recording refusal throws', async () => { + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, record: { enabled: true }, + }); + // cleanReportDir would remove these if it ran; a refused run must not. + vi.mocked(fs.readdirSync).mockReturnValue(['run.json']); + + await expect(runTests({ repeat: 2 })).rejects.toThrow(/record\.enabled/); + + expect(fs.rmSync).not.toHaveBeenCalled(); + expect(puppeteer.launch).not.toHaveBeenCalled(); + }); + + it('disables layout snapshots while repeating', async () => { + const handlers = [{ id: 't', name: 'stable', type: 'test' }]; + const page = createMockPage(); + page.evaluate = vi.fn(async (_fn, _retryCount, ids) => { + if (ids === undefined) return handlers; + return [{ id: 't', status: 'pass' }]; + }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + + await runTests({ repeat: 2, updateSnapshots: true, ci: true }); + + const [inject, flags] = page.evaluateOnNewDocument.mock.calls[0]; + expect(flags).toEqual({ update: false, ci: false, snapshots: false }); + expect(runInjected(inject, flags)).toEqual({ __TWD_SNAPSHOTS__: false }); + const logs = logSpy.mock.calls.map((c) => String(c[0])); + expect(logs.some((l) => l.includes('Skipping layout snapshots (--repeat)'))).toBe(true); + }); + + it('skips contract validation while repeating', async () => { + const handlers = [{ id: 't', name: 'stable', type: 'test' }]; + const page = createMockPage(); + page.evaluate = vi.fn(async (_fn, _retryCount, ids) => { + if (ids === undefined) return handlers; + return [{ id: 't', status: 'pass' }]; + }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, contracts: [{ source: 'api.json' }], + }); + vi.mocked(loadContracts).mockResolvedValue([]); + vi.mocked(validateMocks).mockReturnValue({ results: [], skipped: [] }); + + await runTests({ repeat: 2 }); + + expect(loadContracts).not.toHaveBeenCalled(); + expect(validateMocks).not.toHaveBeenCalled(); + const logs = logSpy.mock.calls.map((c) => String(c[0])); + expect(logs.some((l) => l.includes('Skipping contract validation (--repeat)'))).toBe(true); + }); +}); diff --git a/tests/testSummary.test.js b/tests/testSummary.test.js index 64058b5..9555ac5 100644 --- a/tests/testSummary.test.js +++ b/tests/testSummary.test.js @@ -1,5 +1,6 @@ import { describe, it, expect } from 'vitest'; import { formatRunComplete } from '../src/testSummary.js'; +import { buildRunReport } from '../src/runReport.js'; const handlers = [ { id: 's1', name: 'Login', type: 'suite' }, @@ -250,6 +251,105 @@ describe('formatRunComplete', () => { }); }); +describe('formatRunComplete with --repeat', () => { + const checkout = [ + { id: 's', name: 'Checkout', type: 'suite' }, + { id: 't1', name: 'pays', parent: 's', type: 'test' }, + { id: 't2', name: 'checkout flow', parent: 's', type: 'test' }, + ]; + + it('counts tests and lists a test that both passed and failed as flaky', () => { + const block = formatRunComplete({ + testStatus: [ + { id: 't1', status: 'pass', runs: 10, passes: 10, skips: 0 }, + { + id: 't2', + status: 'fail', + runs: 10, + passes: 8, + skips: 0, + error: 'Expected element [data-testid="total"] to have text "42.00"', + }, + ], + handlers: checkout, + durationMs: 41300, + repeat: 10, + }); + expect(block).toBe( + '--- Run complete ---\n' + + ' Passed: 1 | Failed: 1 | Skipped: 0\n' + + ' Duration: 41.3s\n' + + ' Repeat: 10x\n' + + '\n' + + ' Flaky (1):\n' + + ' ~ Checkout > checkout flow (8/10 passed)\n' + + ' Expected element [data-testid="total"] to have text "42.00"' + ); + }); + + it('lists a test that failed every round with its pass count, not as flaky', () => { + const block = formatRunComplete({ + testStatus: [{ id: 't1', status: 'fail', error: 'boom', runs: 10, passes: 0, skips: 0 }], + handlers, + durationMs: 1000, + repeat: 10, + }); + expect(block).toContain(' Passed: 0 | Failed: 1 | Skipped: 0'); + expect(block).toContain(' Repeat: 10x'); + expect(block).toContain(' × Login > shows error on wrong password (0/10 passed)'); + expect(block).toContain(' boom'); + expect(block).not.toContain('Flaky'); + }); + + it('counts a partly skipped test once, as passed, not by rounds', () => { + const block = formatRunComplete({ + testStatus: [{ id: 't1', status: 'pass', runs: 3, passes: 2, skips: 1 }], + handlers, + durationMs: 1000, + repeat: 3, + }); + expect(block).toContain(' Passed: 1 | Failed: 0 | Skipped: 0'); + expect(block).not.toContain('Flaky'); + expect(block).not.toContain('Failed tests'); + }); + + it('reports the same counts as buildRunReport for 3 tests x 10 rounds, one failing 2 rounds', () => { + const testStatus = [ + { id: 't1', status: 'pass', runs: 10, passes: 10, skips: 0 }, + { id: 't2', status: 'fail', runs: 10, passes: 8, skips: 0, error: 'boom' }, + { id: 't3', status: 'pass', runs: 10, passes: 10, skips: 0 }, + ]; + const block = formatRunComplete({ testStatus, handlers, durationMs: 1000, repeat: 10 }); + const report = buildRunReport({ + shard: { index: 1, total: 1 }, + startedAt: 1000, + endedAt: 2000, + allTestIds: ['t1', 't2', 't3'], + handlers, + tests: testStatus, + executed: 3, + notRun: 0, + stoppedEarly: false, + repeat: 10, + }); + expect(report.summary).toMatchObject({ passed: 2, failed: 1, skipped: 0 }); + expect(block).toContain(' Passed: 2 | Failed: 1 | Skipped: 0'); + }); + + it('leaves a single run unchanged when repeat is absent', () => { + const block = formatRunComplete({ + testStatus: [{ id: 't1', status: 'fail', error: 'boom' }], + handlers, + durationMs: 1000, + }); + expect(block).not.toContain('Repeat:'); + expect(block).not.toContain('Flaky'); + expect(block).toContain(' Passed: 0 | Failed: 1 | Skipped: 0'); + expect(block).toContain(' × Login > shows error on wrong password'); + expect(block).not.toContain('(0/1 passed)'); + }); +}); + describe('formatRunComplete with shards', () => { const handlers = [ { id: 's1', name: 'Login', parent: null, type: 'suite' }, diff --git a/tests/usage.test.js b/tests/usage.test.js index 38a89da..b85976a 100644 --- a/tests/usage.test.js +++ b/tests/usage.test.js @@ -29,6 +29,14 @@ describe("runUsage", () => { expect(runUsage()).toMatch(/--flag=value/); }); + it("documents --repeat as a positive integer that cannot combine with record or shard", () => { + const text = runUsage(); + expect(text).toContain('--repeat '); + expect(text).toMatch(/positive integer/); + expect(text).toMatch(/--record/); + expect(text).toMatch(/--shard/); + }); + it("does not describe merge", () => { expect(runUsage()).not.toMatch(/--out/); });