diff --git a/CHANGELOG.md b/CHANGELOG.md index 490b62e..cedd2a1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,12 @@ entry per wave, newest first; the first six waves (V1–V6, the MVP) predate the are recorded in PLAN.md §B and §J. Each entry carries the opening sentence of its row and the reason the wave was built. +## V50 — Metrics that agree with the reference on every edge + +Measured before anything changed, then checked against scikit-learn rather than against our own reading of it. + +_Why:_ Approved 09/10/2026 after V49. The same rule as V35: a number must not flatter or punish by accident. Two of the three edges were reachable from the demo picker, and the reference that would have caught the first one existed since V42 but held no case with an absent class. + ## V49 — CI that cannot be steered by a branch name The wave opened with zizmor 1.30.1 on the three workflows: **28 findings** (1 template injection, 21 actions pinned by a movable tag, 6 checkouts leaving the git token on the runner disk). diff --git a/PLAN.md b/PLAN.md index 0420689..4ae0fca 100644 --- a/PLAN.md +++ b/PLAN.md @@ -615,7 +615,7 @@ before features, and the one that produces wrong data comes first of all. | **V47 — delivered** | **Workers that cannot answer the wrong question.** Six lifecycle defects in shipped code, each reproduced by a test that failed before its fix. **(1) A result filed under a target it was not trained for**: `setTarget` and `toggleColumn` reset the training state without cancelling the run, the worker kept posting `model-result`, and the auto-save recorded the new target beside the old scores. The store now refuses both changes while anything computed for the current question is running (training, tuning, learning curve, robust ranking, batch scoring), the target picker and the column buttons are disabled for the same span, and a fence (`isStaleResponse`) drops any training message outside a run and any run-derived answer when no run is done; the worker answers in order, so the training status is enough to tell a late answer from a current one, without a run id in the protocol. **(2) Cancel did not reach the parallel phase**: V37's helpers were only checked after every one of them had finished, so cancelling waited for the heaviest families; the phase now polls the flag every 100 ms and terminates the helpers mid-fit. **(3) A crashed ML worker** left `trainStatus` at `training` forever and kept the dead reference; `onerror` now terminates it, clears every busy flag, and the next request starts a fresh worker. **(4) The same crash in the chat** left « thinking » up and still offered a local model that died with its worker. **(5) DuckDB was never closed**: each visit to the SQL panel left a worker and its Wasm heap behind, a failed start leaked its worker too, and a failed Parquet export left its file in the heap; the panel now closes the engine on unmount, on a late start and on failure, and export names come from a counter rather than a millisecond clock. **(6) A webcam stream granted after the page was left** kept the camera lit; it is now stopped on arrival. **(7) Found by this wave's own CI**: touching the chat ran the interpretation bench, which failed on `size-mismatch:tokenizer_config.json`. `prepare-llm.mjs` called its revision « pinned » while it held `main`, and upstream commit `1e94c62` (07/10/2026) had grown that file from 9 762 to 10 416 bytes; the revision is now the commit the sizes were measured on (`d6beae7`), which also keeps the next deploy cache miss from failing the same way. **What this wave deliberately does not do**: add a run id to the worker protocol (the order guarantee makes it redundant today, and it is the first thing to add if requests ever become concurrent), cancel analyses the user cannot cancel today, or touch the deeper share-link validation, which is V48. 883 unit tests, 123 e2e. | Approved 09/10/2026 after the post-V46 audit. The defect in (1) does not crash and does not warn: it saves a run that looks valid and is not, which is the one kind of failure an honest lab must never produce. The others are the same class of bug, a worker outliving the state it was started for, and fixing them together keeps the rule in one place. | | **V48 — delivered** | **A damaged share link ends on a refusal, never on a crash.** The plan for this row was a deeper validator: check `insights.importance`, numeric metrics, `classes` and `confusion`, the fields the audit had named. The wave opened with a measurement instead, and the measurement changed the plan. Three real links were captured from the running app (titanic with uncertainty, threshold and segments; iris with k-NN tuning and groups; mpg as a regression) and every field of each was deleted, nulled, or retyped as a string, a number, an object or an array: **3 948 damaged links, 3 660 accepted by the decoder, 1 020 of those crashed the page, on 99 distinct paths**, from `insights.importance` to `artifacts.exploration.clusters[].traits[].clusterMean`. A validator for 99 paths would be the largest file in the share module and would drift with every analysis a run learns to carry. So the decoder keeps checking what every view reads first, and `SharedRunPage` wraps the run in a render guard: any failure while drawing it shows the same refusal page, with its own explanation (the link decoded, but part of the run does not have the shape LabML writes) and nothing from the run, so no half-read figure can pass for a real one. The three links are committed under `tests/share-links/`, and the measurement is now a test: every field shape is damaged six ways and the page must end on the run or on the refusal; it fails on the code before this wave and adds about 25 s to the unit suite, the price of covering the whole format rather than a sample. The five copies of the download helper become one, `src/lib/download.ts`, which revokes its object URL after the click has been handled instead of in the same tick, where some engines cancel the download silently. **What this wave deliberately does not do**: write a field-by-field schema for every artifact, show a partially readable run, or guard event handlers (a click on a damaged run is not reachable: the refusal replaces it first). 889 unit tests, 123 e2e. | Approved 09/10/2026 after V47. A link is input from a stranger, and the most common stranger is a chat client that cut it. The decoder already refused the shapes the V35 audit had found; the measurement showed it could not keep up with the format, and that the robust answer was at the point of failure rather than at the gate. | | **V49 — delivered** | **CI that cannot be steered by a branch name.** The wave opened with zizmor 1.30.1 on the three workflows: **28 findings** (1 template injection, 21 actions pinned by a movable tag, 6 checkouts leaving the git token on the runner disk). **(1) Injection**: the deploy step expanded `github.head_ref` inside its script, so a branch named `x";…;"` would have run as code; the value now reaches the shell through `env:`. Low severity today (only branches of this repository deploy) and fixed before that stops being true. **(2) Pins**: every action is pinned to a full commit SHA with its release in a trailing comment (`# v7.0.1`), which Dependabot keeps current; a tag can be moved by whoever controls the action's repository, a SHA cannot. **(3) Credentials**: every checkout sets `persist-credentials: false`; no job pushes, so nothing needed the token on disk. After the fixes the offline audit reports **0 findings**, and a new `Workflow audit (zizmor)` job runs it on every pull request, installed from PyPI at a pinned version rather than through a third-party action, with the token the online audits need to check that each pinned SHA really belongs to its repository. Three new tests in `supply-chain.test.ts` hold the same rules per workflow file (SHA plus release comment on every `uses:`, no persisted credentials, no pull-request-controlled `${{ }}` inside a `run:`); they fail on the workflows before this wave. **(4) Coverage**: `npm run test:coverage` (V8 provider) replaces `npm run test` in CI with a floor set from the measured baseline rounded down, not a round number: **64.56 % statements, 53.28 % branches, 54.46 % functions, 65.62 % lines** on 898 unit tests, floors 64 / 53 / 54 / 65. The figures cover unit tests only; the interface is mostly exercised by the e2e suite, which this does not measure. Cost: the quality job grows by about a minute. **What this wave deliberately does not do**: chase a coverage target, add e2e coverage instrumentation, or gate deploys on the audit job. 898 unit tests, 123 e2e. | Approved 09/10/2026 after V48. Continues V44: a security gate is worth its maintenance only when it distinguishes a real exposure, and every rule here is now enforced by a test or a job rather than by a reviewer remembering it. | -| **V50 — pending** | **Metrics that agree with the reference on every edge.** `macroPrf` averages over every class the task knows, so a class absent from both truth and prediction in the test split adds a zero; scikit-learn averages over the labels present. `rmse` and `mae` return 0 on an empty set, which reads as a perfect score. A stratified split sends a single-row class entirely to test. Decide each case against the V42 generator (add the absent-class case to it), then align or document. | The same honesty rule as V35: a number must not flatter or punish by accident. Measure the difference on the demo datasets before changing anything. | +| **V50 — delivered** | **Metrics that agree with the reference on every edge.** Measured before anything changed, then checked against scikit-learn rather than against our own reading of it. **(1) Macro averages** ran over every class the task knows; scikit-learn averages over the labels present in `y_true` or `y_pred`. A perfect score on a batch holding two of three classes read **0.667** instead of **1.0**. Both implementations (`macroPrf` and the V45 decision policy) now follow scikit-learn, and the V42 generator gained an `absentClass` case (four classes declared, three present, one predicted but never true) that the old code failed. **(2) Rare classes**: the stratified split sent a class's only row to test, and the validation carve did the same to a two-row class, so the models never saw an example of either. Measured on 100 rows plus a rare class: 1 row gave test 1 / train 0, 2 rows gave test 1 / validation 1 / train 0. A single-row class now stays on the training side (1 → train 1; 2 → test 1, train 1; 3 and more unchanged, and the seeded draw of every other class is byte-identical). This is not hypothetical: three targets the demos offer have one-row classes (titanic → parch, cafe-sales → city, cafe-sales → product). On titanic → parch every model's macro F1 rises once the empty classes stop counting (GBDT 0.619 → 0.716, ensemble 0.422 → 0.548), and the decomposition shows the metric change alone moves nothing while every class is present: the gain comes from the split. The validation-picked winner moves from GBDT to the forest because the validation rows changed, not because a metric did. When every class has one row, nothing is left to test on and training refuses by name (`too-few-rows`) instead of reporting metrics over an empty set. **(3) Empty sets**: accuracy, log loss, RMSE, MAE, R² and the macro trio now return NaN on zero rows; before, RMSE 0, MAE 0 and R² 1 described a perfect model measured on nothing. No caller passes an empty set today (the test split keeps a row, batch scoring skips unlabeled files), so this is a contract for the next caller; the one theoretical path left, tuning on fewer than four rows, is noted rather than guarded. **What this wave deliberately does not do**: change any figure on a dataset whose classes all have three rows or more (every demo target used by a test or a tutorial), add a warning banner for rare classes, or guard k-fold tuning on tiny data. 906 unit tests, 123 e2e. | Approved 09/10/2026 after V49. The same rule as V35: a number must not flatter or punish by accident. Two of the three edges were reachable from the demo picker, and the reference that would have caught the first one existed since V42 but held no case with an absent class. | | **V51 — pending** | **A store that fits on a screen.** `lab-store.ts` (about 1 000 lines, a 250-line `onmessage` chain) split into feature slices with a dispatch table; tests named after waves (`v35.test.ts` to `v40.test.ts`) renamed after the behaviour they pin; the duplicated missing-ratio and histogram code in `drift.ts` and `reference.ts` merged. No behaviour change, proven by the suite staying identical. | Only after V47 to V50: refactoring code whose defects are not yet pinned by tests moves the defects around instead of removing them. | | **V52 — pending** | **Major upgrades, one at a time, each measured.** Vitest 5, vite-plugin-pwa 2, TypeScript 7. Each in its own pull request with the before/after numbers that matter for it (suite time, build time, service-worker behaviour offline). | V44's rule: a major update keeps its migration cost visible. | diff --git a/README.md b/README.md index f6cb840..15dc58b 100644 --- a/README.md +++ b/README.md @@ -241,7 +241,7 @@ Wikimedia Commons), the portrait is NASA, public domain._ a doc page carries its whole article, in English until the app mounts); Lighthouse mobile ≈ 0.99 on `/ml` under real throttling. Heavy dependencies (Dexie, SheetJS, ONNX Runtime) load lazily. -- **Quality bar.** 898 unit tests and 123 Playwright end-to-end tests across five +- **Quality bar.** 906 unit tests and 123 Playwright end-to-end tests across five projects — desktop, a phone viewport in English and in French, dark mode, and Cloudflare Pages' own routing emulated by `wrangler pages dev` (a real 404, the security headers as served) — covering offline PWA, a fake webcam, a horizontal-overflow guard on diff --git a/docs/superpowers/plans/2026-10-09-v50-metric-edges.md b/docs/superpowers/plans/2026-10-09-v50-metric-edges.md new file mode 100644 index 0000000..f434a18 --- /dev/null +++ b/docs/superpowers/plans/2026-10-09-v50-metric-edges.md @@ -0,0 +1,48 @@ +# V50: metrics that agree with the reference on every edge + +## Measured first + +| Edge | Before | Reference | +| -------------------------------- | ------------------------------- | ------------------------------- | +| Perfect batch, 2 of 3 classes | macro F1 0.667 | scikit-learn 1.0 | +| Rare class, 1 row (100 + 1 rows) | test 1, train 0 | the class is never learned | +| Rare class, 2 rows | test 1, validation 1, train 0 | the class is never learned | +| Empty set | RMSE 0, MAE 0, R² 1, accuracy 0 | undefined (scikit-learn raises) | + +Demo targets with one-row classes: titanic → parch, cafe-sales → city, +cafe-sales → product. + +## Decomposition on titanic → parch (macro F1 per model) + +| Configuration | GBDT | Forest | Ensemble | +| --------------------- | ----- | ------ | -------- | +| old split, old metric | 0.619 | 0.310 | 0.422 | +| old split, new metric | 0.619 | 0.310 | 0.422 | +| new split, old metric | 0.614 | 0.340 | 0.470 | +| new split, new metric | 0.716 | 0.397 | 0.548 | + +The metric alone changes nothing while every class is in the test split. Once +one-row classes stay in training, the test split no longer holds them and the +new average stops counting them as zeros. The validation-picked winner moves +from GBDT to the forest because the validation rows changed. + +## Changes + +- `macroPrf` and `evaluateMulticlassPolicy`: average over classes present in + the true labels or the predictions/decisions. +- `splitIndices`: a one-row class stays in training; the RNG draw for every + other class is unchanged. +- `runTraining`: `too-few-rows` when the split leaves train or test empty. +- Every metric over zero rows returns NaN. +- `tests/golden/generate.py`: new `absentClass` case, regenerated with + scikit-learn 1.7.2 (Python 3.13.16). + +## Tests + +`src/features/ml/train/metric-edges.test.ts` (6 of 7 fail before) and the new +golden case (fails before). + +## Not done + +A rare-class warning in the interface, and a guard for k-fold tuning on fewer +than four training rows. diff --git a/package-lock.json b/package-lock.json index 0abec37..5677231 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "labml", - "version": "1.49.0", + "version": "1.50.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "labml", - "version": "1.49.0", + "version": "1.50.0", "license": "MIT", "dependencies": { "@duckdb/duckdb-wasm": "1.28.0", diff --git a/package.json b/package.json index 2ec1836..407b59c 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "labml", "private": true, - "version": "1.49.0", + "version": "1.50.0", "license": "MIT", "type": "module", "engines": { diff --git a/src/features/ml/train/golden.test.ts b/src/features/ml/train/golden.test.ts index b51165d..31d9448 100644 --- a/src/features/ml/train/golden.test.ts +++ b/src/features/ml/train/golden.test.ts @@ -54,6 +54,17 @@ describe('scikit-learn golden metrics', () => { expect(logLoss(fixture.yTrue, fixture.probabilities)).toBeCloseTo(fixture.logLoss!, 12); }); + it('averages macro metrics over the labels present, as scikit-learn does', () => { + // V50: a class in neither y_true nor y_pred is not scored as a zero; a + // class that is only predicted still is. + const fixture = reference.metrics.absentClass; + expect(macroPrf(fixture.yTrue, fixture.yPred, fixture.classCount)).toEqual({ + precision: expect.closeTo(fixture.precision, 12), + recall: expect.closeTo(fixture.recall, 12), + f1: expect.closeTo(fixture.f1, 12), + }); + }); + it('matches binary ROC-AUC, including tied scores', () => { const fixture = reference.metrics.binaryRoc; expect(rocAuc(fixture.yTrue, fixture.scores)).toBeCloseTo(fixture.rocAuc!, 12); diff --git a/src/features/ml/train/metric-edges.test.ts b/src/features/ml/train/metric-edges.test.ts new file mode 100644 index 0000000..ac1c063 --- /dev/null +++ b/src/features/ml/train/metric-edges.test.ts @@ -0,0 +1,95 @@ +import { describe, expect, it } from 'vitest'; +import { profileColumn } from '@/features/ml/data/profile'; +import { accuracy, logLoss, macroPrf, mae, r2, rmse } from '@/features/ml/train/metrics'; +import { evaluateMulticlassPolicy } from '@/features/ml/train/multiclass-decision'; +import { splitIndices } from '@/features/ml/train/pipeline'; +import { runTraining } from '@/features/ml/train/trainer'; +import type { Cell } from '@/features/ml/data/types'; + +/** + * V50 — three edges where a metric flattered or punished by accident. Each + * expectation below failed on the code before V50; the measurement is in + * PLAN.md under the V50 row. + */ +describe('macro averages count only the classes present', () => { + it('scores a perfect batch that holds two of three classes as perfect', () => { + // Before: 0.667 — the absent third class was averaged in as a zero. + expect(macroPrf([0, 0, 1, 1], [0, 0, 1, 1], 3)).toEqual({ precision: 1, recall: 1, f1: 1 }); + }); + + it('applies the same rule to the multiclass decision policy', () => { + const probabilities = [ + [0.9, 0.05, 0.05], + [0.8, 0.1, 0.1], + [0.1, 0.85, 0.05], + [0.05, 0.9, 0.05], + ]; + const metrics = evaluateMulticlassPolicy([0, 0, 1, 1], probabilities, { + thresholds: [0, 0, 0], + }); + expect(metrics.macroF1).toBe(1); + }); +}); + +describe('a metric over nothing is not a result', () => { + it('returns NaN rather than a perfect or a null score', () => { + // Before: RMSE 0, MAE 0 and R² 1 — a perfect model, measured on nothing. + for (const value of [ + accuracy([], []), + logLoss([], []), + rmse([], []), + mae([], []), + r2([], []), + ]) { + expect(value).toBeNaN(); + } + expect(macroPrf([], [], 3).f1).toBeNaN(); + }); +}); + +describe('a rare class is learned before it is tested', () => { + const split = (rare: number) => { + const rows = Array.from({ length: 100 + rare }, (_, i) => i); + const labels = rows.map((i) => (i < 100 ? (i % 2 ? 'a' : 'b') : 'rare')); + const first = splitIndices(rows, labels, 0.2, 42); + const second = splitIndices( + first.train, + first.train.map((i) => labels[i]), + 0.2, + 43, + ); + const count = (indices: number[]) => indices.filter((i) => i >= 100).length; + return { test: count(first.test), validation: count(second.test), train: count(second.train) }; + }; + + it('keeps a one-row class for training', () => { + // Before: test 1, train 0 — the models never saw an example of it. + expect(split(1)).toEqual({ test: 0, validation: 0, train: 1 }); + }); + + it('keeps one row of a two-row class for training', () => { + // Before: test 1, validation 1, train 0. + expect(split(2)).toEqual({ test: 1, validation: 0, train: 1 }); + }); + + it('leaves larger classes exactly where they were', () => { + expect(split(5)).toEqual({ test: 1, validation: 1, train: 3 }); + }); + + it('refuses by name when nothing is left to test on', async () => { + const data: Record = { + x: ['1', '2', '3', '4'], + label: ['a', 'b', 'c', 'd'], + }; + const columns = new Map(Object.entries(data)); + const profiles = Object.entries(data).map(([name, values]) => profileColumn(name, values)); + await expect( + runTraining( + columns, + profiles, + { target: 'label', features: ['x'], seed: 42, testRatio: 0.2 }, + { onModelStart: () => undefined, onModelResult: () => undefined, isCancelled: () => false }, + ), + ).rejects.toThrow('too-few-rows'); + }); +}); diff --git a/src/features/ml/train/metrics.ts b/src/features/ml/train/metrics.ts index 3238bce..ab0f1ff 100644 --- a/src/features/ml/train/metrics.ts +++ b/src/features/ml/train/metrics.ts @@ -1,12 +1,29 @@ const EPSILON = 1e-15; +/* + * V50: every metric over an EMPTY set is NaN, not a number that reads as a + * result. Before, an empty set scored RMSE 0, MAE 0 and R² 1: a perfect + * model, measured on nothing. No caller passes an empty set today (the test + * split keeps at least one row, batch scoring skips files with no label), so + * this is a contract for the next caller rather than a visible fix. + */ + export function accuracy(yTrue: number[], yPred: number[]): number { let correct = 0; for (let i = 0; i < yTrue.length; i++) if (yTrue[i] === yPred[i]) correct += 1; - return yTrue.length === 0 ? 0 : correct / yTrue.length; + return yTrue.length === 0 ? Number.NaN : correct / yTrue.length; } -/** Macro-averaged precision, recall and F1 over the given number of classes. */ +/** + * Macro-averaged precision, recall and F1. + * + * V50: averaged over the classes PRESENT in `yTrue` or `yPred`, as + * scikit-learn does, not over every class the task knows. A class that + * appears on neither side has nothing to be right or wrong about; counting + * it as a zero made a perfect score on a batch holding two of three classes + * read 0.667. A class that is predicted but never true still counts, as a + * zero, because those predictions are all wrong. + */ export function macroPrf( yTrue: number[], yPred: number[], @@ -15,6 +32,7 @@ export function macroPrf( let precisionSum = 0; let recallSum = 0; let f1Sum = 0; + let present = 0; for (let c = 0; c < classCount; c++) { let tp = 0; let fp = 0; @@ -26,6 +44,8 @@ export function macroPrf( else if (!isTrue && isPred) fp += 1; else if (isTrue && !isPred) fn += 1; } + if (tp + fp + fn === 0) continue; + present += 1; const precision = tp + fp === 0 ? 0 : tp / (tp + fp); const recall = tp + fn === 0 ? 0 : tp / (tp + fn); const f1 = precision + recall === 0 ? 0 : (2 * precision * recall) / (precision + recall); @@ -33,10 +53,11 @@ export function macroPrf( recallSum += recall; f1Sum += f1; } + if (present === 0) return { precision: Number.NaN, recall: Number.NaN, f1: Number.NaN }; return { - precision: precisionSum / classCount, - recall: recallSum / classCount, - f1: f1Sum / classCount, + precision: precisionSum / present, + recall: recallSum / present, + f1: f1Sum / present, }; } @@ -47,7 +68,7 @@ export function logLoss(yTrue: number[], probabilities: number[][]): number { const p = Math.min(1 - EPSILON, Math.max(EPSILON, probabilities[i][yTrue[i]] ?? EPSILON)); sum += -Math.log(p); } - return yTrue.length === 0 ? 0 : sum / yTrue.length; + return yTrue.length === 0 ? Number.NaN : sum / yTrue.length; } /** @@ -81,16 +102,17 @@ export function rocAuc(yTrue: number[], scores: number[]): number | null { export function rmse(yTrue: number[], yPred: number[]): number { let sum = 0; for (let i = 0; i < yTrue.length; i++) sum += (yTrue[i] - yPred[i]) ** 2; - return yTrue.length === 0 ? 0 : Math.sqrt(sum / yTrue.length); + return yTrue.length === 0 ? Number.NaN : Math.sqrt(sum / yTrue.length); } export function mae(yTrue: number[], yPred: number[]): number { let sum = 0; for (let i = 0; i < yTrue.length; i++) sum += Math.abs(yTrue[i] - yPred[i]); - return yTrue.length === 0 ? 0 : sum / yTrue.length; + return yTrue.length === 0 ? Number.NaN : sum / yTrue.length; } export function r2(yTrue: number[], yPred: number[]): number { + if (yTrue.length === 0) return Number.NaN; const mean = yTrue.reduce((a, v) => a + v, 0) / (yTrue.length || 1); let ssRes = 0; let ssTot = 0; diff --git a/src/features/ml/train/multiclass-decision.ts b/src/features/ml/train/multiclass-decision.ts index 3163d19..7ebf5ba 100644 --- a/src/features/ml/train/multiclass-decision.ts +++ b/src/features/ml/train/multiclass-decision.ts @@ -121,8 +121,15 @@ export function evaluateMulticlassPolicy( const f1 = precision + recall === 0 ? 0 : (2 * precision * recall) / (precision + recall); return { classIndex, support: classSupport, precision, recall, f1 }; }); + // V50: the same rule as `macroPrf` and scikit-learn — the macro average + // runs over the classes present among the true labels or the decisions. + const present = perClass.filter( + (metrics) => metrics.support > 0 || falsePositives[metrics.classIndex] > 0, + ); const sum = (field: 'precision' | 'recall' | 'f1') => - perClass.reduce((total, metrics) => total + metrics[field], 0) / classCount; + present.length === 0 + ? Number.NaN + : present.reduce((total, metrics) => total + metrics[field], 0) / present.length; const rows = labels.length; return { diff --git a/src/features/ml/train/pipeline.ts b/src/features/ml/train/pipeline.ts index 45fe07c..1d55997 100644 --- a/src/features/ml/train/pipeline.ts +++ b/src/features/ml/train/pipeline.ts @@ -240,7 +240,12 @@ export function splitIndices( }); for (const bucket of [...byClass.keys()].sort().map((k) => byClass.get(k)!)) { shuffleInPlace(bucket, rng); - const testCount = Math.max(1, Math.round(bucket.length * testRatio)); + // V50: a class with a single row stays on the training side. Sending it + // to test (the old `max(1, …)`) left the models with no example of it at + // all, and the validation carve repeated the loss for two-row classes. + // An untested class is announced by its absence from the test metrics; + // an unlearnable one was not announced anywhere. + const testCount = bucket.length < 2 ? 0 : Math.max(1, Math.round(bucket.length * testRatio)); append(test, bucket, 0, testCount); append(train, bucket, testCount, bucket.length); } diff --git a/src/features/ml/train/trainer.ts b/src/features/ml/train/trainer.ts index 2689aa6..7439339 100644 --- a/src/features/ml/train/trainer.ts +++ b/src/features/ml/train/trainer.ts @@ -215,6 +215,10 @@ export function prepareData( train = second.train; validation = second.test; } + // V50: when every class holds a single row, the stratified split keeps + // them all for training and nothing is left to test on. Refused by name + // rather than reported as metrics computed over an empty set. + if (train.length === 0 || test.length === 0) throw new Error('too-few-rows'); } const trainLabels = isClassification diff --git a/tests/golden/generate.py b/tests/golden/generate.py index 59bb486..04c7b22 100644 --- a/tests/golden/generate.py +++ b/tests/golden/generate.py @@ -54,7 +54,25 @@ def metrics() -> dict[str, object]: regression_true = [3.0, -0.5, 2.0, 7.0, 4.5, 1.25] regression_pred = [2.5, 0.0, 2.0, 8.0, 3.75, 1.5] + # V50: the task knows four classes, the evaluated rows carry three of + # them, and class 2 is predicted but never true. scikit-learn's macro + # average runs over the labels present in y_true or y_pred, so class 3 + # (in neither) does not count and class 2 (predicted only) counts as 0. + # A batch scored on a subset of the classes is the everyday case. + absent_true = [0, 0, 0, 1, 1, 1, 1] + absent_pred = [0, 0, 1, 1, 1, 2, 1] + return { + "absentClass": { + "classCount": 4, + "yTrue": absent_true, + "yPred": absent_pred, + "precision": precision_score( + absent_true, absent_pred, average="macro", zero_division=0 + ), + "recall": recall_score(absent_true, absent_pred, average="macro", zero_division=0), + "f1": f1_score(absent_true, absent_pred, average="macro", zero_division=0), + }, "classification": { "yTrue": classification_true, "yPred": classification_pred, diff --git a/tests/golden/sklearn-reference.json b/tests/golden/sklearn-reference.json index 99d16d2..5d232cb 100644 --- a/tests/golden/sklearn-reference.json +++ b/tests/golden/sklearn-reference.json @@ -1,10 +1,34 @@ { "provenance": { "generator": "tests/golden/generate.py", - "python": "3.13.15", + "python": "3.13.16", "scikitLearn": "1.7.2" }, "metrics": { + "absentClass": { + "classCount": 4, + "yTrue": [ + 0, + 0, + 0, + 1, + 1, + 1, + 1 + ], + "yPred": [ + 0, + 0, + 1, + 1, + 1, + 2, + 1 + ], + "precision": 0.5833333333333334, + "recall": 0.47222222222222215, + "f1": 0.5166666666666667 + }, "classification": { "yTrue": [ 0,