V50: metrics that agree with the reference on every edge

CHANGELOG.md

@@ -6,6 +6,12 @@ entry per wave, newest first; the first six waves (V1–V6, the MVP) predate the
 are recorded in PLAN.md §B and §J. Each entry carries the opening sentence of its row and
 the reason the wave was built.
 
+## V50 — Metrics that agree with the reference on every edge
+
+Measured before anything changed, then checked against scikit-learn rather than against our own reading of it.
+
+_Why:_ Approved 09/10/2026 after V49. The same rule as V35: a number must not flatter or punish by accident. Two of the three edges were reachable from the demo picker, and the reference that would have caught the first one existed since V42 but held no case with an absent class.
+
 ## V49 — CI that cannot be steered by a branch name
 
 The wave opened with zizmor 1.30.1 on the three workflows: **28 findings** (1 template injection, 21 actions pinned by a movable tag, 6 checkouts leaving the git token on the runner disk).

README.md

@@ -241,7 +241,7 @@ Wikimedia Commons), the portrait is NASA, public domain._
   a doc page carries its whole article, in English until the app mounts); Lighthouse
   mobile ≈ 0.99 on `/ml` under real throttling. Heavy dependencies (Dexie, SheetJS, ONNX
   Runtime) load lazily.
-- **Quality bar.** 898 unit tests and 123 Playwright end-to-end tests across five
+- **Quality bar.** 906 unit tests and 123 Playwright end-to-end tests across five
   projects — desktop, a phone viewport in English and in French, dark mode, and
   Cloudflare Pages' own routing emulated by `wrangler pages dev` (a real 404, the security
   headers as served) — covering offline PWA, a fake webcam, a horizontal-overflow guard on

docs/superpowers/plans/2026-10-09-v50-metric-edges.md

@@ -0,0 +1,48 @@
+# V50: metrics that agree with the reference on every edge
+
+## Measured first
+
+| Edge                             | Before                          | Reference                       |
+| -------------------------------- | ------------------------------- | ------------------------------- |
+| Perfect batch, 2 of 3 classes    | macro F1 0.667                  | scikit-learn 1.0                |
+| Rare class, 1 row (100 + 1 rows) | test 1, train 0                 | the class is never learned      |
+| Rare class, 2 rows               | test 1, validation 1, train 0   | the class is never learned      |
+| Empty set                        | RMSE 0, MAE 0, R² 1, accuracy 0 | undefined (scikit-learn raises) |
+
+Demo targets with one-row classes: titanic → parch, cafe-sales → city,
+cafe-sales → product.
+
+## Decomposition on titanic → parch (macro F1 per model)
+
+| Configuration         | GBDT  | Forest | Ensemble |
+| --------------------- | ----- | ------ | -------- |
+| old split, old metric | 0.619 | 0.310  | 0.422    |
+| old split, new metric | 0.619 | 0.310  | 0.422    |
+| new split, old metric | 0.614 | 0.340  | 0.470    |
+| new split, new metric | 0.716 | 0.397  | 0.548    |
+
+The metric alone changes nothing while every class is in the test split. Once
+one-row classes stay in training, the test split no longer holds them and the
+new average stops counting them as zeros. The validation-picked winner moves
+from GBDT to the forest because the validation rows changed.
+
+## Changes
+
+- `macroPrf` and `evaluateMulticlassPolicy`: average over classes present in
+  the true labels or the predictions/decisions.
+- `splitIndices`: a one-row class stays in training; the RNG draw for every
+  other class is unchanged.
+- `runTraining`: `too-few-rows` when the split leaves train or test empty.
+- Every metric over zero rows returns NaN.
+- `tests/golden/generate.py`: new `absentClass` case, regenerated with
+  scikit-learn 1.7.2 (Python 3.13.16).
+
+## Tests
+
+`src/features/ml/train/metric-edges.test.ts` (6 of 7 fail before) and the new
+golden case (fails before).
+
+## Not done
+
+A rare-class warning in the interface, and a guard for k-fold tuning on fewer
+than four training rows.

package.json

@@ -1,7 +1,7 @@
 {
   "name": "labml",
   "private": true,
-  "version": "1.49.0",
+  "version": "1.50.0",
   "license": "MIT",
   "type": "module",
   "engines": {

src/features/ml/train/golden.test.ts

@@ -54,6 +54,17 @@ describe('scikit-learn golden metrics', () => {
     expect(logLoss(fixture.yTrue, fixture.probabilities)).toBeCloseTo(fixture.logLoss!, 12);
   });
 
+  it('averages macro metrics over the labels present, as scikit-learn does', () => {
+    // V50: a class in neither y_true nor y_pred is not scored as a zero; a
+    // class that is only predicted still is.
+    const fixture = reference.metrics.absentClass;
+    expect(macroPrf(fixture.yTrue, fixture.yPred, fixture.classCount)).toEqual({
+      precision: expect.closeTo(fixture.precision, 12),
+      recall: expect.closeTo(fixture.recall, 12),
+      f1: expect.closeTo(fixture.f1, 12),
+    });
+  });
+
   it('matches binary ROC-AUC, including tied scores', () => {
     const fixture = reference.metrics.binaryRoc;
     expect(rocAuc(fixture.yTrue, fixture.scores)).toBeCloseTo(fixture.rocAuc!, 12);

src/features/ml/train/metric-edges.test.ts

@@ -0,0 +1,95 @@
+import { describe, expect, it } from 'vitest';
+import { profileColumn } from '@/features/ml/data/profile';
+import { accuracy, logLoss, macroPrf, mae, r2, rmse } from '@/features/ml/train/metrics';
+import { evaluateMulticlassPolicy } from '@/features/ml/train/multiclass-decision';
+import { splitIndices } from '@/features/ml/train/pipeline';
+import { runTraining } from '@/features/ml/train/trainer';
+import type { Cell } from '@/features/ml/data/types';
+
+/**
+ * V50 — three edges where a metric flattered or punished by accident. Each
+ * expectation below failed on the code before V50; the measurement is in
+ * PLAN.md under the V50 row.
+ */
+describe('macro averages count only the classes present', () => {
+  it('scores a perfect batch that holds two of three classes as perfect', () => {
+    // Before: 0.667 — the absent third class was averaged in as a zero.
+    expect(macroPrf([0, 0, 1, 1], [0, 0, 1, 1], 3)).toEqual({ precision: 1, recall: 1, f1: 1 });
+  });
+
+  it('applies the same rule to the multiclass decision policy', () => {
+    const probabilities = [
+      [0.9, 0.05, 0.05],
+      [0.8, 0.1, 0.1],
+      [0.1, 0.85, 0.05],
+      [0.05, 0.9, 0.05],
+    ];
+    const metrics = evaluateMulticlassPolicy([0, 0, 1, 1], probabilities, {
+      thresholds: [0, 0, 0],
+    });
+    expect(metrics.macroF1).toBe(1);
+  });
+});
+
+describe('a metric over nothing is not a result', () => {
+  it('returns NaN rather than a perfect or a null score', () => {
+    // Before: RMSE 0, MAE 0 and R² 1 — a perfect model, measured on nothing.
+    for (const value of [
+      accuracy([], []),
+      logLoss([], []),
+      rmse([], []),
+      mae([], []),
+      r2([], []),
+    ]) {
+      expect(value).toBeNaN();
+    }
+    expect(macroPrf([], [], 3).f1).toBeNaN();
+  });
+});
+
+describe('a rare class is learned before it is tested', () => {
+  const split = (rare: number) => {
+    const rows = Array.from({ length: 100 + rare }, (_, i) => i);
+    const labels = rows.map((i) => (i < 100 ? (i % 2 ? 'a' : 'b') : 'rare'));
+    const first = splitIndices(rows, labels, 0.2, 42);
+    const second = splitIndices(
+      first.train,
+      first.train.map((i) => labels[i]),
+      0.2,
+      43,
+    );
+    const count = (indices: number[]) => indices.filter((i) => i >= 100).length;
+    return { test: count(first.test), validation: count(second.test), train: count(second.train) };
+  };
+
+  it('keeps a one-row class for training', () => {
+    // Before: test 1, train 0 — the models never saw an example of it.
+    expect(split(1)).toEqual({ test: 0, validation: 0, train: 1 });
+  });
+
+  it('keeps one row of a two-row class for training', () => {
+    // Before: test 1, validation 1, train 0.
+    expect(split(2)).toEqual({ test: 1, validation: 0, train: 1 });
+  });
+
+  it('leaves larger classes exactly where they were', () => {
+    expect(split(5)).toEqual({ test: 1, validation: 1, train: 3 });
+  });
+
+  it('refuses by name when nothing is left to test on', async () => {
+    const data: Record<string, Cell[]> = {
+      x: ['1', '2', '3', '4'],
+      label: ['a', 'b', 'c', 'd'],
+    };
+    const columns = new Map(Object.entries(data));
+    const profiles = Object.entries(data).map(([name, values]) => profileColumn(name, values));
+    await expect(
+      runTraining(
+        columns,
+        profiles,
+        { target: 'label', features: ['x'], seed: 42, testRatio: 0.2 },
+        { onModelStart: () => undefined, onModelResult: () => undefined, isCancelled: () => false },
+      ),
+    ).rejects.toThrow('too-few-rows');
+  });
+});

src/features/ml/train/metrics.ts

@@ -1,12 +1,29 @@
 const EPSILON = 1e-15;
 
+/*
+ * V50: every metric over an EMPTY set is NaN, not a number that reads as a
+ * result. Before, an empty set scored RMSE 0, MAE 0 and R² 1: a perfect
+ * model, measured on nothing. No caller passes an empty set today (the test
+ * split keeps at least one row, batch scoring skips files with no label), so
+ * this is a contract for the next caller rather than a visible fix.
+ */
+
 export function accuracy(yTrue: number[], yPred: number[]): number {
   let correct = 0;
   for (let i = 0; i < yTrue.length; i++) if (yTrue[i] === yPred[i]) correct += 1;
-  return yTrue.length === 0 ? 0 : correct / yTrue.length;
+  return yTrue.length === 0 ? Number.NaN : correct / yTrue.length;
 }
 
-/** Macro-averaged precision, recall and F1 over the given number of classes. */
+/**
+ * Macro-averaged precision, recall and F1.
+ *
+ * V50: averaged over the classes PRESENT in `yTrue` or `yPred`, as
+ * scikit-learn does, not over every class the task knows. A class that
+ * appears on neither side has nothing to be right or wrong about; counting
+ * it as a zero made a perfect score on a batch holding two of three classes
+ * read 0.667. A class that is predicted but never true still counts, as a
+ * zero, because those predictions are all wrong.
+ */
 export function macroPrf(
   yTrue: number[],
   yPred: number[],
@@ -15,6 +32,7 @@ export function macroPrf(
   let precisionSum = 0;
   let recallSum = 0;
   let f1Sum = 0;
+  let present = 0;
   for (let c = 0; c < classCount; c++) {
     let tp = 0;
     let fp = 0;
@@ -26,17 +44,20 @@ export function macroPrf(
       else if (!isTrue && isPred) fp += 1;
       else if (isTrue && !isPred) fn += 1;
     }
+    if (tp + fp + fn === 0) continue;
+    present += 1;
     const precision = tp + fp === 0 ? 0 : tp / (tp + fp);
     const recall = tp + fn === 0 ? 0 : tp / (tp + fn);
     const f1 = precision + recall === 0 ? 0 : (2 * precision * recall) / (precision + recall);
     precisionSum += precision;
     recallSum += recall;
     f1Sum += f1;
   }
+  if (present === 0) return { precision: Number.NaN, recall: Number.NaN, f1: Number.NaN };
   return {
-    precision: precisionSum / classCount,
-    recall: recallSum / classCount,
-    f1: f1Sum / classCount,
+    precision: precisionSum / present,
+    recall: recallSum / present,
+    f1: f1Sum / present,
   };
 }
 
@@ -47,7 +68,7 @@ export function logLoss(yTrue: number[], probabilities: number[][]): number {
     const p = Math.min(1 - EPSILON, Math.max(EPSILON, probabilities[i][yTrue[i]] ?? EPSILON));
     sum += -Math.log(p);
   }
-  return yTrue.length === 0 ? 0 : sum / yTrue.length;
+  return yTrue.length === 0 ? Number.NaN : sum / yTrue.length;
 }
 
 /**
@@ -81,16 +102,17 @@ export function rocAuc(yTrue: number[], scores: number[]): number | null {
 export function rmse(yTrue: number[], yPred: number[]): number {
   let sum = 0;
   for (let i = 0; i < yTrue.length; i++) sum += (yTrue[i] - yPred[i]) ** 2;
-  return yTrue.length === 0 ? 0 : Math.sqrt(sum / yTrue.length);
+  return yTrue.length === 0 ? Number.NaN : Math.sqrt(sum / yTrue.length);
 }
 
 export function mae(yTrue: number[], yPred: number[]): number {
   let sum = 0;
   for (let i = 0; i < yTrue.length; i++) sum += Math.abs(yTrue[i] - yPred[i]);
-  return yTrue.length === 0 ? 0 : sum / yTrue.length;
+  return yTrue.length === 0 ? Number.NaN : sum / yTrue.length;
 }
 
 export function r2(yTrue: number[], yPred: number[]): number {
+  if (yTrue.length === 0) return Number.NaN;
   const mean = yTrue.reduce((a, v) => a + v, 0) / (yTrue.length || 1);
   let ssRes = 0;
   let ssTot = 0;

src/features/ml/train/multiclass-decision.ts

@@ -121,8 +121,15 @@ export function evaluateMulticlassPolicy(
     const f1 = precision + recall === 0 ? 0 : (2 * precision * recall) / (precision + recall);
     return { classIndex, support: classSupport, precision, recall, f1 };
   });
+  // V50: the same rule as `macroPrf` and scikit-learn — the macro average
+  // runs over the classes present among the true labels or the decisions.
+  const present = perClass.filter(
+    (metrics) => metrics.support > 0 || falsePositives[metrics.classIndex] > 0,
+  );
   const sum = (field: 'precision' | 'recall' | 'f1') =>
-    perClass.reduce((total, metrics) => total + metrics[field], 0) / classCount;
+    present.length === 0
+      ? Number.NaN
+      : present.reduce((total, metrics) => total + metrics[field], 0) / present.length;
   const rows = labels.length;
 
   return {

src/features/ml/train/pipeline.ts

@@ -240,7 +240,12 @@ export function splitIndices(
     });
     for (const bucket of [...byClass.keys()].sort().map((k) => byClass.get(k)!)) {
       shuffleInPlace(bucket, rng);
-      const testCount = Math.max(1, Math.round(bucket.length * testRatio));
+      // V50: a class with a single row stays on the training side. Sending it
+      // to test (the old `max(1, …)`) left the models with no example of it at
+      // all, and the validation carve repeated the loss for two-row classes.
+      // An untested class is announced by its absence from the test metrics;
+      // an unlearnable one was not announced anywhere.
+      const testCount = bucket.length < 2 ? 0 : Math.max(1, Math.round(bucket.length * testRatio));
       append(test, bucket, 0, testCount);
       append(train, bucket, testCount, bucket.length);
     }

src/features/ml/train/trainer.ts

@@ -215,6 +215,10 @@ export function prepareData(
       train = second.train;
       validation = second.test;
     }
+    // V50: when every class holds a single row, the stratified split keeps
+    // them all for training and nothing is left to test on. Refused by name
+    // rather than reported as metrics computed over an empty set.
+    if (train.length === 0 || test.length === 0) throw new Error('too-few-rows');
   }
 
   const trainLabels = isClassification

tests/golden/generate.py

@@ -54,7 +54,25 @@ def metrics() -> dict[str, object]:
     regression_true = [3.0, -0.5, 2.0, 7.0, 4.5, 1.25]
     regression_pred = [2.5, 0.0, 2.0, 8.0, 3.75, 1.5]
 
+    # V50: the task knows four classes, the evaluated rows carry three of
+    # them, and class 2 is predicted but never true. scikit-learn's macro
+    # average runs over the labels present in y_true or y_pred, so class 3
+    # (in neither) does not count and class 2 (predicted only) counts as 0.
+    # A batch scored on a subset of the classes is the everyday case.
+    absent_true = [0, 0, 0, 1, 1, 1, 1]
+    absent_pred = [0, 0, 1, 1, 1, 2, 1]
+
     return {
+        "absentClass": {
+            "classCount": 4,
+            "yTrue": absent_true,
+            "yPred": absent_pred,
+            "precision": precision_score(
+                absent_true, absent_pred, average="macro", zero_division=0
+            ),
+            "recall": recall_score(absent_true, absent_pred, average="macro", zero_division=0),
+            "f1": f1_score(absent_true, absent_pred, average="macro", zero_division=0),
+        },
         "classification": {
             "yTrue": classification_true,
             "yPred": classification_pred,