Skip to content

V50: metrics that agree with the reference on every edge - #112

Merged
dapiced merged 1 commit into
mainfrom
claude/labml-improvements-ipscw1
Oct 9, 2026
Merged

dapiced merged 1 commit into
mainfrom
claude/labml-improvements-ipscw1

Conversation

@dapiced

@dapiced dapiced commented Oct 9, 2026

Copy link
Copy Markdown
Owner

Summary

Each edge below was measured before any code changed, and is now checked against scikit-learn instead of our own reading of it.

Edge Before After
Perfect batch containing 2 of 3 classes macro F1 0.667 1.0 (scikit-learn)
Class with 1 row (100 + 1 rows) test 1, train 0 train 1
Class with 2 rows test 1, validation 1, train 0 test 1, train 1
Empty set RMSE 0, MAE 0, R² 1 NaN

1. Macro averages. macroPrf and the V45 decision policy now average only over the labels present in y_true or y_pred, like scikit-learn. Previously they averaged over every class the task knows, so absent classes counted as zeros.

  • The V42 generator gets a new absentClass case: four classes declared, three present, and one predicted but never true. The old code failed it.
  • I regenerated the reference with scikit-learn 1.7.2. Apart from the new case, the only diff is the Python patch version.

2. Rare classes. A class with a single row now stays on the training side. Before, it went entirely to test, and a two-row class lost its other row to the validation carve.

  • Classes with 3 or more rows are unaffected, and the seeded draw for every other class is byte-identical.
  • If every class has one row, nothing is left to test on. Training now stops with the named refusal too-few-rows instead of reporting metrics over an empty set.

3. Empty sets. Every metric computed over zero rows now returns NaN. No caller passes an empty set today, so this is a contract for future callers.

Measurement on real demos

Three targets offered by the demos have one-row classes: titanic → parch, cafe-sales → city, and cafe-sales → product. Macro F1 on titanic → parch:

Configuration GBDT Forest Ensemble
old split, old metric 0.619 0.310 0.422
old split, new metric 0.619 0.310 0.422
new split, old metric 0.614 0.340 0.470
new split, new metric 0.716 0.397 0.548
  • With the old split, every class is in the test set, so the new metric changes nothing.
  • The gain comes from the split. Once one-row classes stay in training, the test set no longer contains them, and the new average stops counting them as zeros.
  • The model picked on validation changes from GBDT to the forest, because the validation rows changed. This is a selection effect, not a metric change.

Test plan

  • metric-edges.test.ts: 6 of 7 fail before the fix, all pass after. The new golden case also fails before.
  • npm run test:coverage: 906 passed, floors met (64.6 / 53.47 / 54.48 / 65.66)
  • lint, typecheck, format:check, build
  • Playwright, Chromium: 97/97

🤖 Generated with Claude Code

https://claude.ai/code/session_01ELAjHYDyxa5zbbdcke1bMA


Generated by Claude Code

- macroPrf and the multiclass decision policy average over the classes
  present in y_true or y_pred, as scikit-learn does (a perfect batch on
  2 of 3 classes read 0.667)
- a one-row class stays in training (1- and 2-row classes got no
  training rows); too-few-rows when nothing is left to test on
- every metric over zero rows is NaN (RMSE 0 / R2 1 read as perfect)
- golden: new absentClass case, regenerated with scikit-learn 1.7.2
- metric-edges.test.ts pins each edge (6 of 7 fail before)
- PLAN/CHANGELOG: V50 delivered with the measurement, version 1.50.0

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELAjHYDyxa5zbbdcke1bMA
@dapiced
dapiced marked this pull request as ready for review October 9, 2026 08:39
@dapiced
dapiced merged commit 0c5b0ab into main Oct 9, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants