Repository navigation
V50: metrics that agree with the reference on every edge - #112
Merged
Merged
Conversation
- macroPrf and the multiclass decision policy average over the classes present in y_true or y_pred, as scikit-learn does (a perfect batch on 2 of 3 classes read 0.667) - a one-row class stays in training (1- and 2-row classes got no training rows); too-few-rows when nothing is left to test on - every metric over zero rows is NaN (RMSE 0 / R2 1 read as perfect) - golden: new absentClass case, regenerated with scikit-learn 1.7.2 - metric-edges.test.ts pins each edge (6 of 7 fail before) - PLAN/CHANGELOG: V50 delivered with the measurement, version 1.50.0 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELAjHYDyxa5zbbdcke1bMA
dapiced
marked this pull request as ready for review
October 9, 2026 08:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Each edge below was measured before any code changed, and is now checked against scikit-learn instead of our own reading of it.
1. Macro averages.
macroPrfand the V45 decision policy now average only over the labels present iny_trueory_pred, like scikit-learn. Previously they averaged over every class the task knows, so absent classes counted as zeros.absentClasscase: four classes declared, three present, and one predicted but never true. The old code failed it.2. Rare classes. A class with a single row now stays on the training side. Before, it went entirely to test, and a two-row class lost its other row to the validation carve.
too-few-rowsinstead of reporting metrics over an empty set.3. Empty sets. Every metric computed over zero rows now returns NaN. No caller passes an empty set today, so this is a contract for future callers.
Measurement on real demos
Three targets offered by the demos have one-row classes: titanic → parch, cafe-sales → city, and cafe-sales → product. Macro F1 on titanic → parch:
Test plan
metric-edges.test.ts: 6 of 7 fail before the fix, all pass after. The new golden case also fails before.npm run test:coverage: 906 passed, floors met (64.6 / 53.47 / 54.48 / 65.66)lint,typecheck,format:check,build🤖 Generated with Claude Code
https://claude.ai/code/session_01ELAjHYDyxa5zbbdcke1bMA
Generated by Claude Code