Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@ entry per wave, newest first; the first six waves (V1–V6, the MVP) predate the
are recorded in PLAN.md §B and §J. Each entry carries the opening sentence of its row and
the reason the wave was built.

## V51 — A store that fits on a screen

No behaviour change, and the proof is the suite: the 906 unit tests and the 97 Chromium e2e tests that passed before pass unchanged after.

_Why:_ Approved 09/10/2026 after V50. Only after V47 to V50, as the plan required: the store's concurrency rules and the metrics were pinned by tests before their code moved, so a regression in the move would have failed loudly.

## V50 — Metrics that agree with the reference on every edge

Measured before anything changed, then checked against scikit-learn rather than against our own reading of it.
Expand Down
2 changes: 1 addition & 1 deletion PLAN.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,7 @@ Wikimedia Commons), the portrait is NASA, public domain._
a doc page carries its whole article, in English until the app mounts); Lighthouse
mobile ≈ 0.99 on `/ml` under real throttling. Heavy dependencies (Dexie, SheetJS, ONNX
Runtime) load lazily.
- **Quality bar.** 906 unit tests and 123 Playwright end-to-end tests across five
- **Quality bar.** 923 unit tests and 123 Playwright end-to-end tests across five
projects — desktop, a phone viewport in English and in French, dark mode, and
Cloudflare Pages' own routing emulated by `wrangler pages dev` (a real 404, the security
headers as served) — covering offline PWA, a fake webcam, a horizontal-overflow guard on
Expand Down
34 changes: 34 additions & 0 deletions docs/superpowers/plans/2026-10-09-v51-store-refactor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# V51: a store that fits on a screen

## Rule

No behaviour change. The proof is the suite: 906 unit tests and 97 Chromium
e2e tests pass unchanged before and after.

## Changes

| Before | After |
| -------------------------------------- | ---------------------------------------------------------- |
| `lab-store.ts`, 1 050 lines | 530 lines: actions only, public names re-exported |
| 250-line `onmessage` `if / else` chain | `store/responses.ts`, one typed entry per `WorkerResponse` |
| State shape and initial values inline | `store/state.ts` |
| V47 fence inline | `store/fence.ts` |
| `thresholdArtifact` inside the closure | `store/artifacts.ts` (pure) |
| `v35` … `v40` test files | named after what they pin |
| Helpers copied in drift and reference | `quality/distribution.ts` |

The dispatch table is a mapped type over `WorkerResponse['kind']`: a missing
or extra handler is a compile error. The old chain's final `else` handled any
unlisted kind as `error` without saying so.

## Coverage gate

Branches became functions: 2 142 → 2 174 functions, floor 54 % failed at
53.72 %. Answered with `store/responses.test.ts` (17 cases pinning the fields
each state-only answer writes), not by lowering the floor. Result: 54.46 %.

## Not done

Feature slices (the actions share one worker and one `send`), splitting the
mixed `v35`/`v36` test files, unifying the two quantile functions (they differ
on an empty list: 0 vs NaN).
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "labml",
"private": true,
"version": "1.50.0",
"version": "1.51.0",
"license": "MIT",
"type": "module",
"engines": {
Expand Down
38 changes: 38 additions & 0 deletions src/features/data/quality/distribution.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
/**
* V51 — the three helpers drift detection (V17) and the reference profile
* (V40) both need, which each file used to carry a copy of. The copies were
* identical; keeping one means a fix to how a number is read, or how a
* missing cell is counted, reaches both features at once.
*/
import { isMissing, parseNumber } from '@/features/ml/data/infer';
import type { Cell } from '@/features/ml/data/types';

/** Every cell that reads as a number, in order; missing and non-numeric cells skipped. */
export function numbersOf(values: Cell[]): number[] {
const numbers: number[] = [];
for (const value of values) {
if (isMissing(value)) continue;
const parsed = parseNumber((value as string).trim());
if (parsed !== null) numbers.push(parsed);
}
return numbers;
}

/** Share of missing cells; 0 for an empty column. */
export function missingRatio(values: Cell[]): number {
if (values.length === 0) return 0;
let missing = 0;
for (const value of values) if (isMissing(value)) missing += 1;
return missing / values.length;
}

/** Shares over bins whose edges are given (open-ended tails); all zero for no numbers. */
export function sharesOverEdges(edges: number[], numbers: number[]): number[] {
const counts = new Array<number>(edges.length + 1).fill(0);
for (const value of numbers) {
let bin = 0;
while (bin < edges.length && value > edges[bin]) bin += 1;
counts[bin] += 1;
}
return counts.map((count) => (numbers.length > 0 ? count / numbers.length : 0));
}
33 changes: 3 additions & 30 deletions src/features/data/quality/drift.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@
* the reference), new/vanished categories, missing-rate shifts. Pure and
* deterministic; the conventional PSI thresholds 0.1 / 0.25 grade severity.
*/
import { inferColumnType, isMissing, parseNumber } from '@/features/ml/data/infer';
import { inferColumnType, isMissing } from '@/features/ml/data/infer';
import { quantileSorted } from '@/features/data/quality/checks';
import { missingRatio, numbersOf, sharesOverEdges } from '@/features/data/quality/distribution';
import type { Cell, ColumnType } from '@/features/ml/data/types';

const NUMERIC_BINS = 10;
Expand Down Expand Up @@ -62,34 +63,6 @@ export function psi(reference: number[], current: number[]): number {
return total;
}

function numbersOf(values: Cell[]): number[] {
const numbers: number[] = [];
for (const value of values) {
if (isMissing(value)) continue;
const parsed = parseNumber((value as string).trim());
if (parsed !== null) numbers.push(parsed);
}
return numbers;
}

function missingRatio(values: Cell[]): number {
if (values.length === 0) return 0;
let missing = 0;
for (const value of values) if (isMissing(value)) missing += 1;
return missing / values.length;
}

/** Shares over bins whose edges are the reference's quantiles (open-ended tails). */
function numericShares(edges: number[], numbers: number[]): number[] {
const counts = new Array<number>(edges.length + 1).fill(0);
for (const value of numbers) {
let bin = 0;
while (bin < edges.length && value > edges[bin]) bin += 1;
counts[bin] += 1;
}
return counts.map((count) => (numbers.length > 0 ? count / numbers.length : 0));
}

function numericDrift(column: string, ref: Cell[], next: Cell[]): ColumnDrift {
const refNumbers = numbersOf(ref).sort((a, b) => a - b);
const newNumbers = numbersOf(next);
Expand All @@ -101,7 +74,7 @@ function numericDrift(column: string, ref: Cell[], next: Cell[]): ColumnDrift {
const value =
refNumbers.length === 0 || newNumbers.length === 0
? 0
: psi(numericShares(edges, refNumbers), numericShares(edges, newNumbers));
: psi(sharesOverEdges(edges, refNumbers), sharesOverEdges(edges, newNumbers));
const mean = (numbers: number[]) =>
numbers.length > 0 ? numbers.reduce((a, v) => a + v, 0) / numbers.length : 0;
return {
Expand Down
30 changes: 2 additions & 28 deletions src/features/data/quality/reference.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,15 @@
* salary distribution, not anybody's salary. That is what makes it safe to
* commit next to the code it describes, which is the whole point of having it.
*/
import { isMissing, inferColumnType, parseNumber } from '@/features/ml/data/infer';
import { isMissing, inferColumnType } from '@/features/ml/data/infer';
import {
psi,
severityOf,
type ColumnDrift,
type DriftSeverity,
} from '@/features/data/quality/drift';
import type { Cell } from '@/features/ml/data/types';
import { missingRatio, numbersOf, sharesOverEdges } from '@/features/data/quality/distribution';

/** Same bin count as V11, so a profile and a live comparison agree. */
const NUMERIC_BINS = 10;
Expand Down Expand Up @@ -59,23 +60,6 @@ export interface DataProfile {

const OTHER = '__other__';

function numbersOf(values: Cell[]): number[] {
const numbers: number[] = [];
for (const value of values) {
if (isMissing(value)) continue;
const parsed = parseNumber((value as string).trim());
if (parsed !== null) numbers.push(parsed);
}
return numbers;
}

function missingRatio(values: Cell[]): number {
if (values.length === 0) return 0;
let missing = 0;
for (const value of values) if (isMissing(value)) missing += 1;
return missing / values.length;
}

function quantile(sorted: number[], q: number): number {
if (sorted.length === 0) return 0;
const position = (sorted.length - 1) * q;
Expand All @@ -84,16 +68,6 @@ function quantile(sorted: number[], q: number): number {
return low === high ? sorted[low] : sorted[low] + (position - low) * (sorted[high] - sorted[low]);
}

function sharesOverEdges(edges: number[], numbers: number[]): number[] {
const counts = new Array<number>(edges.length + 1).fill(0);
for (const value of numbers) {
let bin = 0;
while (bin < edges.length && value > edges[bin]) bin += 1;
counts[bin] += 1;
}
return counts.map((count) => (numbers.length > 0 ? count / numbers.length : 0));
}

function categoryShares(values: Cell[]): { shares: Record<string, number>; names: string[] } {
const counts = new Map<string, number>();
let total = 0;
Expand Down
Loading
Loading