Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 23 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,8 @@ The project follows three non-negotiable principles:
are implemented from scratch in TypeScript, seeded end to end — the same seed always
reproduces the same run. Core metrics plus compatible ridge and k-NN predictions are
checked against committed scikit-learn references generated outside the TypeScript
implementation.
implementation, including the edge where a class is missing from the rows being scored:
macro averages run over the classes present, as scikit-learn computes them.

## Features

Expand All @@ -78,7 +79,7 @@ The project follows three non-negotiable principles:
| No target? | Seeded k-means (k chosen by silhouette) + power-iteration PCA projection, groups described in plain language; date column? **Holt-Winters forecasting** validated by rolling-origin backtest |
| MLOps loop | Score a **new batch** with honest test-vs-batch metrics; **compare two runs** side by side with cross-run uncertainty verdicts, or **up to six at once** read against the oldest of the selection; **export a model as JSON and re-import it later** — the exact predictor is rebuilt (byte-identical predictions) and scores any CSV without retraining |
| Speed | Heavy families train on **helper cores** (announced on the leaderboard, split by measured cost, never silent), and a model crosses back as JSON so it is rebuilt through the same path an imported model uses. Measured on a 60 000-row run: **74 s → 9.5 s**, with every leaderboard number identical |
| Persistence | Local run history with attached artifacts, opted-in dataset storage (compressed, explicit 50 MB budget), self-contained HTML reports, data-free share links |
| Persistence | Local run history with attached artifacts, opted-in dataset storage (compressed, explicit 50 MB budget), self-contained HTML reports, data-free share links (a damaged link is refused, never half-rendered) |

**What the run gives you, past the ranking:**

Expand Down Expand Up @@ -217,7 +218,12 @@ Wikimedia Commons), the portrait is NASA, public domain._
## Engineering notes

- **Everything off the main thread.** Parsing, cleaning, training, scoring and analysis
run in dedicated Web Workers behind typed message protocols.
run in dedicated Web Workers behind typed message protocols. Every answer a worker can
send has exactly one handler, checked by the compiler, and an answer that arrives for a
state already thrown away is dropped: the target and the features are locked while a
computation that depends on them runs, so a result can never be filed under a question
it did not answer. A worker that crashes is replaced rather than left holding the page,
and the DuckDB engine is closed when its panel goes away.
- **From-scratch algorithms**, unit-tested against known results: gradient boosting
(quantile bins, second-order gains, Newton leaves), MLP, k-means++, PCA, Holt-Winters,
isolation forest, PSI, Shapley values, bootstrap intervals, PR/ROC/calibration curves,
Expand All @@ -241,16 +247,21 @@ Wikimedia Commons), the portrait is NASA, public domain._
a doc page carries its whole article, in English until the app mounts); Lighthouse
mobile ≈ 0.99 on `/ml` under real throttling. Heavy dependencies (Dexie, SheetJS, ONNX
Runtime) load lazily.
- **Quality bar.** 923 unit tests and 123 Playwright end-to-end tests across five
- **Quality bar.** 924 unit tests and 123 Playwright end-to-end tests across five
projects — desktop, a phone viewport in English and in French, dark mode, and
Cloudflare Pages' own routing emulated by `wrangler pages dev` (a real 404, the security
headers as served) — covering offline PWA, a fake webcam, a horizontal-overflow guard on
every route, and axe-core WCAG A/AA checks on every page including the twenty-four
documentation pages. Plus strict TypeScript, ESLint, Prettier, and Lighthouse budgets —
all enforced in CI.
documentation pages. Plus strict TypeScript, ESLint, Prettier, Lighthouse budgets and
a **coverage floor** set from the measured baseline (statements, branches, functions,
lines), all enforced in CI.
- **Supply-chain checks.** Dependabot reviews npm and GitHub Actions weekly, CodeQL
analyzes JavaScript and TypeScript on pull requests and `main`, and CI blocks critical
advisories anywhere plus high advisories in dependencies shipped to production.
advisories anywhere plus high advisories in dependencies shipped to production. Every
GitHub Action is pinned to a full commit SHA, no checkout leaves the git token on the
runner, no pull-request value is expanded inside a script, and **zizmor** audits the
workflows on every pull request. A test holds each of those rules per workflow file.
The local language model is fetched at a pinned Hugging Face revision, not `main`.
- **One dependency does not come from npm.** SheetJS left the registry, and the copy
still published there (`xlsx@0.18.5`) carries two unfixable high advisories. The
dependency points at the project's official tarball instead, which fixes both;
Expand All @@ -277,6 +288,7 @@ npm run dev # start the dev server
| Script | Purpose |
| --------------------------------------- | -------------------------------------------------------------- |
| `npm run test` | Unit tests (Vitest) |
| `npm run test:coverage` | Unit tests with the coverage floor CI enforces |
| `npm run e2e` | End-to-end tests (Playwright) |
| `npm run typecheck` | TypeScript, strict mode |
| `npm run lint` / `npm run format:check` | ESLint / Prettier |
Expand Down Expand Up @@ -330,7 +342,10 @@ CI builds, tests and deploys on every push: pull requests get a Cloudflare Pages
## Roadmap

Development proceeds in planned "caps" of feature waves; six caps have shipped (MVP
through the lab meeting the real world — real photos, real text, real file sizes). The full plan, delivery log and design decisions live in
through the lab meeting the real world: real photos, real text, real file sizes). Cap 7
makes what already ships hold up under pressure: worker lifecycle, damaged share links,
CI hardening, metric edge cases and a store refactor are delivered; major dependency
upgrades come next. The full plan, delivery log and design decisions live in
[PLAN.md](PLAN.md); [CHANGELOG.md](CHANGELOG.md) is extracted from it — one entry per
wave, newest first — by `npm run changelog`, and a test fails when the two disagree.

Expand Down
9 changes: 9 additions & 0 deletions src/content/docs/en/limits.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,9 @@ They are not equivalent, and mixing them would be dishonest:
| Catching async errors and shipping crash reports | React boundaries cannot see callbacks or event handlers — claiming otherwise would be false; and nothing leaves the browser, telemetry included |
| A daily Dependabot and `npm audit fix --force` | A daily cadence drowns the signal; `--force` moves majors without reading the break. The remaining advisories sit in build tooling, and we say so instead of hiding it |
| Picking multiclass thresholds automatically | The lab shows its decisions instead of making them for you; and abstention is neither uncertainty calibration nor a conformal guarantee |
| A run id in every worker message | The worker answers in the order it is written to, so the training status is enough to recognise a stale answer. The id becomes necessary the day requests can overlap |
| A « rare class » warning in the interface | A one-row class stays in training, and a target where every class has one row is refused by name; a banner would flag a problem the split already handles |
| Splitting the ML Lab store into independent slices | Every action shares one worker: slicing would have moved code without separating anything |

## Dropped after measurement

Expand All @@ -65,6 +68,12 @@ person — is covered for 0 MB by an honest refusal. CLIP would answer a
arbitrate: the current detector's two misses are inert — both images are
already named correctly.

**A field-by-field validator for share links** (V48). Planned. Measured on
three real links with every field deleted, nulled or retyped: the links the
decoder accepted crashed the page on 99 distinct paths. A validator that size
would drift with every new analysis; a guard at render time covers them all,
and a test replays the measurement on every run.

## Predictions measurement refuted

This is the most useful category, and the most uncomfortable.
Expand Down
21 changes: 21 additions & 0 deletions src/content/docs/en/reference-ml.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,11 @@ triggers task detection (binary, multi-class, regression) and **leak
detection**: a column that almost perfectly mirrors the target is excluded
automatically and named.

While a run trains, and while anything computed from it is running
(hyperparameter search, learning curve, robust ranking, batch scoring), the
target and the columns are **locked**. Changing them at that moment would file
results under a question they do not answer; wait for the end or cancel.

## Training and the leaderboard

Eight hand-written families, plus an ensemble, plus a naive baseline. Split
Expand All @@ -43,6 +48,17 @@ that is the price of selection, and hiding it would make the score flattering.
The baseline always answers the majority class (or the mean for regression). A
model that does not beat it has learned nothing, whatever its absolute score.

The split is stratified by class. A class with **a single row** stays on the
training side: sending it to test would leave the models with no example of it
at all. If every class has a single row, nothing is left to test on, and
training refuses with `too-few-rows` rather than publishing metrics computed
over an empty set.

Precision, recall and F1 are **macro** averages, computed as scikit-learn does:
over the classes present in the true labels or in the predictions. A class
absent from both does not count as a zero; a class that is predicted but never
true does, because those predictions are all wrong.

## How solid the numbers are

The test set is bootstrap-resampled 1,000 times and the metric recomputed each
Expand Down Expand Up @@ -123,6 +139,11 @@ rule travels with the export, and the scored CSV then carries two extra columns
— see [the formats page](/docs/formats). The five named import refusals are on
the [refusals page](/docs/refus).

A share link carries the run's metrics and charts in the URL fragment, never
the data. A cut or damaged link never yields a half-read page: if it does not
decode, or if part of the run it carries does not have the shape LabML writes,
the page refuses to show it and says why.

## Comparing runs

Two runs side by side (`/ml/compare/:a/:b`) or up to six
Expand Down
16 changes: 8 additions & 8 deletions src/content/docs/en/refusals.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,14 +30,14 @@ Two audiences, and the distinction matters:

## ML Lab — training

| Refusal | What triggers it | What to do |
| ------------------- | --------------------------------------------------------------- | ----------------------------------------------------------- |
| `no-features` | Every column was excluded, or none is usable | Re-include at least one column in the columns panel |
| `target-not-found` | The target column is no longer in the file | Pick a target again |
| `task-undetectable` | The target is neither continuous numeric nor usable categorical | Choose another column, or force its type in the Data Studio |
| `too-few-rows` | Grouping without a target needs more rows than there are | Load a larger file |
| `too-few-points` | The time series is too short for an honest forecast | Extend the period, or aggregate less finely |
| `missing-columns` | The file to score lacks columns the model expects | Add the columns named in the message |
| Refusal | What triggers it | What to do |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| `no-features` | Every column was excluded, or none is usable | Re-include at least one column in the columns panel |
| `target-not-found` | The target column is no longer in the file | Pick a target again |
| `task-undetectable` | The target is neither continuous numeric nor usable categorical | Choose another column, or force its type in the Data Studio |
| `too-few-rows` | Grouping without a target needs more rows than there are, or every class of the target has a single row and nothing is left to test on | Load a larger file, or pick a target whose classes repeat |
| `too-few-points` | The time series is too short for an honest forecast | Extend the period, or aggregate less finely |
| `missing-columns` | The file to score lacks columns the model expects | Add the columns named in the message |

### The announced splits

Expand Down
Loading
Loading