Conversation
Seven Grinsztajn regression sets, 5-fold OOF, estimator defaults: skill vs the fold-marginal forecast, 20-cell occupancy KL, interval coverage, and predict_thresh reliability (ECE + Brier skill). Verdict recorded in QUANTILE_PLAN.md: calibrated throughout, tail cells run 1-5 points heavy raw, conformalize=True closes them at ~1 point of CRPS skill. Issue #106 follow-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…losses docs/whitepaper.md (~2,600 words, layered for practitioners then engineers) plus a committed PDF render and the Edge-headless build script. New benchmarks/tabarena/LEADERBOARD_SNAPSHOT.md records the 2026-09-01 official leaderboard read (default Elo 1315, #2 GBDT on defaults behind CatBoost 1379), finally giving the README's ranking claim a dated, sourced citation, and the stale Elo-1278 line in PROJECT_STATUS.md is retired in the same change. Every number traces to a named file or the dated leaderboard fetch; the stale-flagged internal tables and the retracted quantile speed claim are excluded by construction. The quickstart snippet is executed as printed (one fix fell out: the quantile fit needs cat_features too), and the site builds --strict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same facts, different narrator. Every em dash is gone, the 'X, not Y' flourishes are gone, and passages that read like the maintainer talking shop (the defensible sentence, section-number shorthand, goldens, first read) now speak to the reader instead. PDF re-rendered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The TabArena site defaults to the imputation_yes pool (79 entrants, imputed scores for models with missing task results); the first snapshot quoted imputation_no (69 entrants), so our printed Elos ran ~15-20 higher than what a reader sees on tabarena.ai. Whitepaper, snapshot, and PROJECT_STATUS now quote the site-default slice (ChimeraBoost default 1297 vs CatBoost 1357, tuned+ens 1360), with the imputation_no read kept in the snapshot as a reference. Ordering and every claim are unchanged in both slices. PDF re-rendered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reread the XGBoost and CatBoost abstracts for the voice, then rewrote: first-person-plural authorial voice, contributions enumerated in the introduction, measurements reported in past tense, second person gone, and a proper author-year reference list replacing the informal sources paragraph. Structure now runs abstract, introduction, system overview, training pipeline, quantile head, evaluation, limitations, availability. Facts and numbers unchanged from the imputation_yes read. PDF re-rendered; site builds --strict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TabArena scores classification and point regression only, so the abstract's claim that the multi-quantile head accounts for part of the leaderboard standing was wrong. The TabArena result now rests on the replay refit alone, and the quantile head is introduced as a separate capability beyond what TabArena measures. PDF re-rendered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Stamp the paper 0.33.0: predict_thresh is unreleased at 0.32.0, so the paper now names the release that will actually contain what it documents (0.33.0 should ship with or before this PR's merge). - Anchor the three orphaned references: Shi, Li and Li at linear leaves, Zhang et al. at cross features, and a new sentence on minimum-variance row subsampling for Ibragimov and Gusev. - SHAP claim tightened: exact over its background set (200-row default cap), replacing the overstated "no sampling approximation". - Public-suite ranks disclosed as facet-balanced, with a pointer to the unweighted numbers that favor CatBoost slightly. - TabArena entry noted as running 0.30.0, with the bit-identity argument for why the accuracy figures carry over. - OOF calibration range quoted exactly (0.029 to 0.051), conformalization priced at about one point of CRPS skill, GBDT-MO dated 2021 (TNNLS vol. 32, issue 7). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nathan's call: a user cares what a setting costs relative to what pip install gives them, not relative to LightGBM. make_public_pareto.py now normalizes per-dataset fit times by the ChimeraBoost default instead of the fastest arm, and the chart, quality-ladder tables (recipes, whitepaper), the benchmarks.md caption, and the whitepaper's public-suite table all follow: default 1x, ensemble 2.6x on the ladder / 2.4x on the public suite, CatBoost 6.5x, LightGBM 0.1x. Two loose ends closed on the way: the docs caption's 1.90/1.88 read traced to run 20260801-174024 (it was flagged untraceable in review), so PUBLIC_PLAN.md now records that run and this axis change; and the public-pareto footer no longer claims "weighted" under --no-weights. The whitepaper's public table switches to that same run so the table, chart, and caption agree. Internal decision tooling (make_pareto.py, summarize, campaign records) keeps the vs-fastest convention for comparability with recorded reference points. Chart regenerated, PDF re-rendered, chart tests green, site builds --strict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nathan's call: not a true limitation and no real information conveyed,
so the bullet goes, along with the dangling forward reference in the
design section. Also unwound the two label-style nominalizations it
exemplified ("the oblivious constraint", "the pure-Python design
constraint") into plain sentences. PDF re-rendered.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The GPU bullet said only that the absence was intentional, which conveys nothing. It is now merged into the training-speed limitation with the substance: the internal scaling study measured the fit-time gap to LightGBM widening from about 5x at 50k rows to roughly 10x at 500k (numba histogram kernels vs hand-tuned C++ SIMD), GPU training is the escape hatch the compiled libraries offer in exactly that regime, and the paper's fit-time comparisons should not be extrapolated to very large data. Predict throughput noted as parity with LightGBM at 2M rows. Scaling scripts added to the sources paragraph. PDF re-rendered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The two mechanisms the library leans on hardest were each one paragraph; they are now sections 4 and 5, with the later sections renumbered. Structure-replay refit (section 4) gets the full story from attribution.md: the early-stopping data tax, why a from-scratch retrain overpays (tree growing is 83-85% of a fit, an oblivious structure is d pairs), the borrowed-halves honesty (XGBoost refresh + AutoGluon refit_full, the default-on composition is ours), the binner-reuse leakage post-mortem, and the full result set including high-card and the four-suite accuracy wash. Selection during the fit (section 5) documents the racing mechanism: shared 100-round budget, best-validation-loss pick, winner-only early stopping, the two races (linear leaves, cross features), decisions exposed as fitted attributes, composition with replay, and the measured evidence that the budget is load-bearing (a 25-round race was rejected because the full-data refit amplifies a bad pick, and a different pick rule did not rescue it). Three references added and cited: Maron and Moore 1993 (Hoeffding races), Jamieson and Talwalkar 2016, Erickson et al. 2020 (AutoGluon). Abstract mentions the racing. PDF re-rendered; now 10 pages. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prompted by "the right answer is dataset-dependent", which no data scientist would write. Swept the paper for the family: "the right answer" is now "the optimal configuration depends on the dataset", "quietly pays for" / "the remedy overpays" / "load-bearing" / "earns its width" / "buys that rung" / "bites" / "the user's burden" / "the library's case rests" / "inspectable rather than hidden" / "deliberately simple" / "composes with" / "worth recording" all rewritten as the plain thing they meant. No numbers changed. PDF re-rendered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first pass swapped metaphors for abstractions; this one swaps the abstractions for the specific fact. "Has a cost that is easy to overlook" now states the withheld rows directly, "does more work than the problem requires" became "repeats work the first fit already did", "the size of the budget matters, and we verified this" became "we empirically verified that shortening the budget degrades the selection", "matters most in exactly this regime" now names the large-dataset GPU story explicitly, and a full fresh reread caught the rest of the family: "carries a cost for research", "the single largest cost in the library", "which is where the accuracy was being lost", "reclaiming what each member gave up", "a different resource class", "a third of CatBoost's cost". PDF re-rendered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
docs/whitepaper.md(~2,600 words) plus a committed PDF render and its build script (docs/build_whitepaper_pdf.py: markdown -> HTML -> Edge headless, no new dependencies). Layered for two readers: benefits and evidence up front for practitioners choosing a GBDT, design and mechanisms in the back for engineers. Linked from the README and the docs nav.The claims are sourced
benchmarks/tabarena/LEADERBOARD_SNAPSHOT.md: a dated (2026-09-01) read of the official leaderboard data file. Default Elo 1315 — Rename iterations -> n_estimators; accept cat_features by column name #2 GBDT on defaults behind CatBoost (1379), 101 Elo clear of XGBoost (1214); tuned+ensembled 1383 edges CatBoost's default. This gives the README's "Rename iterations -> n_estimators; accept cat_features by column name #2 On TabArena GBDT Elo (Defaults)" banner its first dated, sourced citation, and the stale Elo-1278 (2026-07-23) line in PROJECT_STATUS.md is retired in the same change.Also aboard: the OOF calibration study commit
68101e6(out-of-fold calibration study of the quantile head) was committed locally after PR #107 merged, so it never reached main — the same silent-orphan pattern as PR #56. This branch carries it to main.Verified
cat_featurestoo).mkdocs build --strictpasses; PDF checked programmatically (8 pages, chart embedded, no encoding damage).🤖 Generated with Claude Code