Plan: why NGBoost still wins two quantile datasets (I072, I073)

[bbstats]

Why NGBoost still beats the quantile model on two datasets, and why the fixes don't ship. Plan files only.

The finding. On visualizing_soil and SGEMM, NGBoost wins entirely on where it puts the centre. Our own interval shape, moved onto NGBoost's centre, beats NGBoost itself on both. The centre comes from our squared-error regressor, fitted without its usual full-data refit.

What was tried, on five datasets (the two losses, pol, and two controls we already win), all on the benchmark's exact splits. Each script reproduced the benchmark's scores exactly first.

centre setting	visualizing_soil	SGEMM	cpu_act (control)	verdict
254 bins instead of 128	+24%	+7%	-2.5%	fails: overfits cpu_act
254 bins as extra candidates	+24%	+7%	-2.5%	fails: validation can't see the overfit
8000 rounds instead of 2000	+5%	0%	0%	can't pass the gate: hits the cap on 1 of 59 datasets
bagged x5	+30%	0%	+0.7%	ceiling only: the default doesn't ensemble

NGBoost's leads are +25.5% and +7.6% on the same scale. So finer bins would match NGBoost, but only at a cost elsewhere that the held-out rows can't detect.

One open question for the maintainer (recorded in QUANTILE_PLAN.md): without an eval_set, the quantile model never trains on its own validation fold, while the point regressor refits on all rows (worth 8.8% RMSE on visualizing_soil). The benchmark can't measure that, since it always passes an eval_set.

Records: CAMPAIGN_PLAN.md I071 (merge), I072, I073, I073b; QUANTILE_PLAN.md Q10, Q11. The next quantile step is Q4, spread-aware categorical encoding.

🤖 Generated with Claude Code