Before you swap a tabular model, see exactly what changes.
Quibble compares prediction methods on the same examples, showing which answers change, how strong the evidence is, and what each method costs to run.
In plain English | Technical overview | See the workflow | Quick start | Your data | Documentation
Imagine two programs predicting whether a shop will run out of a product. Both get 80 out of 100 answers right, but they might make mistakes on different products. The same score does not tell you whether switching programs is a good idea.
Quibble gives both programs the same test. It keeps the answers hidden while they make predictions, then checks which program was right when they disagreed. It also measures how long each takes and how much memory it uses.
You get a side-by-side report before replacing one prediction tool with another. Sometimes the honest answer is that there is not enough evidence to decide.
For explanations of each capability, with everyday examples and explicit limits, read What Quibble does.
Quibble runs paired evaluations of tabular classifiers and regressors using shared holdout or cross-validation splits. Adapters cover tree-based learning, AutoML, and pretrained tabular predictors, subject to their dependencies and model access. Saved out-of-sample predictions support quality metrics, uncertainty intervals, row-level disagreements, calibration, exploratory slices, latency, memory, and explicit errors or refusals.
Configurable replacement checks return PASS, FAIL, or INCONCLUSIVE with CLI exit codes for CI. Those verdicts depend on the data, split design, settings, and limits; they are not deployment certification. Use the local web workbench or CLI. An opt-in bounded tree-tuning tier keeps its search inside training folds. Quibble is not an unrestricted AutoML service or model-serving platform.
Chat models can also analyze tables or use tools to run predictors. Quibble does not yet include a chat-LLM predictor; see the scope and proposed evaluation.
- Choose existing data or design a synthetic dataset, then review its sample.
- Select methods, device policy and splits, then explicitly start the reviewed request.
- Open the saved report, including uncertainty, disagreements, costs, and failures.
This excerpt from a real local full comparison using a holdout split shows the saved report
for HistGB and LightGBM on the fictional grocery_sample dataset using the current
neutral theme. No provider request or private data is involved. It is an
illustration, not evidence about real grocery operations.
See the visual walkthrough for all three screens, or the report guide for the evidence and its limits.
| Look beyond the score | What you get |
|---|---|
| Quality and uncertainty | Test-set bootstrap and approximate corrected split intervals (5+ splits), calibration, and exploratory slices |
| Prediction-level differences | Disputed rows, who was right, and shared row traits |
| Real execution costs | Fit time, prediction latency, memory, errors, and refusals |
| Replacement decisions | PASS / FAIL / INCONCLUSIVE checks with configurable limits and CI exit codes; quibble-ml gate asks whether a model beats a simple baseline at all |
| Data checks | Flags for near-perfect single features, ID-like columns, duplicate rows, and time-like columns on a random split; group and temporal splits for local data |
| Reproducible evidence | Saved predictions, source fingerprints for local data, dated reports, and explicit device labels |
It is a comparison and verification tool, not a model server or deployment platform. See the project rationale for the scope and motivation.
Use Python 3.13 for this setup; core Quibble supports Python 3.10+.
Run from Quibble's root. If another virtual environment is active, run deactivate first.
Registry dependencies use PyPI by default. CI and Docker use the public lockfile. For restricted networks, configure your approved package mirror locally and use the alternate-index installation workflow. Do not commit machine-specific index settings or a mirror-specific lockfile.
If you only have a CPU, or do not want to use your GPU, use this setup. It installs the core tree/baseline methods, browser, Parquet support, and tests. Torch and optional model libraries are not required.
# Requires uv. Creates or syncs this checkout's .venv from uv.lock.
uv sync --locked --python 3.13 --extra web --extra data
& .\.venv\Scripts\Activate.ps1
$env:QUIBBLE_DEVICE = "cpu"
python -c "import sys; print(sys.executable)"
uv pip check
python -m quibble_ml doctorLinux and macOS:
uv sync --locked --python 3.13 --extra web --extra data
source .venv/bin/activate
export QUIBBLE_DEVICE=cpu
uv pip check
python -m quibble_ml doctorKeep QUIBBLE_DEVICE=cpu in your .env for future sessions; the commands above
set it only for the current shell. In the workbench, choose CPU for the run.
Doctor showing optional packages as None, gpu: None, and
torch is not installed is expected for this base setup. Its backend verification
labels describe historical project checks, not installed libraries or tests on
your machine.
After the setup checks, start the browser workbench:
python -m quibble_ml web --openThis opens the local site at http://127.0.0.1:8765. Choose Open workbench,
then CPU when configuring a run. Keep the terminal running; press Ctrl+C
to stop the server.
For TabPFN and TabICL on CPU, add their extras without windows-cuda:
uv sync --locked --python 3.13 --extra web --extra data --extra tabpfn --extra tabicl
$env:QUIBBLE_DEVICE = "cpu"
uv pip check
python -m quibble_ml doctorThese models require Torch, additional RAM, and access to their weights under the applicable terms. CPU execution does not guarantee CPU-only Torch wheels. Select only the models you need; Mitra-v2 is CUDA-only.
Use this alternative to the CPU setup if you have a compatible NVIDIA GPU and driver. From the repository root, install CUDA wheels and the models you want. This example selects TabPFN and TabICL:
uv sync --locked --python 3.13 --extra web --extra data --extra tabpfn --extra tabicl --extra windows-cuda
& .\.venv\Scripts\Activate.ps1
uv pip check
python -m quibble_ml doctorDoctor must report CUDA OK with your GPU name; a +cu128 Torch version alone
does not establish usable CUDA. Only after that check, start the GPU workbench:
$env:QUIBBLE_DEVICE = "cuda"
python -m quibble_ml web --openChoose Open workbench, then CUDA in run settings. Keep the terminal open;
Ctrl+C stops the server. Set QUIBBLE_DEVICE=cuda in .env for future sessions
and remove any lingering shell override set to cpu. Explicit CUDA refuses
when unavailable; it does not silently fall back to CPU. Incompatible methods
are excluded, and model-weight access/terms are still separate from installation.
Keep every wanted extra, including windows-cuda, on future sync commands.
Most registry packages use PyPI; this CUDA extra uses the official
PyTorch index and needs separate network access. It does not configure AMD/Intel
GPUs, Linux, or macOS. See GPU and Docker for WSL2 and
container guidance and their unverified runtime limits.
Environment requirements, CUDA wheels, and maintenance
- The interpreter must end in
quibble\.venv\Scripts\python.exe. pyproject.tomlis the only dependency manifest;uv.lockrecords resolved versions and hashes. The setup includes the browser, Parquet, and development tests, but no optional foundation-model libraries or weight downloads.- Add selected extras with the same sync command, for example
--extra tabfm,--extra kumo,--extra tabpfn,--extra tabicl,--extra mitra-v2, or--extra autogluon. TabFM/Kumo need Python 3.11+; AutoGluon needs Python below 3.14. The experimental Mitra-v2 adapter requires Python 3.11-3.13 and CUDA on Windows/Linux; its classification and regression recipes differ. - Sync is exact: keep all wanted
--extraflags on subsequent commands or uv removes their packages. Use--dry-runfirst when migrating an existing environment. - Native Windows CUDA wheels require the explicit
--extra windows-cudaselection: Torch 2.11.0 and torchvision 0.26.0, CUDA 12.8, from the official PyTorch index. Other registry packages use PyPI; a compatible NVIDIA driver is still required. The optional Mitra-v2 source still uses Hugging Face. - Use
uv add,uv remove, anduv lockto maintain dependencies, then checkuv lock --check. Never sync an environment while a sweep is running. - Optional extras can need several GB before pretrained weights are downloaded. Dependency resolution or CUDA detection is not a successful model fit; see validation status.
python -m quibble_ml web --openBeside Scenario description, use local suggestions or enter rough notes such as retail sales prediction. The link becomes Refine my scenario for your own text; refinement uses the configured provider only after consent and can incur charges. Review the refined text before requesting a design. See the workbench guide.
Design a synthetic prediction task, review its assumptions and sample rows, save the
dataset, and run smoke/full comparisons at http://127.0.0.1:8765. Four ready-made
datasets are available. The default address opens About; choose Open workbench
to enter the experiment workflow at /workbench. The four bundled
datasets cover grocery and retail clothing, each with binary and regression targets.
Select one to reuse its saved rows without a cloud connection or regeneration.
Home and the sidebar provide existing data, synthetic design and local uploads.
Follow Choose data -> Methods and settings -> Start and monitor.
The run draft uses one reviewed dataset; Experiment history opens saved reports
independently, without losing the draft or interrupting execution monitoring.
The single existing-data selector filters classification (binary or multiclass) and
regression, grouping built-in, bundled, and saved tables without implying different
readiness levels. New synthetic designs support binary classification or regression.
After preview and review, Save and continue saves and opens configuration.
For a variant, the adjacent Discard changes and continue uses the unchanged
original instead. Neither starts a run automatically. Preview generation is disabled
for an unchanged saved recipe or a completed preview until the recipe changes.
The product header makes Quibble prominent with a smaller explanatory subtitle. All pages share Workbench / Blog / About / GitHub / Docs navigation. Blog, About, and Docs open separately from the workbench to preserve drafts. Blog and Docs are readable locally; GitHub links to the source repository. Version and license details belong in About's project-status section, not the header. Arbitrary scenario drafting uses a user-configured OpenAI-compatible provider, including OpenAI, Azure, or a compatible local service, with explicit consent and API-key, Microsoft Entra ID, or local no-auth configuration. See the workbench guide for setup, limits, and provenance. Synthetic results describe the chosen simulation, not real-world business performance.
Why generate fresh data? Public benchmarks may have appeared in a pretrained predictor's training data. New sampled rows reduce reuse of those benchmark examples and let you control noise, missingness, and irrelevant inputs. They do not prove the underlying task is unseen or eliminate pretraining-exposure concerns. Repeat across generation seeds and check findings on representative real data.
The browser is an additional interface to the existing evaluation workflow, not a replacement for CLI commands. Use it for interactive review; keep the commands below for automation and more extensive configurations.
python -m quibble_ml demoOpen results/report/report.md for the interpreted comparison, or use the report under
your configured output root.
Note
The default demo is CPU-only and works offline after installation. Use benchmark
for device-aware CPU/GPU comparisons.
What the demo runs, expected sanity checks, and optional foundation models
- Runs a sweep, analysis, swap checks, and report on built-in
breast_cancer,wine, anddiabetes_reg; typically about a minute for the default models. - Uses
histgb,lightgbm, and deliberately weakweak, with model seed 0 and CV3x2. The HistGB-to-Weak sanity check is expected to FAIL. - Always forces CPU, overriding
.env, even with foundation models selected. It has no--deviceoption. Installing model packages does not automatically select them. - Default models need no pretrained downloads and work offline after installation.
- Open
results/report/report.md, or the report under your configured output root.
To include foundation models in the CPU demo:
python -m quibble_ml demo --backends histgb,lightgbm,tabfm,kumo_tabular --jobs 1This still uses all three demo datasets, but may download weights and run much longer.
Swaps use HistGB as incumbent when selected, otherwise the first selected backend.
For GPU measurements, use benchmark or run instead.
Create a named folder with a labeled table and a small declaration:
data/
payments/
data.csv Or data.parquet
dataset.json Target, task, and optional column rules
python -m quibble_ml datasets --verify --names payments
python -m quibble_ml benchmark --datasets payments --backends histgb,lightgbm --device cpu --jobs 1 --smokeFollow Testing your own data for the manifest, accepted formats, a runnable sample, full comparisons, output locations, interpretation, reruns, and privacy. Inputs stay separate from download caches; changed local data cannot silently reuse old comparisons.
benchmark honors QUIBBLE_DEVICE or an explicit --device cpu|cuda|auto|per-model.
For per-model, pass --model-devices '{"histgb":"cpu","catboost":"cuda"}'
with an assignment for exactly every selected backend. The workbench offers the same
policy with reviewed device choices alongside selected models. Assignments never fall back;
mixed-device timings are not an equal-device speed comparison.
It defaults to wine, CV 3x2, and model seed 0.
| Preset | Backends |
|---|---|
trees |
HistGB, LightGBM, CatBoost |
foundations (default) |
HistGB, LightGBM, Google TabFM, NVIDIA KumoTabular |
all |
Every non-internal backend, including TabPFN and AutoGluon |
# Bounded setup check; may download weights.
python -m quibble_ml benchmark --preset foundations --smoke
# Full comparison; archive and replace selected cached measurements.
python -m quibble_ml benchmark --preset foundations --forceCustom selections, saved models, participation, and exit codes
# Optional fitted models and AutoGluon's validation leaderboard.
# Use a separate root when changing from the CV scheme above to holdout.
python -m quibble_ml benchmark --backends histgb,autogluon --cv holdout --keep-models --out results_holdout--backendsoverrides the preset. Other controls include--datasets,--cv,--model-seeds,--jobs,--threads, and--out. Use--jobs 1for measured comparisons.- The participation table distinguishes selected, cached, incompatible, failed, refused, and missing cells, and shows fits used in the comparison. Preset comparisons exclude incompatible cached backends until rerun.
- Exit codes: 0 when requested fits completed; 1 for failed/refused/missing/incompatible cells; 2 for configuration/usage errors. A swap verdict of FAIL is not a fit failure.
Important
Smoke reports interpret bounded setup results, not full-data model quality. They do not issue deployment recommendations or swap approval.
Sampling limits, output isolation, and download requirements
- One holdout split, model seed 0, at most 128 training and 32 test rows.
- Identical sampled rows across backends; writes to
results/smoke/<unique-id>/. - Reports say
SMOKE TEST ONLYand interpret setup outcomes, scores, resources, and disagreements, without deployment recommendations or swap verdicts. - Checks setup, not full-data quality or capacity. Weight downloads can still be large or require authentication.
| Policy | Behavior |
|---|---|
cpu |
Pins every model to CPU |
cuda |
Requires CUDA; runs only compatible models and visibly excludes the others |
auto (default) |
Resolves once per sweep to CUDA if available, otherwise CPU |
CatBoost, TabPFN, TabFM, and KumoTabular support CUDA. When the resolved device is CUDA,
HistGB, LightGBM, Weak, and AutoGluon are excluded with reasons rather than run on CPU.
HistGB/Weak have no CUDA implementation; this LightGBM adapter uses CPU, and AutoGluon
cannot guarantee GPU-only components. An all-excluded selection is rejected.
CPU comparisons retain all selected methods. CUDA methods run serially; CPU comparisons
can use --jobs. Data preparation, metrics, and reports still use CPU.
See what Full does and what the results establish for the split, fit, held-out prediction, and analysis sequence.
Explicit CUDA example, resource failures, and mixed-device reporting
python -m quibble_ml doctor
python -m quibble_ml benchmark --backends catboost,tabfm,kumo_tabular --datasets wine --cv 3x2 --model-seeds 0 --device cuda --jobs 1 --out results_cuda --force- Let a current sweep finish before another writes to the same output root.
- CPU requests stay on CPU. CUDA is strict: no silent CPU fallback after memory failures.
- No setting guarantees sufficient RAM/VRAM for every dataset. Model-level memory errors, timeouts, worker crashes, and cleanup errors are recorded while other cells continue. Storage/archive or configuration failures can still stop the command.
- Reports show requested, resolved, and actual devices and exclusions, flag historical mixed CPU/GPU measurements, and preserve original device labels on cached runs.
Ensemble defaults, retained models, and AutoGluon diagnostics
- TabFM defaults to 4 estimators; Kumo defaults to
smallwith 4 estimators. SetQUIBBLE_TABFM_N_ESTIMATORS,QUIBBLE_KUMO_SIZE, andQUIBBLE_KUMO_N_ESTIMATORS. - Both support classification/regression and use full training splits outside smoke mode. Repeated single-row foundation-model inference can be slow.
--keep-modelsorQUIBBLE_KEEP_MODELS=1retains HistGB, Weak, LightGBM, CatBoost, and AutoGluon fits underresults/artifacts/; off by default.- TabPFN, TabFM, and Kumo fitted estimators are not serialized. Retention requests are
reported as unsupported; downloaded checkpoints still remain in
.cache. - Each retained fit has a unique directory, linked rather than copied into snapshots.
- AutoGluon validation leaderboards are saved as analysis CSVs even without retention; held-out test labels are not used to select its best model.
See backends, configuration, and model reload instructions.
Runtime downloads live under the project's .cache by default, rather than being
scattered across user-level library caches.
Cache layout, Google/NVIDIA checkpoint folders, and overrides
Quibble configures caches before importing model libraries. Relative cache paths resolve from the project root, independent of the shell's working directory:
.cache/
huggingface/ Shared weights/assets (Google TabFM, NVIDIA KumoTabular)
hub/
models--google--tabfm-1.0.0-pytorch/
models--nvidia--Kumo-Tabular/
tabpfn/ Prior Labs TabPFN checkpoints
torch/ Torch Hub weights, extensions, and compiler cache
triton/ Triton compiler cache, when used
sklearn/ scikit-learn data
openml/ OpenML datasets
xdg/ Libraries using the XDG cache convention
-
Set
QUIBBLE_CACHE_DIRto relocate the root. -
Leave individual overrides blank to follow that root, or set explicit paths:
QUIBBLE_CACHE_DIR=.cache QUIBBLE_SKLEARN_CACHE_DIR=.cache\sklearn QUIBBLE_HF_CACHE_DIR=.cache\huggingface QUIBBLE_TRANSFORMERS_CACHE_DIR=.cache\huggingface\hub
-
No other project's environment or cache is required.
-
QUIBBLE_DATA_HOMEoverrides OpenML's location; blank followsQUIBBLE_CACHE_DIR/openml. -
Quibble overrides inherited library cache variables so an old shell does not scatter downloads. Transformers shares the Hugging Face hub by default to avoid duplicate weights.
-
Existing downloads are not moved automatically. Missing weights download when selected models need them; provider folders appear as files are downloaded. Later runs reuse cached files where supported, although remote metadata checks may still occur.
-
Pip/uv installation caches are separate and unchanged. Runtime caches are not copied into result snapshots.
python -m quibble_ml doctorprints the effective cache root.
See datasets and data locations for actual source URLs, download triggers, inventory, installed files, and inspection commands. Built-in data ships with scikit-learn; OpenML tables and pretrained weights are separate downloads.
| Open | What it contains |
|---|---|
results/report/report.md |
Latest full comparison, interpretation, figures, and swap checks |
results/smoke/<unique-id>/report/report.md |
Isolated setup-check interpretation |
results/report/history.md |
Dated snapshots of earlier results and reports |
Demo, benchmark, run, and report share QUIBBLE_OUT_DIR, normally results. Separate roots
are optional for intentionally isolated configurations.
Tip
Saved runs, including errors, are reused. Use --force to retry selected cells.
Use a new output root when changing local data or incompatible split assignments.
Output tree, cache reuse, source protection, and snapshot behavior
results/
<dataset>/ Current records, predictions, and analysis
invocation.json Latest sweep selection and execution/cache status
artifacts/ Optional retained fits, stored once per fit
smoke/<unique-id>/ Isolated setup checks
report/
report.md Stable latest report
history.md Dated-history index
legacy/ Outputs preserved during the earlier migration
runs/YYYY-MM-DD/<id>/
snapshot.json Timestamp, reason, datasets, and source run IDs
invocation.json Sweep metadata, when available
<dataset>/ Preserved records, arrays, and analysis
report/ Report and figures, for report snapshots
- Saved runs, including errors, are skipped unless
--forceis supplied.0 to runmeans cached cells, not automatic model discovery. Changing environments does not invalidate them. --forcereruns all selected cells after archiving existing outputs. Changed settings may otherwise warn and skip. All models being compared must use compatible splits/seeds.- Different test-row arrays are rejected rather than overwritten. If changing CV or split
assignment changes those rows, use a separate
--out;--forcealone does not bypass this guard. - Training saves before/after snapshots; report regeneration also archives a version. New records contain timestamps and invocation IDs; sweep children share an ID.
- Older snapshots keep their figures, arrays, and CSV links. Raw-training snapshots do not attach an older report. Weights, environments, and retained model directories are not duplicated.
- History grows without automatic pruning. Regeneration replaces manual edits to the live report; keep publication notes separately. History does not make incompatible measurements comparable.
Regenerate without training:
python -m quibble_ml report --swap lightgbm:tabfm --swap lightgbm:kumo_tabularRead results/report/report.md for current results or results/report/history.md for history.
Reports provide measured quality/resource shortlists and explain blocking swap checks;
they are dataset-specific recommendations, not guarantees. A swap FAIL is not an execution error.
CLI command reference and an OpenML benchmark example
| Command | Purpose |
|---|---|
quibble-ml demo |
Offline CPU demonstration; optional --backends and --force |
quibble-ml benchmark |
Device-aware presets, --smoke, optional --keep-models |
quibble-ml run --sweep --backends a,b --datasets x,y |
One worker process per model/dataset/split/seed; --retry-failed reruns only errors |
quibble-ml analyze --dataset x |
Intervals, calibration, slices, disputed-row CSVs |
quibble-ml swap --incumbent A --candidate B --dataset x |
Replacement verdict; exit codes 0 / 1 / 2; --require-thresholds requires an explicit policy |
quibble-ml gate --candidate B --dataset x |
Does B beat a simple baseline (linear, dummy)? Exit codes 0 / 1 / 2 |
quibble-ml report --swap A:B |
Markdown report and figures |
quibble-ml compare --dataset x |
Quick single-split summary across model seeds |
quibble-ml datasets --verify --names credit-g,phoneme |
Verify selected OpenML entries before use |
quibble-ml doctor |
Environment, cache root, CUDA detection, and available VRAM |
For a real OpenML benchmark:
# Create settings only if absent, then edit device and other settings as needed.
if (-not (Test-Path .env)) { Copy-Item .env.example .env }
python -m quibble_ml datasets --verify --names credit-g,phoneme
python -m quibble_ml run --sweep --backends histgb,lightgbm,catboost,autogluon --datasets credit-g,phoneme --cv 5x2 --model-seeds 0 --jobs 1
python -m quibble_ml report --swap histgb:lightgbmSee the CLI reference for all options and exit codes.
Same splits do not make every conclusion valid. Reports call out uncertainty, partial coverage, device differences, and policy-limit failures.
Statistical assumptions, preprocessing, timing, and swap thresholds
- Fixed split scheme, assignment seed, and split ID determine rows independently of model seed. LightGBM and HistGB on small datasets can be deterministic, giving zero seed instability.
ordinalencodes categoricals for supported adapters.nativeenables native handling in HistGB, LightGBM, CatBoost, TabFM, and Kumo; AutoGluon always uses its own preprocessing. Effective settings are recorded; compare preprocessing modes in isolated roots.- Timings are single-machine measurements. Use
--jobs 1; process peak RSS and PyTorch-only peak VRAM do not describe the same memory pool or every library's GPU allocations. - Bootstrap intervals cluster repeated predictions by original row (or by group when the
dataset declares
group_column). They measure test-row noise. With 5 or more splits, swap also computes an approximate Nadeau-Bengio corrected interval over splits and uses the outer bounds of the two. The combined verdict's error rates are not calibrated. Marginal interval overlap alone does not decide a paired difference or establish equivalence. - Swap checks PASS when the whole interval is within a limit, FAIL when wholly beyond, otherwise INCONCLUSIVE. A check with no usable interval is INCONCLUSIVE, never PASS. Latency and memory compare medians; memory is a baseline-adjusted process peak, not model-only footprint.
- The worst-slice bootstrap recomputes the maximum over fixed, target-selected masks.
Family-wise coverage remains unverified. Swap defaults live in the
packaged swap_thresholds.json, shared by the CLI and UI.
These are the limits Quibble uses, not universally safe acceptance criteria.
Export a copy with
quibble-ml swap --export-thresholds my_limits.json, edit it, and pass--thresholds my_limits.json. Use--require-thresholdsin CI to require a non-empty policy; partial overrides remain allowed and inherited defaults are listed explicitly. In the workbench, choose a full run and open Swap acceptance policy (full runs) to review defaults, edit or import custom JSON, and download the effective policy. Full experiments save their policy; reports show all effective limits. See Swap checks for units, validation, and examples. - With 2 or more model seeds, the disagreement check subtracts the incumbent's own seed-to-seed
disagreement. Disputed predictions differ in class, or by more than
regression_gap_std(default 0.25) target standard deviations for regression.
The same environment includes pytest. Contributions should include a focused regression test.
& .\.venv\Scripts\Activate.ps1
python -m pytestAdd a backend
-
To add a backend, subclass
_SKorBackend, implement fit/predict, and register it inBACKENDS. A simple_SKadapter can be about 20 lines:class MyModel(_SK): name, native_ok, gpu = "mymodel", False, False def make(self, task, seed, cfg, device, cat_idx): from mylib import Clf, Reg return (Clf if task == "classification" else Reg)(random_state=seed)
-
Raise
Refused("reason")for unsupported cases and add tests. See development.
Validation coverage, CI, Docker, and release status
- The CI definition targets Python 3.10 to 3.13 on Linux plus Python 3.13 on Windows, installing the locked dev group and web/data extras, but not heavy model backends. Remote execution of the updated workflow remains unverified.
- Docker entry points use modules, for example
docker compose run --rm cpu quibble_ml.doctor. Containers run as UID/GID 1000 and remain unvalidated; see GPU and Docker. - Evaluation, analysis, swap checks, and reporting are implemented. The package is not published; its distribution name is
quibble-mlbecause plainquibblebelongs to another project. The import isquibble_mland the command isquibble-ml; see migration instructions before reusing an older environment or retained model. - CPU tests and real smoke checks cover the tree backends, AutoGluon, TabFM, and Kumo.
Local records also show CUDA classification fits for CatBoost/TabFM/Kumo; they do not establish
full GPU coverage. All fourteen OpenML entries loaded and reloaded offline;
real HistGB/Weak holdouts covered
credit-gandabalone. TabPFN end-to-end, Docker and remote CI still need validation. TabICLv2 has completed real offline Wine classification and diabetes regression holdouts on CPU and CUDA, with HistGB comparison on CPU; broader coverage remains experimental. Its pinned adapter keeps cached-only weights by default. See current trust levels. - Hosted services are deferred. Remaining work includes the verified multi-dataset benchmark, release validation, final attribution, and reviewed benchmark configurations.
- Shipped: leakage and data flags, group and temporal splits for local data, simple baselines
(
dummy,linear), andquibble-ml gate. Still proposed: a Pythonrecommend()API and license-aware selection; see recommend and abstain. - The synthetic experiment workbench supports bounded independent-row generation; it does not validate domain realism. The configured planner drafts the specification, not the evaluated predictions. Controlled LLM prediction comparison remains proposed work.
Quibble is not published yet. Check trust levels, limitations, and the project plan before relying on a particular model or platform.
Start with the standalone documentation entry or full documentation index. The guides include installation and navigation without requiring this README as their starting point. Read Predicting from Tables for the three-part story: the opportunity for tabular foundation models, the model-choice problem, and the engineering of trustworthy comparisons, with diagrams and curated smoke evidence.
| Topic | Guides |
|---|---|
| Product and browser | About Quibble, Experiment workbench |
| Setup and operation | Getting started, CLI reference, Configuration, GPU and Docker, Troubleshooting |
| Models and data | Testing your own data, Backends, Datasets and source/download locations, Running benchmarks |
| Understanding evidence | Concepts, Analysis, Swap checks, Reports, Results format |
| Contributing and plans | Development, Limitations, Project plan |
Quibble is MIT licensed. Datasets, optional libraries, and pretrained checkpoints have their own licenses and access conditions.
Source repository: ushakrishnan/quibble. The GitHub Pages workflow builds the public docs and blog, including navigation and diagrams, without publishing local data, results, environment files, or model weights. About is the site's landing page. Workbench on the static site explains how to run experiments locally; GitHub Pages cannot host the Python server.
After pushing main, set repository Settings > Pages > Source to GitHub Actions.
The workflow's deployment summary supplies the actual site URL. The expected project
URL is https://ushakrishnan.github.io/quibble/, but it is not live until deployment succeeds.
See deployment and local preview.
