Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -683,8 +683,9 @@ on a particular model or platform.
Start with the [standalone documentation entry](docs/README.md) or
[full documentation index](docs/index.md). The guides include installation and navigation
without requiring this README as their starting point.
Read [Predicting from Tables](docs/blog/README.md) for the two-part project story and
engineering lessons, with diagrams and curated smoke evidence.
Read [Predicting from Tables](docs/blog/README.md) for the three-part story: the
opportunity for tabular foundation models, the model-choice problem, and the
engineering of trustworthy comparisons, with diagrams and curated smoke evidence.

| Topic | Guides |
|---|---|
Expand Down
7 changes: 4 additions & 3 deletions docs/about.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,11 @@ implemented adapters from paths verified on real hardware.

For the motivation and engineering decisions, read the
[Predicting from Tables blog series](blog/README.md):
[Same Accuracy, Different Mistakes](blog/01-same-accuracy-different-mistakes.md) and
[Making Prediction Comparisons Worth Trusting](blog/02-making-prediction-comparisons-worth-trusting.md).
[A Place for Tabular Foundation Models](blog/a-place-for-ltm.md),
[Same Accuracy, Different Mistakes](blog/same-accuracy-different-mistakes.md), and
[Making Prediction Comparisons Worth Trusting](blog/making-prediction-comparisons-worth-trusting.md).
The articles provide context; the guides remain the reference for current behavior.
In the running workbench, About links to a local blog reader at `/blog`, with both
In the running workbench, About links to a local blog reader at `/blog`, with all three
articles, diagrams, and evidence downloads. Public GitHub hosting is not required.

## Browser workflow
Expand Down
18 changes: 13 additions & 5 deletions docs/blog/README.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,26 @@
---
title: Predicting from Tables
description: A two-part Quibble story about prediction-level differences and engineering trustworthy comparisons.
description: From the opportunity for tabular foundation models to model decisions and the engineering of trustworthy comparisons.
---

## Predicting from Tables

A two-part series for applied ML engineers and technical leaders: why a model
comparison needs more than a score, and how Quibble makes its evidence inspectable.
A three-part series for applied ML engineers and technical leaders: the opportunity
for tabular foundation models, the decisions that opportunity creates, and the
engineering needed to make those decisions inspectable.

1. [Same Accuracy, Different Mistakes](01-same-accuracy-different-mistakes.md):
1. [A Place for Tabular Foundation Models](a-place-for-ltm.md):
what is possible today, what may come next, and the longer-term ambition for
reusable prediction across business data.
2. [Same Accuracy, Different Mistakes](same-accuracy-different-mistakes.md):
the decision problem, changed predictions, and the limits of a small smoke check.
2. [Making Prediction Comparisons Worth Trusting](02-making-prediction-comparisons-worth-trusting.md):
3. [Making Prediction Comparisons Worth Trusting](making-prediction-comparisons-worth-trusting.md):
row identity, source fingerprints, failure/device accounting, and local-data workflows.

Read in this order for the full argument: why the direction matters, how to judge
a candidate, and how to preserve the evidence. Each article starts with a short
summary and can also be read on its own.

The articles explain motivation and engineering decisions. The
[documentation index](../index.md) remains the reference for current behavior and
[validation status](../index.md#trust-levels-what-has-actually-been-run).
Expand Down
259 changes: 259 additions & 0 deletions docs/blog/a-place-for-ltm.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,259 @@
---
title: A Place for Tabular Foundation Models
description: What tabular foundation models offer today, what may come next, and why their ambition needs trustworthy comparisons.
---

## A Place for Tabular Foundation Models

*Predicting from Tables, Part 1: the opportunity.*

> Summary: Tabular foundation models could make learning across business tables
> reusable. Here is what that means today, what may come next, and the longer-term
> ambition, without treating a promising direction as a guaranteed outcome.

Suppose you want to predict which invoices will be paid late. You have a table of
past invoices, customer attributes, payment terms, and known outcomes. For each new
invoice, you need a prediction using only information available when it is issued.

A tree-based model is a reasonable place to start. So is an AutoML workflow that
trains and compares several methods. Now there is another candidate: a predictor
that arrives with knowledge learned before it sees your table.

That is what interests me about tabular foundation models, sometimes called
Large Tabular Models (LTMs).

Not the idea that every business needs a bigger model. The possibility of moving
from separate models for separate tables toward a reusable predictive layer for
structured business data.

Today, that starts with predictions on a defined table. The larger ambition is
to make useful predictions across business problems without rebuilding the
learning workflow each time. That is a bigger idea than a faster classifier,
even though a classifier may be where we first test it.

The question is not whether this is a newer category of AI. It is whether that
prior learning makes a useful difference on the task in front of us.

## Today: reusable learning for a new table

With conventional supervised ML, we typically fit a model to examples from the
task we want it to perform. For the invoice problem, that means learning from
past invoices and their payment outcomes.

A tabular foundation model has already been pretrained across a collection of
tasks or datasets. That pretraining may use real tables, synthetic tasks, or a
mixture, depending on the model. The aim is to learn patterns that help it make
predictions on a new table.

Some approaches use *in-context learning*: they condition on labeled examples
from the new task to predict outcomes for unlabeled rows, without a conventional
round of task-specific weight updates. Other approaches may include fine-tuning
or additional adaptation.

This is not only a future aspiration: pretrained tabular predictors are available
to evaluate today. Their supported tasks, operating limits, and adaptation options
depend on the model, version, and configuration. The category name does not
establish that every predictor behaves the same way.

There are two qualifications I would keep close to the promise.

First, "no task-specific weight training" does not mean "no task-specific data."
The model may still need labeled examples from your business. Second, work has
not disappeared because a library's `fit()` call is short. Loading weights,
preparing context, and prediction can still require substantial computation.

The potential advantage is a different starting point for learning, not an
exemption from data preparation or evaluation.

## A place alongside other tools, not above them

It helps to separate the jobs in a business question:

> Which customers generated more than $100,000 in revenue last year, and which
> of them are likely to pay their next invoice late?

The first part is a query. The second is a prediction.

| Job | Suitable role |
|---|---|
| Calculate historical revenue | A database or query engine operating on authoritative records |
| Estimate late-payment risk | An evaluated predictor, whether tree-based, pretrained, or another method |
| Interpret the request and communicate results | An application or an LLM-assisted workflow with checked data and tool access |

This is a division of responsibilities, not a rule that language models cannot
predict or that databases cannot host ML. It keeps us clear about which component
produced a number and how that number should be checked.

An LLM might write a query, invoke a predictor, and explain its output. That does
not make the LLM the source of the historical revenue or establish the quality
of the predictor it invoked.

Equally, AutoML and tabular foundation models are not mutually exclusive
categories. AutoML is a way to select, tune, and combine predictors. A pretrained
tabular model can be one of its candidates.

The useful architecture depends on the work, not the label attached to a tool.

## Where I would try one

The most interesting starting point is a recurring prediction task where getting
to a credible baseline takes too much effort.

Perhaps there are relatively few labeled invoices. Perhaps the team has several
related prediction problems and wants to reduce repeated model-selection work.
Perhaps a new dataset needs an initial assessment before anyone commits to a
larger training pipeline.

These are reasons to include a pretrained predictor in an experiment. They are
not guarantees of an advantage.

A conventional model can work very well on a small table. A pretrained model can
struggle when the task differs from what its pretraining prepared it for. A good
AutoML baseline may already remove much of the manual work we hoped to save.

I would therefore turn the promise into a question:

> Can prior learning deliver useful predictions with fewer labeled examples,
> less task-specific tuning, or a better operational tradeoff?

Each possible benefit needs its own measurement. Fewer labels is not the same
benefit as lower inference cost. Faster experimentation is not the same as
higher predictive quality.

## What may come next: less rebuilding between tasks

The next step I would look for is not merely a higher score on another table.
It is a reduction in how much work must be repeated when the question changes.

For the invoice example, that could mean moving from predicting late payment to
estimating days overdue, then adapting to another business unit with different
customer and payment patterns. The targets still need definitions, and the
available data still needs review. The opportunity is to reuse more of the
learning and adaptation machinery around those tasks.

Useful progress could take several forms:

- Adaptation to a new task with fewer labeled examples.
- Broader support for table sizes, data types, and imperfect inputs.
- More economical repeated prediction, not only a short initial fit.
- Better ways to identify when transferred knowledge is a poor match.

These are capabilities to watch and test, not a release schedule. Some already
exist in particular implementations; none should be assumed for the whole
category. I would say "what may come next," rather than "what we will see soon."

## Further out: a predictive layer across business data

Real invoice data rarely begins as a single clean training table. Customer
history, orders, payments, and support interactions may live in different
systems. Someone still has to decide how to join them, what a row represents,
and which facts were available at prediction time.

A pretrained single-table predictor does not automatically solve that work.

The more ambitious possibility is a system that can work across those connected
records, adapt to a company's operating patterns, and support several prediction
tasks without a separate hand-built pipeline for each one.

Imagine asking which outstanding invoices are at risk, using payment history,
changing order patterns, and unresolved support issues. The system would need
to connect those records, respect what was known at the time, estimate risk,
and expose enough evidence for someone to challenge the answer.

That is where tabular prediction, relational modeling, and language-driven
interfaces could meet. The language interface makes the request accessible;
it does not remove the need for an evaluated predictive system underneath.

There are harder boundaries too. Learning that two variables move together does
not establish what an intervention will cause. A late-payment predictor cannot,
by itself, tell us whether changing payment terms will improve collections.
Moving from prediction to recommendations needs additional assumptions,
constraints, and validation.

This is the horizon I find most interesting: not a universal model that
magically understands every spreadsheet, but more reusable prediction across
the connected data a business actually uses. It is an ambition, not a capability
I would infer from the phrase "tabular foundation model."

## What would earn that ambition a place?

For the invoice task, I would start with a credible conventional baseline and a
pretrained candidate. An AutoML workflow could provide another practical
comparison. Then I would ask:

| Question | Evidence to collect |
|---|---|
| Does it predict better? | Task-appropriate held-out metrics on the same evaluation rows |
| Does it need fewer labels? | Learning curves using comparable subsets of training data |
| Does it reduce experimentation effort? | Recorded preparation, tuning, and elapsed time under a declared budget |
| What changes if we use it? | Paired predictions, changed decisions, and performance on relevant groups |
| Can we operate it? | Fit and prediction costs, memory, hardware needs, and execution failures |
| Can we use its confidence? | Probability-quality and calibration checks where outputs support them |

The split must represent the intended use. Predicting future invoices can require
time-aware evaluation; repeated customers can require group-aware evaluation,
depending on whether we expect to serve known or new customers. A random split
does not automatically answer either question.

The same evaluation rows are necessary, but not sufficient. Information access,
preprocessing, tuning budgets, hardware, and failures also affect the comparison.
Pretraining itself represents prior data and compute, even if its cost does not
appear in a local run.

And a public benchmark needs another caveat: we may not know whether its data
appeared during a model's pretraining. Fresh synthetic rows can help test a
controlled hypothesis, but their generating rules may favor particular methods.
Neither is an automatic certificate of generalization to a new business.

## From the opportunity to the evidence

This is where my interest in tabular foundation models meets the reason I built
**Quibble**.

The opportunity comes first. If learning becomes more reusable, more teams may
have credible new predictors to consider. That makes the next question harder:
how do we know when changing predictors is actually worthwhile?

[Part 2, Same Accuracy, Different Mistakes](same-accuracy-different-mistakes.md),
takes up that decision.
[Part 3, Making Prediction Comparisons Worth Trusting](making-prediction-comparisons-worth-trusting.md),
then explores the engineering needed to preserve the evidence behind it.

A pretrained predictor makes those questions more relevant, not less.

If it matches a tree's accuracy, does it make the same mistakes? If it improves
the average, does an important group become worse? If fitting is fast, what
happens to repeated prediction? If it cannot complete some runs, are those
failures visible?

Quibble compares supported prediction libraries using aligned evaluation rows,
saved predictions, and recorded execution outcomes. It is not a new foundation
model, and the work described in the series is not evidence that pretrained
methods outperform conventional ML.

Nor does the current workflow answer every question above. Time/group-aware
evaluation, a broader verified dataset campaign, and measurements of human
engineering effort would need additional work. The
[limitations](../limitations.md) and
[validation status](../index.md#trust-levels-what-has-actually-been-run)
keep those boundaries explicit.

Its contribution is narrower: make a particular comparison inspectable enough
to question.

## A challenger, not a foregone conclusion

I would give tabular foundation models a place in the candidate set, not a
reserved place in production.

If one produces better predictions under the constraints that matter, use that
evidence. If it produces comparable predictions with less task-specific effort,
that can be valuable too. If a tree is cheaper, more reliable, or better suited
to the task, keeping it is a useful outcome.

The promise is reusable learning for structured data.

The test is whether that reuse improves the work we actually need to do.

Continue with [Part 2: Same Accuracy, Different Mistakes](same-accuracy-different-mistakes.md)
for why a headline score cannot settle that question.
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,23 @@ description: Inside Quibble's controlled execution, source fingerprints, paired

## Making Prediction Comparisons Worth Trusting

*Predicting from Tables, Part 2: engineering the evidence.*
*Predicting from Tables, Part 3: engineering the evidence.*

> Summary: A trustworthy comparison needs traceable rows, saved predictions,
> explicit failures, and clearly defined costs. This post explains how Quibble
> preserves that evidence and where its current guarantees stop.

A comparison report is only as useful as the chain of evidence behind it.

I built **Quibble**, a tool that compares ways to predict categories or numbers from
tables, to keep that chain inspectable: the evaluation rows, predicted answers,
execution outcomes, and checks used to assess a proposed change.

[Part 1, Same Accuracy, Different Mistakes](01-same-accuracy-different-mistakes.md),
explains the problem behind it. Here, I walk through the engineering decisions that
connect an experiment to a report a reviewer can question.
[Part 1](a-place-for-ltm.md) introduces the opportunity for tabular foundation
models. [Part 2, Same Accuracy, Different Mistakes](same-accuracy-different-mistakes.md),
explains why choosing a predictor needs more than a score. Here, I walk through
the engineering decisions that connect an experiment to a report a reviewer can
question.

The task is to predict categories or numbers for cases whose outcomes are held back
for evaluation. Current methods include trees, AutoML, and pretrained tabular predictors.
Expand Down Expand Up @@ -464,4 +470,4 @@ For anyone building an evaluation system, the transferable question is the same:

Start with [Getting started](../getting-started.md), follow the
[local-data walkthrough](../testing-your-data.md), or return to
[Part 1](01-same-accuracy-different-mistakes.md) for the decision framework.
[Part 2](same-accuracy-different-mistakes.md) for the decision framework.
Loading
Loading