diff --git a/README.md b/README.md index 793cf76..9774893 100644 --- a/README.md +++ b/README.md @@ -683,8 +683,9 @@ on a particular model or platform. Start with the [standalone documentation entry](docs/README.md) or [full documentation index](docs/index.md). The guides include installation and navigation without requiring this README as their starting point. -Read [Predicting from Tables](docs/blog/README.md) for the two-part project story and -engineering lessons, with diagrams and curated smoke evidence. +Read [Predicting from Tables](docs/blog/README.md) for the three-part story: the +opportunity for tabular foundation models, the model-choice problem, and the +engineering of trustworthy comparisons, with diagrams and curated smoke evidence. | Topic | Guides | |---|---| diff --git a/docs/about.md b/docs/about.md index 04f2657..7e19be2 100644 --- a/docs/about.md +++ b/docs/about.md @@ -29,10 +29,11 @@ implemented adapters from paths verified on real hardware. For the motivation and engineering decisions, read the [Predicting from Tables blog series](blog/README.md): -[Same Accuracy, Different Mistakes](blog/01-same-accuracy-different-mistakes.md) and -[Making Prediction Comparisons Worth Trusting](blog/02-making-prediction-comparisons-worth-trusting.md). +[A Place for Tabular Foundation Models](blog/a-place-for-ltm.md), +[Same Accuracy, Different Mistakes](blog/same-accuracy-different-mistakes.md), and +[Making Prediction Comparisons Worth Trusting](blog/making-prediction-comparisons-worth-trusting.md). The articles provide context; the guides remain the reference for current behavior. -In the running workbench, About links to a local blog reader at `/blog`, with both +In the running workbench, About links to a local blog reader at `/blog`, with all three articles, diagrams, and evidence downloads. Public GitHub hosting is not required. ## Browser workflow diff --git a/docs/blog/README.md b/docs/blog/README.md index 769a1f1..07f736c 100644 --- a/docs/blog/README.md +++ b/docs/blog/README.md @@ -1,18 +1,26 @@ --- title: Predicting from Tables -description: A two-part Quibble story about prediction-level differences and engineering trustworthy comparisons. +description: From the opportunity for tabular foundation models to model decisions and the engineering of trustworthy comparisons. --- ## Predicting from Tables -A two-part series for applied ML engineers and technical leaders: why a model -comparison needs more than a score, and how Quibble makes its evidence inspectable. +A three-part series for applied ML engineers and technical leaders: the opportunity +for tabular foundation models, the decisions that opportunity creates, and the +engineering needed to make those decisions inspectable. -1. [Same Accuracy, Different Mistakes](01-same-accuracy-different-mistakes.md): +1. [A Place for Tabular Foundation Models](a-place-for-ltm.md): + what is possible today, what may come next, and the longer-term ambition for + reusable prediction across business data. +2. [Same Accuracy, Different Mistakes](same-accuracy-different-mistakes.md): the decision problem, changed predictions, and the limits of a small smoke check. -2. [Making Prediction Comparisons Worth Trusting](02-making-prediction-comparisons-worth-trusting.md): +3. [Making Prediction Comparisons Worth Trusting](making-prediction-comparisons-worth-trusting.md): row identity, source fingerprints, failure/device accounting, and local-data workflows. +Read in this order for the full argument: why the direction matters, how to judge +a candidate, and how to preserve the evidence. Each article starts with a short +summary and can also be read on its own. + The articles explain motivation and engineering decisions. The [documentation index](../index.md) remains the reference for current behavior and [validation status](../index.md#trust-levels-what-has-actually-been-run). diff --git a/docs/blog/a-place-for-ltm.md b/docs/blog/a-place-for-ltm.md new file mode 100644 index 0000000..223bea8 --- /dev/null +++ b/docs/blog/a-place-for-ltm.md @@ -0,0 +1,259 @@ +--- +title: A Place for Tabular Foundation Models +description: What tabular foundation models offer today, what may come next, and why their ambition needs trustworthy comparisons. +--- + +## A Place for Tabular Foundation Models + +*Predicting from Tables, Part 1: the opportunity.* + +> Summary: Tabular foundation models could make learning across business tables +> reusable. Here is what that means today, what may come next, and the longer-term +> ambition, without treating a promising direction as a guaranteed outcome. + +Suppose you want to predict which invoices will be paid late. You have a table of +past invoices, customer attributes, payment terms, and known outcomes. For each new +invoice, you need a prediction using only information available when it is issued. + +A tree-based model is a reasonable place to start. So is an AutoML workflow that +trains and compares several methods. Now there is another candidate: a predictor +that arrives with knowledge learned before it sees your table. + +That is what interests me about tabular foundation models, sometimes called +Large Tabular Models (LTMs). + +Not the idea that every business needs a bigger model. The possibility of moving +from separate models for separate tables toward a reusable predictive layer for +structured business data. + +Today, that starts with predictions on a defined table. The larger ambition is +to make useful predictions across business problems without rebuilding the +learning workflow each time. That is a bigger idea than a faster classifier, +even though a classifier may be where we first test it. + +The question is not whether this is a newer category of AI. It is whether that +prior learning makes a useful difference on the task in front of us. + +## Today: reusable learning for a new table + +With conventional supervised ML, we typically fit a model to examples from the +task we want it to perform. For the invoice problem, that means learning from +past invoices and their payment outcomes. + +A tabular foundation model has already been pretrained across a collection of +tasks or datasets. That pretraining may use real tables, synthetic tasks, or a +mixture, depending on the model. The aim is to learn patterns that help it make +predictions on a new table. + +Some approaches use *in-context learning*: they condition on labeled examples +from the new task to predict outcomes for unlabeled rows, without a conventional +round of task-specific weight updates. Other approaches may include fine-tuning +or additional adaptation. + +This is not only a future aspiration: pretrained tabular predictors are available +to evaluate today. Their supported tasks, operating limits, and adaptation options +depend on the model, version, and configuration. The category name does not +establish that every predictor behaves the same way. + +There are two qualifications I would keep close to the promise. + +First, "no task-specific weight training" does not mean "no task-specific data." +The model may still need labeled examples from your business. Second, work has +not disappeared because a library's `fit()` call is short. Loading weights, +preparing context, and prediction can still require substantial computation. + +The potential advantage is a different starting point for learning, not an +exemption from data preparation or evaluation. + +## A place alongside other tools, not above them + +It helps to separate the jobs in a business question: + +> Which customers generated more than $100,000 in revenue last year, and which +> of them are likely to pay their next invoice late? + +The first part is a query. The second is a prediction. + +| Job | Suitable role | +|---|---| +| Calculate historical revenue | A database or query engine operating on authoritative records | +| Estimate late-payment risk | An evaluated predictor, whether tree-based, pretrained, or another method | +| Interpret the request and communicate results | An application or an LLM-assisted workflow with checked data and tool access | + +This is a division of responsibilities, not a rule that language models cannot +predict or that databases cannot host ML. It keeps us clear about which component +produced a number and how that number should be checked. + +An LLM might write a query, invoke a predictor, and explain its output. That does +not make the LLM the source of the historical revenue or establish the quality +of the predictor it invoked. + +Equally, AutoML and tabular foundation models are not mutually exclusive +categories. AutoML is a way to select, tune, and combine predictors. A pretrained +tabular model can be one of its candidates. + +The useful architecture depends on the work, not the label attached to a tool. + +## Where I would try one + +The most interesting starting point is a recurring prediction task where getting +to a credible baseline takes too much effort. + +Perhaps there are relatively few labeled invoices. Perhaps the team has several +related prediction problems and wants to reduce repeated model-selection work. +Perhaps a new dataset needs an initial assessment before anyone commits to a +larger training pipeline. + +These are reasons to include a pretrained predictor in an experiment. They are +not guarantees of an advantage. + +A conventional model can work very well on a small table. A pretrained model can +struggle when the task differs from what its pretraining prepared it for. A good +AutoML baseline may already remove much of the manual work we hoped to save. + +I would therefore turn the promise into a question: + +> Can prior learning deliver useful predictions with fewer labeled examples, +> less task-specific tuning, or a better operational tradeoff? + +Each possible benefit needs its own measurement. Fewer labels is not the same +benefit as lower inference cost. Faster experimentation is not the same as +higher predictive quality. + +## What may come next: less rebuilding between tasks + +The next step I would look for is not merely a higher score on another table. +It is a reduction in how much work must be repeated when the question changes. + +For the invoice example, that could mean moving from predicting late payment to +estimating days overdue, then adapting to another business unit with different +customer and payment patterns. The targets still need definitions, and the +available data still needs review. The opportunity is to reuse more of the +learning and adaptation machinery around those tasks. + +Useful progress could take several forms: + +- Adaptation to a new task with fewer labeled examples. +- Broader support for table sizes, data types, and imperfect inputs. +- More economical repeated prediction, not only a short initial fit. +- Better ways to identify when transferred knowledge is a poor match. + +These are capabilities to watch and test, not a release schedule. Some already +exist in particular implementations; none should be assumed for the whole +category. I would say "what may come next," rather than "what we will see soon." + +## Further out: a predictive layer across business data + +Real invoice data rarely begins as a single clean training table. Customer +history, orders, payments, and support interactions may live in different +systems. Someone still has to decide how to join them, what a row represents, +and which facts were available at prediction time. + +A pretrained single-table predictor does not automatically solve that work. + +The more ambitious possibility is a system that can work across those connected +records, adapt to a company's operating patterns, and support several prediction +tasks without a separate hand-built pipeline for each one. + +Imagine asking which outstanding invoices are at risk, using payment history, +changing order patterns, and unresolved support issues. The system would need +to connect those records, respect what was known at the time, estimate risk, +and expose enough evidence for someone to challenge the answer. + +That is where tabular prediction, relational modeling, and language-driven +interfaces could meet. The language interface makes the request accessible; +it does not remove the need for an evaluated predictive system underneath. + +There are harder boundaries too. Learning that two variables move together does +not establish what an intervention will cause. A late-payment predictor cannot, +by itself, tell us whether changing payment terms will improve collections. +Moving from prediction to recommendations needs additional assumptions, +constraints, and validation. + +This is the horizon I find most interesting: not a universal model that +magically understands every spreadsheet, but more reusable prediction across +the connected data a business actually uses. It is an ambition, not a capability +I would infer from the phrase "tabular foundation model." + +## What would earn that ambition a place? + +For the invoice task, I would start with a credible conventional baseline and a +pretrained candidate. An AutoML workflow could provide another practical +comparison. Then I would ask: + +| Question | Evidence to collect | +|---|---| +| Does it predict better? | Task-appropriate held-out metrics on the same evaluation rows | +| Does it need fewer labels? | Learning curves using comparable subsets of training data | +| Does it reduce experimentation effort? | Recorded preparation, tuning, and elapsed time under a declared budget | +| What changes if we use it? | Paired predictions, changed decisions, and performance on relevant groups | +| Can we operate it? | Fit and prediction costs, memory, hardware needs, and execution failures | +| Can we use its confidence? | Probability-quality and calibration checks where outputs support them | + +The split must represent the intended use. Predicting future invoices can require +time-aware evaluation; repeated customers can require group-aware evaluation, +depending on whether we expect to serve known or new customers. A random split +does not automatically answer either question. + +The same evaluation rows are necessary, but not sufficient. Information access, +preprocessing, tuning budgets, hardware, and failures also affect the comparison. +Pretraining itself represents prior data and compute, even if its cost does not +appear in a local run. + +And a public benchmark needs another caveat: we may not know whether its data +appeared during a model's pretraining. Fresh synthetic rows can help test a +controlled hypothesis, but their generating rules may favor particular methods. +Neither is an automatic certificate of generalization to a new business. + +## From the opportunity to the evidence + +This is where my interest in tabular foundation models meets the reason I built +**Quibble**. + +The opportunity comes first. If learning becomes more reusable, more teams may +have credible new predictors to consider. That makes the next question harder: +how do we know when changing predictors is actually worthwhile? + +[Part 2, Same Accuracy, Different Mistakes](same-accuracy-different-mistakes.md), +takes up that decision. +[Part 3, Making Prediction Comparisons Worth Trusting](making-prediction-comparisons-worth-trusting.md), +then explores the engineering needed to preserve the evidence behind it. + +A pretrained predictor makes those questions more relevant, not less. + +If it matches a tree's accuracy, does it make the same mistakes? If it improves +the average, does an important group become worse? If fitting is fast, what +happens to repeated prediction? If it cannot complete some runs, are those +failures visible? + +Quibble compares supported prediction libraries using aligned evaluation rows, +saved predictions, and recorded execution outcomes. It is not a new foundation +model, and the work described in the series is not evidence that pretrained +methods outperform conventional ML. + +Nor does the current workflow answer every question above. Time/group-aware +evaluation, a broader verified dataset campaign, and measurements of human +engineering effort would need additional work. The +[limitations](../limitations.md) and +[validation status](../index.md#trust-levels-what-has-actually-been-run) +keep those boundaries explicit. + +Its contribution is narrower: make a particular comparison inspectable enough +to question. + +## A challenger, not a foregone conclusion + +I would give tabular foundation models a place in the candidate set, not a +reserved place in production. + +If one produces better predictions under the constraints that matter, use that +evidence. If it produces comparable predictions with less task-specific effort, +that can be valuable too. If a tree is cheaper, more reliable, or better suited +to the task, keeping it is a useful outcome. + +The promise is reusable learning for structured data. + +The test is whether that reuse improves the work we actually need to do. + +Continue with [Part 2: Same Accuracy, Different Mistakes](same-accuracy-different-mistakes.md) +for why a headline score cannot settle that question. diff --git a/docs/blog/02-making-prediction-comparisons-worth-trusting.md b/docs/blog/making-prediction-comparisons-worth-trusting.md similarity index 97% rename from docs/blog/02-making-prediction-comparisons-worth-trusting.md rename to docs/blog/making-prediction-comparisons-worth-trusting.md index 92675f3..63cd783 100644 --- a/docs/blog/02-making-prediction-comparisons-worth-trusting.md +++ b/docs/blog/making-prediction-comparisons-worth-trusting.md @@ -5,7 +5,11 @@ description: Inside Quibble's controlled execution, source fingerprints, paired ## Making Prediction Comparisons Worth Trusting -*Predicting from Tables, Part 2: engineering the evidence.* +*Predicting from Tables, Part 3: engineering the evidence.* + +> Summary: A trustworthy comparison needs traceable rows, saved predictions, +> explicit failures, and clearly defined costs. This post explains how Quibble +> preserves that evidence and where its current guarantees stop. A comparison report is only as useful as the chain of evidence behind it. @@ -13,9 +17,11 @@ I built **Quibble**, a tool that compares ways to predict categories or numbers tables, to keep that chain inspectable: the evaluation rows, predicted answers, execution outcomes, and checks used to assess a proposed change. -[Part 1, Same Accuracy, Different Mistakes](01-same-accuracy-different-mistakes.md), -explains the problem behind it. Here, I walk through the engineering decisions that -connect an experiment to a report a reviewer can question. +[Part 1](a-place-for-ltm.md) introduces the opportunity for tabular foundation +models. [Part 2, Same Accuracy, Different Mistakes](same-accuracy-different-mistakes.md), +explains why choosing a predictor needs more than a score. Here, I walk through +the engineering decisions that connect an experiment to a report a reviewer can +question. The task is to predict categories or numbers for cases whose outcomes are held back for evaluation. Current methods include trees, AutoML, and pretrained tabular predictors. @@ -464,4 +470,4 @@ For anyone building an evaluation system, the transferable question is the same: Start with [Getting started](../getting-started.md), follow the [local-data walkthrough](../testing-your-data.md), or return to -[Part 1](01-same-accuracy-different-mistakes.md) for the decision framework. +[Part 2](same-accuracy-different-mistakes.md) for the decision framework. diff --git a/docs/blog/01-same-accuracy-different-mistakes.md b/docs/blog/same-accuracy-different-mistakes.md similarity index 95% rename from docs/blog/01-same-accuracy-different-mistakes.md rename to docs/blog/same-accuracy-different-mistakes.md index 73bb4d7..80e0318 100644 --- a/docs/blog/01-same-accuracy-different-mistakes.md +++ b/docs/blog/same-accuracy-different-mistakes.md @@ -7,7 +7,15 @@ description: How do you choose a way to predict from tables when the score does *How do you choose a way to predict from tables when the score doesn't tell the whole story?* -*Predicting from Tables, Part 1.* +*Predicting from Tables, Part 2: the decision.* + +> Summary: Matching accuracy can hide different mistakes, costs, and failure +> patterns. Quibble makes those differences inspectable so a model change can be +> judged on more than its headline score. + +[Part 1, A Place for Tabular Foundation Models](a-place-for-ltm.md), +sets out the opportunity for reusable prediction. Here, the question becomes +how to decide whether a new predictor is worth using. Suppose you have a table of historical invoices. Each row describes an invoice, the information available when it was issued, and whether it was paid late. You want to @@ -65,7 +73,7 @@ itself made those predictions unaided. Quibble currently evaluates supported prediction libraries, not a general ChatGPT workflow. A controlled LLM comparison is a future direction, not a completed experiment. It would need explicit data access, prompts, tool permissions, output validation, and -cost accounting. [Part 2](02-making-prediction-comparisons-worth-trusting.md#what-a-fair-llm-comparison-would-require) +cost accounting. [Part 3](making-prediction-comparisons-worth-trusting.md#what-a-fair-llm-comparison-would-require) outlines that extension. So the question is not "traditional ML or AI?" It is: **which approach produces useful @@ -306,7 +314,7 @@ Before accepting a replacement, ask for more than its score: That is the problem I chose to work on: not simply running more models, but making the decision between them easier to inspect. -Continue with **[Part 2: Making Prediction Comparisons Worth Trusting](02-making-prediction-comparisons-worth-trusting.md)** for +Continue with **[Part 3: Making Prediction Comparisons Worth Trusting](making-prediction-comparisons-worth-trusting.md)** for the architecture, source-identity controls, operational tradeoffs, and local-data workflow. To try it first, start with [Getting started](../getting-started.md) or [Testing your own data](../testing-your-data.md). diff --git a/quibble_ml/web.py b/quibble_ml/web.py index 17c07f2..10fd88f 100644 --- a/quibble_ml/web.py +++ b/quibble_ml/web.py @@ -33,8 +33,9 @@ ("Reference & support", ("cli-reference.md", "troubleshooting.md", "limitations.md", "index.md", "development.md", "project-plan.md")), ("Ideas & rationale", ("rationale.md", "recommend-abstain.md")), - ("Blog", ("blog/README.md", "blog/01-same-accuracy-different-mistakes.md", - "blog/02-making-prediction-comparisons-worth-trusting.md")), + ("Blog", ("blog/README.md", "blog/a-place-for-ltm.md", + "blog/same-accuracy-different-mistakes.md", + "blog/making-prediction-comparisons-worth-trusting.md")), ) DOCS = Path(__file__).with_name("docs") if not DOCS.is_dir(): diff --git a/quibble_ml/web_assets/about.html b/quibble_ml/web_assets/about.html index f2243ec..b0ee190 100644 --- a/quibble_ml/web_assets/about.html +++ b/quibble_ml/web_assets/about.html @@ -57,8 +57,9 @@
Version __QUIBBLE_VERSION__ · Pre-release · MIT license
Source: ushakrishnan/quibble on GitHub. Docs and Blog are available through this site. Start with the documentation for installation, the workbench guide, CLI reference, troubleshooting, and validation status.
Read the Predicting from Tables blog series: - Same Accuracy, Different Mistakes - and Making Prediction Comparisons Worth Trusting. + A Place for Tabular Foundation Models, + Same Accuracy, Different Mistakes, + and Making Prediction Comparisons Worth Trusting. Articles and diagrams open locally; no public hosting is required.
Quibble is an experimental pre-release. The package name is quibble-ml; the Python import and command are quibble. See the repository's MIT license and issue tracker. Optional libraries, datasets, and checkpoints have their own terms.