Activation Alignment for Tabular In-Context Learning
Official implementation of the paper Closing the Context Gap: Activation Alignment for Tabular In-Context Learning.
Tabular foundation models like TabPFN-3 and TabFM learn in-context. They are transformers that attend over the labeled training set, so unlike classical models such as XGBoost, they never decouple training from inference: every prediction carries the whole training set through the forward pass, at a cost quadratic in context length. Caching the intermediate key-value states avoids the recompute, but only by trading it for device memory that grows with the context. Shrinking the context dodges both costs, and hurts accuracy.
Activation alignment buys back a significant portion of that accuracy without enlarging the context. We use the full dataset once, offline, to teach a linear aligner how a small-context "student" should have represented its inputs had it seen the entire dataset like the "teacher". At inference the aligner is applied on top of the student's intermediate layer activations, and the context stays small.
- Trains on unlabeled synthetic queries, so no extra labels are needed.
- The aligner itself trains in seconds to minutes on a CPU. No fine-tuning the foundation model, whose weights stay frozen.
- Costs one matrix multiply at inference. Context size stays small, and so does the inference cost.
38 TabArena classification datasets, 5 repeats, ROC-AUC (binary) and negative log-loss (multiclass).
| TabPFN-3 | TabFM | |
|---|---|---|
| Win rate vs. unaligned student | 82.2% | 79.8% |
| Win rate vs. XGBoost trained on 100% of the data | 81.1% | 80.3% |
| Median teacher gap closed at a 10% context budget | 48.3% | 44.7% |
| Average rank (aligned / unaligned / XGBoost) | 1.37 / 2.19 / 2.44 | 1.40 / 2.13 / 2.47 |
All win rates are significant at
Data amplification. At a 10% context budget the aligned student matches an unaligned student given roughly two to four times as many labeled examples.
Let
| Stage | Script | What happens |
|---|---|---|
| 1. Synthesize queries | create_synthetic_dataset.py |
Sample 1,000 unlabeled queries |
| 2. Extract activations | extract_activations.py |
Push |
| 3. Fit the aligner | train_activation_aligner.py |
Fit |
| 4. Align at inference | evaluate_aligned_student.py |
Register the hook on real test data and replace the layer- |
Residual parameterization. The aligner predicts the correction rather than the target:
Weights start near zero, so the aligner begins as approximately a no-op and only learns to move activations where it helps.
Requires Python 3.13.
python -m venv venv
./venv/bin/python -m pip install -r requirements.txtTabPFN-3 weights are license-gated. The first run opens a browser so you can log in at ux.priorlabs.ai and accept the license.
TabFM weights (google/tabfm-1.0.0-pytorch,
non-commercial license) are not gated, so no authentication is required.
Run the full pipeline on a single dataset to see it end to end:
./venv/bin/python main.py \
--model tabpfn \
--dataset tabarena/diabetes \
--student_n 0.1 \
--n_repeats 1This synthesizes queries, extracts teacher and student activations, fits the aligner, and evaluates the aligned
student against the unaligned student, the full-context teacher, and XGBoost. Intermediate artifacts are cached
under results/, so re-runs are cheap.
| Flag | Meaning | Default |
|---|---|---|
--model |
tabpfn or tabfm
|
tabpfn |
--dataset |
Dataset name(s), or tabarena for all 38 |
tabarena |
--student_n |
Context budgets, as fractions in |
0.1 ... 0.9 |
--layers |
Layer indices to hook |
23 (final layer) |
--n_samples |
Synthetic queries used to fit the aligner | 1000 |
--n_repeats |
OpenML split repeats | 1 |
--n_estimators |
Ensemble members |
8 (TabPFN) / 1 (TabFM) |
--lr |
Aligner learning rate |
1e-3 (TabPFN) / 1e-4 (TabFM) |
create_figures.py builds every figure and table, except the layer-sensitivity table (see below).
It reads only from the cache and silently skips (dataset, student_n, repeat) combinations
it cannot find, so run the sweep first or you will get partial plots with no error.
./venv/bin/python create_figures.py \
--model tabpfn tabfm \
--dataset tabarena \
--layer 23 \
--student-n 0.1 0.2 0.3 0.4 0.5 \
--n-repeats 5 \
--output paper/figures| Output | Content |
|---|---|
win_rate_bar_chart.png |
Win rates vs. the unaligned student and full-data XGBoost, with Wilcoxon annotations |
scaling_curves.png |
Sample efficiency |
avg_rank_histogram.png |
3-way average ranking against the baseline student and XGBoost |
sample_efficiency_table.tex |
Per-dataset |
The layer-sensitivity table (relative gap closed vs. aligned layer layer_sensitivity_table.py. Beforehand, run the sweep on the 10 datasets used in the paper:
./venv/bin/python -m modal run --detach main_modal.py::sweep \
--model tabpfn \
--dataset "tabarena/GiveMeSomeCredit,tabarena/taiwanese_bankruptcy_prediction,tabarena/in_vehicle_coupon_recommendation,tabarena/HR_Analytics_Job_Change_of_Data_Scientists,tabarena/website_phishing,tabarena/seismic-bumps,tabarena/Diabetes130US,tabarena/students_dropout_and_academic_success,tabarena/Is-this-a-good-customer,tabarena/Marketing_Campaign" \
--student-n 0.3 \
--layers "5 11 17 20 21 22 23" \
--n-repeats 5Then build the table. With no arguments it uses the same 10 datasets and writes
layer_sensitivity/layer_table_tabpfn_alpha0.3.tex under the results directory:
./venv/bin/python -m modal run main_modal.py::layer_sensitivity_tableLike create_figures.py, it reads only from the cache; missing (dataset, layer, repeat) combinations are
reported as missing: ... and rendered as ---.
The full sweep is 38 datasets x 9 context budgets x 5 repeats. Modal fans this out so that
each (dataset, student_n, repeat, layer) combination runs as its own container, with a persistent Volume
(tabular-cache) holding the joblib cache, model weights, and OpenML downloads.
# One-time authentication
./venv/bin/python -m modal setup
# Smoke test
./venv/bin/python -m modal run main_modal.py::sweep \
--dataset tabarena/diabetes --student-n 0.1 --n-repeats 1
# Full sweep, detached so you can close your laptop
./venv/bin/python -m modal run --detach main_modal.py::sweep \
--dataset tabarena \
--student-n "0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9" \
--layers 23 \
--n-repeats 5sweep takes the same options as main.py with dashes instead of underscores. Multi-value options accept either
quoted spaces (--layers "1 2") or commas (--layers 1,2).
./venv/bin/python -m modal app list # find your app id
./venv/bin/python -m modal app logs <app-id> # stream logs
./venv/bin/python download_results.py # pull results back into ./results in case you want to continue locally| Path | Role |
|---|---|
main.py |
Local sweep over datasets and context budgets |
main_modal.py |
Same sweep, fanned out across Modal containers |
create_synthetic_dataset.py |
Unlabeled synthetic query generation |
extract_activations.py |
Forward-hook activation capture for teacher and student |
train_activation_aligner.py |
Aligner training |
evaluate_aligned_student.py |
Inference-time hook injection and metric computation |
train_xgboost.py |
XGBoost baseline |
create_figures.py |
Paper figures, tables, and statistical tests |
layer_sensitivity_table.py |
Layer-sensitivity table (gap closed vs. aligned layer) |
download_results.py |
Sync results from the Modal Volume |
model_utils.py, data_utils.py, xgboost_utils.py |
Model wrappers, TabArena loading, baseline helpers |
cache_utils.py, cli_utils.py, consts.py, utils.py |
Caching, CLI parsing, dataset registry, shared helpers |
Paper: arXiv:2610.06679
@misc{zeldes2026tabalign,
title = {Closing the Context Gap: Activation Alignment for Tabular In-Context Learning},
author = {Zeldes, Yoel},
year = {2026},
eprint = {2610.06679},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2610.06679}
}Yoel Zeldes - LinkedIn
