Tabular Foundation Models Beat Tuned GBDTs — and They Fit on a Desk

Credit scoring, churn models, predictive maintenance, clinical risk scores — most high-value predictions in industry run on rows in a spreadsheet. For twenty years the answer has been gradient-boosted decision trees (GBDTs): XGBoost, LightGBM, CatBoost. In 2026 that default is over. On the standard benchmarks, tabular foundation models now beat tuned tree ensembles with statistical significance, and the best ones fit on a single GPU — some on a laptop CPU. This article traces how that happened, what the independent evidence actually says, and when you should still reach for LightGBM.

The shift, in one number

The peer-reviewed milestone came in January 2025, when TabPFN appeared in Nature. In 2.8 seconds, a single forward pass of the pretrained transformer outperformed an ensemble of the strongest baselines tuned for four hours — a 5,140× speedup in classification, 3,000× in regression, on datasets up to 10,000 rows and 500 features. Pretraining took two weeks on eight RTX 2080s — once, for every future table.

The field consolidated around TabArena, the curated benchmark (51 datasets selected from 1,053 candidates, NeurIPS 2025 Datasets & Benchmarks track) that compares trees, deep learning and foundation models on identical splits. Since then the leaderboard has moved fast:

Release	Date	Claim
TabPFN v2	2024 / Nature 2025	First tabular FM to beat tuned GBDTs on standard benchmarks, ≤10k rows
TabPFN-2.5	Nov 2025	100k rows, 2,000 features; matches AutoGluon 1.4 (4h-tuned) in a forward pass
TabDPT	NeurIPS 2025	ICL + self-supervised column masking on real data; 78M params
TabPFN-3	May 2026	1M rows × 200 features; 160 classes natively; up to 20× faster than 2.5
TabFM	30 Jun 2026	Google Research hybrid; won 15/19 datasets in an independent benchmark
TabPFN-3.5	Sep 2026	First overall on TabArena and BeyondArena (142 datasets)

TabPFN-3.5 (September 2026) currently ranks first on both benchmarks, finishing ~150 Elo ahead of the previous leader on BeyondArena's grouped, temporal, high-cardinality and text-rich data. That a vendor's own report tops the charts is expected; what matters is that the independence of the trend holds up, which we'll come back to.

What a tabular foundation model actually is

The conceptual break is easy to miss: the model is not trained on your dataset at all. A tabular foundation model is a transformer pretrained once on millions of synthetic tables generated from a structural causal model (SCM) prior. At inference, your training rows are fed in as context — in-context learning — and predictions come out of a single forward pass. No gradient training, no hyperparameter search, no validation split. You can even get calibrated uncertainty intervals for free, because the model is approximating Bayesian inference over its prior.

Different lines make different bets on the architecture:

• TabPFN (Prior Labs) — the ICL transformer line; attention-based retrieval decoder treats class prediction as soft nearest-neighbour retrieval over the in-context set, so class count is no longer a fixed hyperparameter (160 classes natively in v3).
• TabDPT (Yandex) — adds retrieval and self-supervised column-masking pretraining on real data; deliberately small (78M params, 16 layers).
• TabICLv2 / Mitra / LimiX — academic ICL variants; TabICLv2 is the cheap-and-cheerful entry (~3 s per fold).
• TabFM (Google Research) — an expensive hybrid: row-and-column attention over raw cells, row compression, then a TabPFN-style ICL transformer, pretrained on hundreds of millions of synthetic tables.

One caution: LLM-based "tabular" methods (Tabula-8B and friends) lose badly — tokenising numbers as text saturates context windows. The dedicated small transformers won.

Making a 1M-row forward pass practical

The original TabPFN stopped at a thousand rows; the reason v3 can now claim a million lives in three engineering decisions worth stealing:

• Row-chunking. The distribution embedder computes an inducing-vector summary once over the full training set, then streams rows through in fixed-size chunks that reuse the cached summary as attention keys/values. Peak memory stops depending on dataset size.
• Multi-query attention. Test-row queries attend to train-row keys with a single KV head, cutting the KV cache to ~7GB even for a million training rows on one GPU.
• Test-time compute ("Thinking mode"). The API-only Plus variants spend extra effort at fit time — no LLMs, real data or internet involved — and on TabArena that buys 200+ Elo over any non-TabPFN model and beats AutoGluon 1.5 extreme (tuned for four hours) by over 100 Elo at less than a tenth of the runtime.

The local library kept its scikit-learn interface, so onboarding is a few lines:

pip install tabpfn   # torch >= 2.5

from tabpfn import TabPFNClassifier

clf = TabPFNClassifier(n_estimators=8, random_state=0)  # deterministic since v3
clf.fit(X_train, y_train)

prob = clf.predict_proba(X_test)[:, 1]                       # point prediction
interval = clf.predict_proba_with_uncertainty(X_test)        # calibrated interval

SHAP values via shapiq run up to 120× faster with the KV cache.

The independent reality check

Vendor Elo tables deserve scepticism, so an independent 19-dataset run (5-fold stratified CV, identical splits, default hyperparameters on one RunPod with two B200s) is the one to judge by. Its verdict, backed by a Friedman test (χ² = 72.8, p < 10⁻¹²) with Nemenyi post-hoc:

• Foundation models beat GBDTs on accuracy, with statistical significance, across small, medium and large tables. No boosted tree won a single dataset; the best classical model was LightGBM, 538 Elo behind the leader.
• TabFM won 15 of 19 datasets but is not the obvious pick — it costs roughly 40× TabPFN-3's compute, takes 20+ minutes to compile on first run, caps at 500 features / 10 classes, and needs datacentre GPUs. Its lead over TabPFN-3 (rank gap 1.55) and TabICLv2 (1.34) is below the critical difference, so which FM tops your table is not yet settled — the "FM over GBDTs" claim is.
• The encoding caveat matters. FMs beat GBDTs only when categoricals are passed natively as designed. Forcing one-hot encoding cripples in-context models, so a bad pipeline choice can flip the result — one reason XGBoost's weaker native categorical support (10,894 vs 4,367 RMSE on one high-cardinality regression) is worth checking before you trust it.

Read it honestly: tuned tree ensembles remain competitive, and an FM with a tuned GBDT fallback is the defensible production posture, not "delete your XGBoost."

When GBDTs still win

The envelope is real but bounded. Stay with trees when:

• Datasets exceed ~1M rows — the validated SOTA envelope for TabPFN-3 tops out at 1M rows × 200 features (or 100k × 2,000); beyond it, trees still scale.
• Streaming or online learning — GBDTs retrain incrementally; the FM's in-context paradigm assumes you can refit the context.
• Ultra-low latency at scale — a distilled MLP or tree ensemble (Prior Labs ships a distillation engine) is still orders of magnitude cheaper per prediction than any attention pass.
• Authorised audit tooling — your compliance workflow already embeds tree-based explainability; swapping the base model means re-validating it.

Why this is a sovereignty story

The polite headline is accuracy. The strategic one is portability. Tabular foundation models are the first "foundation model" category that fits on a desk: TabDPT is 78M parameters, and TabPFN-3's inference runs in ~0.2s per million rows on a single H100 — a 140W-class workstation serves production scoring. Just as important, the prior is public synthetic data — your rows never leave the network to train anything; adaptation happens in context, inside your process, on your hardware. For a regulated organisation — bank, health insurer, public administration — that combination (no training-data export, on-prem/VPC deployment options) is precisely the digital-sovereignty profile that LLMs struggle to offer.

Two honest asterisks. TabPFN-3's base weights ship under the TABPFN-3.0 Licence ("permissive for research and internal evaluation"); the strongest Plus/Thinking variants are API/enterprise, including on-prem and VPC on AWS SageMaker and Azure AI Foundry — so check the licence before you bake it into a product. And the accuracy-topping TabFM is datacentre-only: "small" applies to TabPFN-3 and TabICLv2, not the whole field.

What to do on Monday

1. Baseline your next project with an FM in a forward pass. pip install tabpfn, feed the table raw (categoricals native, no one-hot), compare against your tuned tree ensemble on your own validation splits.
2. Treat the benchmarks as directional. Curated 51-dataset suites are not your data — measure on your own rows, especially high-cardinality categoricals.
3. Keep the tree ensemble as fallback. An FM plus a tuned GBDT, voted or pick-by-regime, dominates either alone — and if your data is streaming or past a million rows, the trees are the answer today.
4. Audit with shapiq and keep the calibrated intervals in your reporting — uncertainty is a feature your logistic regression never had.
5. If sovereignty is on the requirements list, you now have a genuine option: state-of-the-art tabular scoring with no data egress, on hardware you already own.

The GBDT default lasted twenty years because it was simple, fast and good enough. The tabular foundation model doesn't merely match it — it wins on accuracy and removes the tuning ritual, in a package small enough to run where the data lives.