Closing the Context Gap: Activation Alignment for Tabular In-Context Learning

Closing the Context Gap. Activation Alignment for Tabular In-Context Learning. By Yoel Zeldes, working independently. Published on the fifth of October. Its code repository is public and real: about twenty source files.

Tabular foundation models perform in-context learning. They condition each prediction on labeled training examples supplied as context. Traditional models separate training from inference. These do not, and must process every training example in every forward pass, which makes each prediction expensive. The more data you have, the more every single prediction costs. The obvious economy is to show fewer examples. That does cut the cost, but it substantially degrades performance, so the saving is paid for in accuracy.

Activation alignment declines that trade. Instead of discarding the context, it uses the full context to teach the model how to behave when it sees only a subset. The setup is a teacher and a student. The teacher is the model with all the data in context. The student is data-constrained, working from only part of it. The aim is for the student, seeing little, to act more like the teacher, which sees everything.

Between them sits a lightweight linear transformation, the aligner. It is trained to map the student's intermediate activations toward the teacher's. The intervention happens inside the model, on its internal activations, rather than on its outputs. The training uses synthetic, unlabeled data, so no extra labels are needed to fit it. The aligner is also cheap: it needs no GPU, and it converges in seconds to minutes on commodity hardware.

The evaluation covers thirty-eight classification datasets from the TabArena benchmark, using the two leading tabular foundation models, TabPFN-3 and TabFM. Across all context budgets, the aligned student gives broad, statistically significant improvements over the unaligned baseline, and does so for both models, so the gain is not confined to one architecture or one budget. In low-data regimes, alignment recovers nearly half of the teacher's predictive advantage over the unaligned student.

Before, the choice had two settings. A full context was accurate and slow; a small one was fast and worse. Alignment offers a point between them: the inference speed of a compact context, with a significant fraction of the gap to the full-context teacher closed. The price of that point is a linear map fitted on synthetic data in minutes, on ordinary hardware, without a GPU.