Causal Foundation Models

Causal Foundation Models. By Christopher Stith and two others, from Layer 6 A I and T D Bank Group. Published on the second of September. Its code repository is public and real: thirteen source files.

Causal inference asks what a treatment or an intervention actually does, estimated from data. Traditionally every new question gets its own pipeline. Someone first proposes a causal mechanism, then selects an estimator that is compatible with it, and only then trains that estimator. Each problem is built by hand, from the ground up.

Much of machine learning has moved in the opposite direction. Across very different settings and modalities, the dominant paradigm is now the foundation model: a network pretrained once, at scale, and then applied to new tasks without any fine-tuning. Causal foundation models, or CFMs, carry that paradigm over to causal inference. A CFM is a pretrained neural network that estimates a causal quantity, the average treatment effect being the standard example, on a dataset it has never seen. It does this through in-context learning, so no model update is required. The new dataset is the input, and the estimate is the output.

What the authors offer is a practical introduction to this emerging area rather than a new model. The route runs in order. First comes the background needed from causal inference, then the background needed from machine learning, and only then the models themselves. Example code and Jupyter notebooks run alongside the text throughout, so each idea arrives with something that can be run.

The shift being introduced is easy to state. Where the old workflow meant designing a pipeline for each question, a causal foundation model is meant to be trained once and then asked. The introduction is written for someone who wants to understand that shift and try it, starting from the foundations of both fields.