ALICE: In-context, Zero-shot, Mutual Information Estimation

ALICE. In-context, Zero-shot, Mutual Information Estimation. By Giulio Franzese and two others, from EURECOM. Published on the twenty-ninth of September. Its code repository is public and real: sixteen source files.

Estimating mutual information from samples, how much knowing one variable tells you about another, is a central objective across many sciences. Modern neural estimators do it accurately when data is plentiful. They have three limitations, though. They fall short when data is scarce. Each one must be fit again for every distribution under study. And they are tied to particular data types. In many applications, training per distribution is impractical and samples are few, and those constraints keep the estimators from being adopted. The researcher with a small dataset and a new kind of measurement is precisely the one these tools serve worst.

ALICE is a foundation model that removes per-distribution training while staying competitive in accuracy. The key move is what it estimates. Rather than mutual information directly, ALICE works as an in-context estimator of rectified-flow velocity fields. Shown samples from a distribution it has never seen, it estimates that distribution's velocity field with no explicit training at all. The samples play the role a prompt plays for a language model: they are read, not trained on.

ALICE is trained only on synthetic data, drawn from a broad family of distributions. At use time it is conditioned on the samples in front of it. Mutual information then comes out of a fixed identity: integrate the squared difference between the joint velocity field and the conditional one. The learned part of the system estimates fields, and a formula that never changes turns those fields into the number the scientist wanted.

Validation runs on a standard, challenging benchmark. Beyond that, ALICE is applied in three domains whose data it has never seen: biology, genetics and neuroscience. The authors report that, for the first time, a single model closes the gap with neural estimators that were trained separately for each distribution. It also natively handles different data dimensionality and different numbers of samples. That answers two of the three limits from the start directly: no per-distribution fitting, and no tie to one data shape. The small-sample case is not reported separately; the headline is that one model draws level with estimators trained one distribution at a time.

The accounting is what changes. A neural estimator used to cost a training run per question asked. ALICE pays that cost once, on synthetic distributions, and afterwards each new dataset is just context, which is what makes zero-shot mutual information analysis across scientific domains possible.