Predictive Closure: A Set-Level Characterization of JEPA's Feature Selection

Predictive Closure. A Set-Level Characterization of JEPA's Feature Selection. By Mosh Levy and two others, from Bar-Ilan University, the University of California, Berkeley, and one other institution. Published on the nineteenth of October. Its code repository is public and real: about thirty source files.

In a joint-embedding predictive architecture, or JEPA, an encoder and a predictor are trained together. The goal is a representation that minimizes the error of predicting the representation of the next observation. Since the representation of the current observation and of the next one are both learned, it is not obvious which features will end up in them. Nothing in the objective names the features to keep. The encoder is free to choose, and what it chooses is whatever makes the prediction task go well. Earlier characterizations answered in terms of each feature's own predictability. That answer carries a worry: a feature that cannot itself be predicted, a traffic light for example, might never be encoded.

The paper's claim is that this looks at features one at a time when the prediction error does not. Errors also depend on how features relate to each other. The colour of the light helps predict whether the traffic will stop. The authors propose a characterization called predictive closure that accounts for such relationships. From a reformulation of the JEPA training objective, they show that which features get selected depends on properties of sets of features, not of features in isolation. A feature is judged by the company it keeps. The traffic light may be hard to forecast on its own, yet it earns its place if it makes the rest of the scene easier to forecast.

The characterization is tested in controlled experiments, and two results come out of them. Unpredictable features can become better represented when they help predict other features. And changing the predictive relationships of one feature can affect how well other features are represented. The authors also show consequences for steering what a JEPA selects. Targeting one feature with an auxiliary objective, or handing it to the predictor as side information, can strengthen the representation of a different feature, namely one whose future values the targeted feature helps predict. Steering, in other words, can work indirectly, through the relationships between features.

The new view does not overturn the old one everywhere. When features evolve independently of one another, predictive closure agrees with the earlier, individual-predictability account. The two part ways only when features are related, and that is where the new predictions live, such as a traffic light being retained despite being unpredictable on its own. The empirical support described comes from controlled experiments, designed to test exactly these set-level predictions.

Who gains from this is anyone trying to decide, or to influence, what a JEPA encoder keeps. The account moves the question from a single feature to a set of related ones, and with it the lever: to strengthen one feature, the place to push may be a neighbour that helps predict it. What the abstract leaves for later is how far that set-level picture carries beyond the controlled settings where it was tested.