Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models

Latent-Foresight. End-to-End Learning Predictable Representations for Latent World Models. By Efstathios Karypidis and two others, from Archimedes at the Athena Research Center, Valeo AI, and three other institutions. Published on the first of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Predicting how a scene will evolve is a fundamental capability for world modeling. Recent work has shown that operating in the feature space of Vision Foundation Models yields semantically rich representations, which support diverse future scene understanding tasks. The existing approaches, though, rely on two-stage pipelines. First, the VFM features are compressed, either with a fixed dimensionality reduction such as PCA, or with an independently trained autoencoder. Then a separate predictor is trained on top of the resulting latent space, which is frozen by then.

That decoupling is the difficulty. The representation is learned without any regard to the prediction that will later be made from it. Approaches that skip compression and apply predictors directly on raw VFM features share the weakness. Neither route offers any guarantee that the latent space is structured for predictable dynamics.

Latent-Foresight joins the two stages. It is an end-to-end framework that jointly learns a latent tokenizer and a flow-based generative dynamics model. Because both are trained together, the representation is explicitly shaped to support temporal predictability, instead of being fixed first and predicted from afterwards. The tokenizer decides what the latent space looks like. The dynamics model decides how well the future can be forecast within it. Training them as one system lets the second have a say in the first.

Training the two jointly has to stay stable, and the authors introduce several key design choices for that purpose. These choices prevent latent collapse, and align reconstruction with the generative objectives. The abstract does not detail them. One practical consequence is stated, though: separate training stages are eliminated, including during high-resolution adaptation. Where a two-stage pipeline would train a compressor and then a predictor, and repeat that sequence when moving to higher resolution, this framework trains one system throughout.

Extensive experiments report two results. The approach learns latent representations that are more temporally coherent. And it consistently outperforms two-stage baselines across multiple future scene understanding tasks and across prediction horizons. The implementation code and model weights are released.

Should a representation be learned first and predicted from later, or learned for the purpose of being predicted? Latent-Foresight takes the second view, and its comparison against two-stage baselines is the test of whether shaping the latent space for its future pays off in the forecasts.