World Embedding Benchmark

World Embedding Benchmark. By Yiqi Liu and nine others. Published on the second of October. Its code repository is public and real: about forty source files.

World models and video generators are increasingly judged on physical fidelity, on whether what they produce behaves the way the physical world does. Much less is understood about the representations underneath: how a video embedding encodes physical information in the first place. There has been no direct way to ask an embedding what physics it holds.

The World Embedding Benchmark is built for that question. It contains eight thousand controlled simulation cases drawn from eighty families, spanning fluid mechanics, solid mechanics, dynamics, and optics and electromagnetism. Every case pairs a rendered video with physical annotations derived from the simulation, so the ground truth is known exactly.

Three complementary tasks run on top of those cases: retrieval between text and video, regression of physical properties, and multiple-choice classification of pairs of video and description. The tasks are chosen to separate two things that are easy to conflate. One is cross-modal physical alignment, whether a video and the right description land near each other. The other is recoverability, whether quantitative physical information can be read back out of the embedding at all.

Pre-trained omnimodal embedding models show weak retrieval, and close to chance pair classification within a family. Yet lightweight probes recover useful physical information from those same frozen video embeddings. The information is present even where alignment is poor. Continual contrastive training on physics-specific video and text pairs improves retrieval and pair classification, but degrades property regression, so alignment and quantitative recoverability trade off against each other.

Finally, the embeddings are put to use. They retrieve reference videos for retrieval-augmented generation with MiniMax-H3, and the retrieved references improve the physical fidelity of the generated videos, with stronger retrieval models yielding larger gains in these experiments.

Two lessons are drawn. Physical alignment and property recoverability need to be evaluated jointly, because improving one can cost the other, and a single score would hide that. And physical representations are not only something to diagnose: they can feed back into video generation and make it more faithful.