PhysBrain 1.5: From Vision-Language Models to Physical Foundation Models
DeepCybo Team, Yu Bin, Haipeng Cao, Zheng Chang, Kai Chen, Youning Chen, Kailin Deng, Yichao Du and 46 others
Published September 14, 2026 · on the feed September 15, 2026 · ▲ 172 · arXiv 2609.14973
The abstract
We present PhysBrain 1.5, a unified model for understanding physical environments, generating actions, and predicting future states. Motivated by the physical loop of observation, interaction, and environmental change, we bring these capabilities into a common learning framework. Starting from a general vision--language model, we encode language responses, end-effector motion, and dense visual targets as discrete sequences and jointly optimize them with autoregressive next-token prediction. Pre-training draws its embodied supervision entirely from human interaction videos, using task-centered episodes to pair semantic and spatial context with recovered motion and subsequent observations. We then adapt the model through supervised fine-tuning on a mixture of human demonstrations, robot trajectories, and simulated experience. Across 28 embodied understanding benchmarks, our 8B model achieves an average score of 72.5, setting a new open-source state of the art and performing on par with leading proprietary models such as GPT-6-Astra and Gemini 3.6 Flash. It achieves the best open-source results on 14 benchmarks while retaining general multimodal capabilities. Beyond these understanding evaluations, qualitative examples show the model's ability to produce end-effector trajectories and predict future scenes through spatially aligned RGB, depth, and robot-mask outputs.
The Brief
2:35A short podcast summary of the paper. Written and voiced by AI.
Transcript
PhysBrain 1.5. From Vision-Language Models to Physical Foundation Models. By DeepCybo Team and fifty-three others. Published on the fourteenth of September. Its code repository is a placeholder: a readme, and nothing to run yet.
The authors start from a physical loop: observation, interaction, and the environmental change that follows. Understanding a physical environment, generating actions in it, and predicting its future states are three capabilities along that loop. PhysBrain 1.5 is motivated by bringing all three into one common learning framework rather than treating them apart. On that loop, what is observed shapes the action, and the action shapes what is observed next, so the three capabilities are tied together.
PhysBrain 1.5 is a unified model for understanding physical environments, generating actions, and predicting future states. It starts from a general vision-language model. Language responses, end-effector motion, and dense visual targets are all encoded as discrete sequences, and all three are optimized jointly with autoregressive next-token prediction. Words, motion and images become one kind of output, predicted the same way.
Pre-training draws its embodied supervision entirely from human interaction videos. Task-centered episodes pair semantic and spatial context with the motion recovered from the video and with the observations that follow it. The model is then adapted through supervised fine-tuning on a mixture of human demonstrations, robot trajectories, and simulated experience. Robot data arrives only at fine-tuning. What the model learns about motion during pre- training comes from watching people.
Across twenty-eight embodied understanding benchmarks, the eight billion parameter model achieves an average score of seventy-two point five. That sets a new open-source state of the art, and performs on par with leading proprietary models such as GPT-6-Astra and Gemini 3.6 Flash. It achieves the best open-source results on fourteen of those benchmarks, while retaining general multimodal capabilities. Parity with proprietary models is claimed on the understanding benchmarks.
Beyond the understanding evaluations, qualitative examples show the model producing end-effector trajectories and predicting future scenes, through spatially aligned RGB, depth and robot-mask outputs. Those outputs are shown by example rather than scored. Action generation and prediction are demonstrated rather than benchmarked.
The title names the direction of travel: from vision-language models to physical foundation models. Here that means a vision-language model taught, first by watching human interaction and then by fine-tuning on robot and simulated data, to answer in motion and in future frames as well as in words.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Seeing, acting and predicting are one loop. The wall is that they speak different languages.
You want one model that can understand a scene, decide how a robot hand should move, and picture what the scene will look like next. These are steps of one loop: observe, act, watch the world change.
The road ahead
- 1 · The problem ~7 min One model should understand a scene, move a robot hand and picture what comes next, but those three jobs produce very different kinds of output.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Turn words, hand motion and future images into tokens of one vocabulary, train them all with next-token prediction, and pre-train on videos of people.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Take a vision-language model, add action and visual tokens, pre-train on human video episodes, then fine-tune on human, robot and simulated data.
- 6 · The experiment ~6 min You’ll guess: Across embodied understanding benchmarks, how did the open 8B model compare with leading proprietary models such as Gemini 3.6 Flash?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.