PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Control
Chuhao Chen, Peter Wonka, Chaoyang Wang, Chen Wang, Qiao Feng, Sergey Tulyakov, Lingjie Liu
University of Pennsylvania · Snap Inc. · KAUST
Published September 15, 2026 · on the feed September 16, 2026 · ▲ 10 · arXiv 2609.17521
The abstract
Interactive control for video generation is moving from coarse prompts toward fine-grained, physically meaningful manipulation of dynamic scenes. Yet existing controllable methods either require the full control schedule before generation starts, or use pixel-space signals that dictate object positions rather than physical dynamics. To address these limitations, we propose PhysStream, an autoregressive model for physics-grounded image-to-video synthesis that incorporates structured scene memory---positional maps and object tracking maps derived online from previously generated frames---and supports fine-grained motion control via sparse velocity-increment signals that encode physical quantities, letting the model learn the underlying dynamics. We train our model in two stages: a bidirectional model is first finetuned with motion-control conditioning, then a causal autoregressive model is trained with additional structured scene memory, further improving physical consistency. PhysStream enables interactive, mid-generation control over multi-object tabletop rigid-body scenes---a capability not supported by prior methods---reducing motion distribution distance (FVMD) by 33% and trajectory error by 12% over the strongest baselines on synthetic benchmarks, and is preferred by human evaluators in over 85% of in-the-wild comparisons. Please check our website for more details: https://czzzzh.github.io/PhysStream
The Brief
2:17A short podcast summary of the paper. Written and voiced by AI.
Transcript
PhysStream. Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Control. By Chuhao Chen and six others, from the University of Pennsylvania, Snap, and one other institution. Published on the fifteenth of September. No code repository is linked to it.
Interactive control of video generation is moving from coarse prompts toward fine-grained, physically meaningful manipulation of dynamic scenes. Existing controllable methods fall short in one of two ways. Either they require the full control schedule before generation begins, so nothing can be changed once the video is under way. Or they use pixel-space signals that dictate where objects are, rather than the physical dynamics that should move them there.
PhysStream is an autoregressive model for physics-grounded image-to-video synthesis, built to address both. Control arrives as sparse velocity-increment signals that encode physical quantities. Instead of being told where an object should be, the model is told how its motion changes, and it learns the underlying dynamics that follow. Alongside this, the model keeps a structured scene memory: positional maps and object tracking maps, derived online from the frames it has already generated.
Training happens in two stages. First, a bidirectional model is fine-tuned with motion-control conditioning. Then a causal autoregressive model is trained with the structured scene memory added, which further improves physical consistency. The autoregressive form is what allows control to arrive while generation is running, and the scene memory gives each new frame an explicit record of where objects are and how they have been tracked so far.
The result is interactive, mid-generation control over tabletop scenes with multiple rigid objects, which the authors describe as a capability prior methods did not support. On synthetic benchmarks, PhysStream reduces motion distribution distance, measured as FVMD, by thirty-three percent, and trajectory error by twelve percent, relative to the strongest baselines. In in-the-wild comparisons, human evaluators prefer it more than eighty-five percent of the time.
Two choices work together. Controls expressed as changes in velocity let a user push on the physics while the video is playing out, and a memory built from the model's own output keeps track of where everything is as the scene unfolds. One without the other would leave either the control or the consistency behind.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a video that will not let you touch it once it starts.
You are building a video generator where a user nudges objects on a tabletop: push the dish, watch it slide, then push again while it moves.
The road ahead
- 1 · The problem ~7 min You want users to nudge tabletop objects while a video is being made, but current control methods either fix the plan up front or just dictate positions.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Generate video frame by frame, let users add sparse velocity changes at any moment, and give each frame a running memory of the scene.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Finetune a whole-clip model on velocity control, convert it to a causal model with scene memory, then generate while accepting pushes.
- 6 · The experiment ~6 min You’ll guess: On synthetic benchmarks, by how much did PhysStream reduce trajectory error compared with the strongest baselines?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.