4Director: Controlling Video World Models with Rigid 3D Geometry
Wei Cao, Hao Zhang, Vikram Voleti, Yuqun Wu, Mallikarjun B R, Shimon Vainer, Mark Boss, Yaoyao Liu
Stability AI · University of Illinois Urbana-Champaign
Published October 1, 2026 · on the feed October 2, 2026 · ▲ 36 · arXiv 2610.02160
The abstract
Precise control over camera and object motion is essential for professional video production. Existing methods control objects only coarsely, through image-plane cues that are ambiguous in depth and rotation or through 3D tracks and blobs that lack complete geometry and lose consistency across viewpoint changes. We introduce 4Director, a video world model conditioned on an explicit 4D scene representation: each object is reconstructed once from the input image as a canonical mesh and moved by one prescribed rigid transformation per frame. This representation provides an intuitive 3D control interface and prevents unobserved geometry from being regenerated independently in every frame. We render the controlled scene as a depth video and introduce a Motion Adapter that transforms this geometric scaffold into video while synthesizing view-consistent appearance, illumination, and non-rigid dynamics. For training, we construct RealCOD-Rigid, a new dataset of 20,774 clips annotated with rigid 3D scenes by our automatic pipeline. We further introduce Identity-Gated IoU (IG-IoU), which jointly evaluates adherence to prescribed object motion and preservation of object identity. Experiments demonstrate that 4Director consistently outperforms prior methods in visual quality and in camera and object control.
The Brief
2:21A short podcast summary of the paper. Written and voiced by AI.
Transcript
4Director. Controlling Video World Models with Rigid 3D Geometry. By Wei Cao and seven others, from Stability AI and the University of Illinois Urbana-Champaign. Published on the first of October. Its code repository is a placeholder: a readme, and nothing to run yet.
Professional video production needs precise control over how the camera moves and how the objects in the shot move. Existing methods control objects only coarsely. Some use cues in the image plane, which are ambiguous about depth and rotation: a flat cue cannot say whether an object is turning or moving away. Others use 3D tracks and blobs, which lack complete geometry and lose consistency once the viewpoint changes. Either way, the director asks for a motion and gets an approximation of it, and the parts of an object nobody has seen yet are left for the model to guess anew.
4Director is a video world model conditioned on an explicit 4D scene representation. Each object is reconstructed once, from the input image, as a canonical mesh, and then moved by one prescribed rigid transformation per frame. That gives a user an intuitive 3D interface for control. It also stops geometry the camera has not yet seen from being regenerated independently in every frame, since the object exists once and is only moved.
The controlled scene is rendered as a depth video. A Motion Adapter then turns that geometric scaffold into the final video, synthesizing appearance, illumination and non-rigid dynamics that stay consistent across views. To train it, the authors built RealCOD-Rigid, a new dataset of twenty thousand seven hundred and seventy-four clips, each annotated with a rigid 3D scene by their automatic pipeline. The division of labour is clear. Geometry and motion are fixed by the scaffold, and the adapter is responsible for everything that makes the result look like real footage.
Measuring this kind of control needed a new metric. Identity-Gated IoU jointly scores whether objects follow their prescribed motion and whether they keep their identity while doing it, so an object that moves correctly but turns into something else does not get credit. Experiments show 4Director consistently outperforming prior methods in visual quality, in camera control and in object control.
A depth render of rigid meshes is a plain thing to condition on, and that is its appeal. Motion is specified in 3D, once, where it is unambiguous, and everything the scaffold leaves out is left to the video model to fill in.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A director asks for a turn and gets a guess. Start with what stands in the way.
You are building a video generator for directors. From one input image, they want to say exactly how the camera moves and how each object turns, slides or moves away.
The road ahead
- 1 · The problem ~7 min Directors want exact 3D control of camera and objects from one image, but today's controls are either ambiguous flat cues or incomplete 3D shells.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Rebuild each object once as a complete mesh, move it rigidly in 3D, render a depth video, and let the generator paint the look on top.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Build meshes once, move them rigidly per frame, render a depth video, then let the Motion Adapter turn it into footage.
- 6 · The experiment ~6 min You’ll guess: Against prior methods for controlled video generation, where did 4Director come out ahead?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.