LIFT: Layout-In-Future Video Generation under Large Viewpoint Change via On-Policy Self-Distillation
Shengxiang Ji, Boyang Wang, Haiyang Xu, Bingnan Li, Yucheng Mao, Zeyuan Chen, Xiaojun Shan, Xiang Zhang and 4 others
UC San Diego · University of Virginia · Meta · Amazon · Lambda
Published September 29, 2026 · on the feed September 30, 2026 · ▲ 6 · arXiv 2609.38146
The abstract
We introduce LIFT, a unified image-to-video generation framework that complements camera control with Layout-In-FuTure control, enabling users to specify what should appear in a future view and where it should appear. This addresses a practical need in controllable video generation: given an initial image, users often care not only about how the camera moves, but also about what the scene should look like at key future moments, especially the final frame. Existing camera controls specify viewpoint trajectories, while text prompts provide only coarse semantic guidance; neither precisely determines the content and spatial layout of future views. This limitation becomes particularly pronounced under large viewpoint changes, where the camera reveals regions that are not visible in the first frame. LIFT therefore uses the last-frame layout as an explicit control signal for the desired future scene. Since learning from such sparse layout guidance is substantially more challenging than conditioning on dense per-frame layouts, we introduce on-policy self-distillation (OPSD) to transfer the control capability of a dense-layout teacher to a last-frame-layout student. We further curate LIFT-Vista, a dataset featuring large viewpoint changes with camera and temporally consistent layout annotations. Experiments show that LIFT improves video quality, future-layout controllability, and camera controllability over other methods.
The Brief
2:17A short podcast summary of the paper. Written and voiced by AI.
Transcript
LIFT. Layout-In-Future Video Generation under Large Viewpoint Change via On-Policy Self-Distillation. By Shengxiang Ji and eleven others, from UC San Diego, the University of Virginia, and three other institutions. Published on the twenty-ninth of September. Its code repository is public and real: about forty source files.
Start from a single image and ask for a video. Users often care about more than how the camera moves. They care what the scene should look like at key moments ahead, and above all in the final frame. Existing controls do not reach that far. A camera control specifies the viewpoint's trajectory. A text prompt gives only coarse semantic guidance. Neither pins down what a future view should contain, or where in the frame it should sit. The gap widens under large changes of viewpoint, when the camera swings round to reveal regions that were not visible in the first frame at all.
LIFT is a unified image-to-video framework that adds a second kind of control beside the camera: Layout-In-FuTure control. The user specifies what should appear in a future view, and where. Concretely, the layout of the last frame becomes an explicit control signal for the scene the video should arrive at.
Learning from that signal is hard, because it is sparse. Conditioning on a dense layout for every frame is substantially easier than conditioning on one layout at the very end. LIFT bridges the two with on-policy self-distillation, or OPSD. A teacher that sees dense layouts already has the control capability, and OPSD transfers that capability to a student that only gets the last-frame layout. Training also needs suitable data, so the authors curate LIFT-Vista, a dataset built around large viewpoint changes, annotated with camera information and with layouts that stay consistent over time.
In experiments, LIFT improves on other methods along three axes: the quality of the video, how well the future layout can be controlled, and how well the camera can be controlled. The abstract reports these as improvements without giving figures.
Picture directing a shot by sketching only its last frame, roughly where the door should be and where the table should stand, and letting the camera find its way there. That is the control LIFT is after, and the teacher that saw every frame is what lets a student with only the ending learn to deliver it.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
The camera can go anywhere. Saying what it should find there is the hard part.
You turn a single photo into a video while the camera swings far round a room. You can tell the camera which path to take, and a text prompt can say roughly what is there.
The road ahead
- 1 · The problem ~6 min Camera paths and text prompts cannot say what a newly revealed part of the scene should contain, or where it should sit.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Let the user set the last frame's layout, and pass a dense-layout teacher's control to a student that sees only that layout.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Curate suitable videos, train a dense-layout teacher, distil its control into a last-frame-layout student, then generate from image, camera path and final layout.
- 6 · The experiment ~6 min You’ll guess: Adding last-frame layout control could trade off other qualities. Against other methods, on which of video quality, future-layout control and camera control did LIFT improve?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.