H3-World: Turning Language Understanding into World Control
Danze Chen, Zeqing Wang, Ziyue Lin, Xingyi Yang, Yeying Jin
Tencent · National University of Singapore · The Hong Kong Polytechnic University
Published September 1, 2026 · on the feed September 2, 2026 · ▲ 47 · arXiv 2609.01560
The abstract
We present H3-World, an efficient framework that turns the 33B MiniMax-H3 video generator into an interactive world model. Our key finding is that, as large video generators become more capable, language is emerging as a natural interface for control. MiniMax-H3, for example, already supports zero-shot control of character behavior and camera motion through natural-language instructions. Building on this, H3-World turns this coarse language interface into precise, temporally grounded world control, without introducing dedicated action modules. Specifically, we represent each action as a structured combination of character and camera instructions, and align them with the corresponding temporal video latents. To make the control temporally precise, we further introduce temporal attention routing, which restricts each instruction to its intended time interval and reduces control leakage across actions. Importantly, H3-World directly reuses the semantic representations learned during large-scale video pretraining and requires only lightweight adaptation. With only 8,000 gameplay samples, 10,000 LoRA optimization steps, and 0.199% trainable parameters, H3-World achieves effective character and camera control while preserving strong generation quality. It also generalizes to unseen scenarios. These results show that the control capabilities emerging in large video generators can be efficiently transformed into interactive world control.
The Brief
2:32A short podcast summary of the paper. Written and voiced by AI.
Transcript
H3-World. Turning Language Understanding into World Control. By Danze Chen and four others, from Tencent, the National University of Singapore, and one other institution. Published on the first of September. Its code repository is public but slight: nine source files.
An interactive world model needs to respond to actions, moment by moment. The question is how to get that responsiveness out of a video generator. The authors' key finding points the way. As large video generators become more capable, language is emerging as a natural interface for control. MiniMax-H3, a video generator with thirty-three billion parameters, already supports zero-shot control of character behaviour and camera motion through natural-language instructions. That interface, though, is coarse. It is not yet precise, and it is not tied to particular moments in time.
H3-World turns that coarse language interface into precise, temporally grounded control of the world, without introducing any dedicated action modules. It reuses the semantic representations MiniMax-H3 already learned during large-scale video pretraining, and needs only lightweight adaptation on top. In other words, the understanding of language that the generator already has is treated as the control channel. The work is in making that channel precise, not in building a new one beside it.
Each action is represented as a structured combination of a character instruction and a camera instruction, and these are aligned with the matching temporal video latents. To make the control precise in time, the framework adds temporal attention routing. It restricts each instruction to its intended time interval, which reduces control leakage from one action into another.
Put together, the two pieces answer the two weaknesses of the raw interface. The structured pairing of character and camera instructions gives each action a definite form, and the routing keeps each instruction confined to the stretch of video where it belongs, so that one command does not bleed into the next.
The adaptation is small. It uses eight thousand gameplay samples, ten thousand LoRA optimization steps, and zero point one nine nine percent of the parameters made trainable. With that, H3-World achieves effective character and camera control while keeping generation quality strong, and it generalizes to scenarios it has not seen. The scale of those numbers is the point: a generator of thirty-three billion parameters is steered with a tiny fraction of them adjusted.
Control here is not bolted on. It is drawn out of abilities that were already emerging in a large video generator, and the authors take the result as showing that those abilities can be turned efficiently into interactive world control.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A video generator can paint a world. Making it obey a player is another matter.
You have a huge pretrained video generator and want to turn it into an interactive world: a player presses keys, and the character and camera respond moment by moment.
The road ahead
- 1 · The problem ~7 min A pretrained video generator makes good-looking video, but it does not by itself respond to a player's key presses moment by moment.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Write each action as text the generator already understands, tie each instruction to its own moment of video, and fence it off from other moments.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Turn keys into character and camera text, bind each to its video latent, mask attention to that latent, then train small LoRA updates on gameplay.
- 6 · The experiment ~6 min You’ll guess: To turn the large video generator into a controllable world, what share of its parameters did H3-World make trainable?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.