DirCast: Live Esports Directing and Commentary with a Single Streaming Video-Language Model

DirCast. Live Esports Directing and Commentary with a Single Streaming Video-Language Model. By Zixian Gao and seven others, from Alaya Lab, the University of Tokyo, and three other institutions. Published on the twenty-second of October. Its code repository is public and real: nineteen source files.

A live esports broadcast needs two people. A director chooses which camera goes on air, and a caster talks over that choice. Both do it every second, with no knowledge of what comes next. Streaming video language models have learned the caster's half, but only over a broadcast a human director has already produced. The director's half, and doing both at once, is left out. Someone still has to decide, second by second, which of many cameras the audience sees, and the commentary has to follow that decision.

The task the authors set is live broadcast production. The model is handed the raw camera of every player and has to direct and commentate at the same time, using only what has already happened. DirCast is a single streaming video language model that does both jobs.

Each second, DirCast scans every camera at low resolution. A hold-or-switch director then decides whether to cut, and if so, to which camera. The chosen camera is read at high resolution before the model speaks. So every camera gets a quick look at low resolution, and only the camera going on air gets the detailed one. The directing decision comes first, and the commentary is spoken over whatever that decision put on screen.

Memory is selective as well. The context keeps what went on air and what was said, and lets the other views expire. What the audience never saw is not carried forward. That bounds the cost of a stream, however long it runs.

Alongside the model, the authors build a dataset of twenty-seven professional Dota 2 matches. Each has a camera for every hero, along with the human director's choices and the casters' commentary.

As a director, DirCast does better than rule-based directors and zero-shot ones, GPT-5 among them. As a commentator, it is compared against GPT-4o under the Proact-VL protocol and wins twenty-nine point two percent of those comparisons. The strongest of eleven streaming commentators that watch the finished human broadcast wins nineteen point four percent.

Streaming commentators had needed a human to cut the feed for them. DirCast makes the cut itself, from every player's camera, and its commentary still wins more often against GPT-4o than commentators handed the finished human broadcast.

DirCast. Live Esports Directing and Commentary with a Single Streaming Video-Language Model. By Zixian Gao and seven others, from Alaya Lab, the University of Tokyo, and three other institutions. Published on the twenty-second of October. This is the long version: the whole paper rather than the abstract.

A live esports broadcast is made by two people working as one. The director watches a wall of camera feeds, one per player, and decides every second which goes on air. The caster talks over whatever the director chose. Neither can wait for the future: when a fight breaks out across the map, the director has a few seconds to cut to it. That is why regional leagues and amateur games are streamed with a fixed camera and no commentary. Streaming video language models such as LiveCC and Proact-VL have learned the caster's job. But every one of them watches a finished broadcast, inheriting the human director's judgment for free. Hand them the raw feeds and they cannot be used.

The task here is live broadcast production. Given the ten hero cameras of a live Dota 2 match and the game events, a model must choose the on-air feed every second and commentate over its own choice, using only the past. The two halves make each other harder. Ten feeds produce more visual tokens than a streaming model can hold for long. Mistakes compound: a director that misses a fight shows the commentator the wrong scene. Training on human broadcasts only ever shows the program a human chose, the classic exposure bias. Human directors hold most shots, so a model imitating them learns never to cut. And a model allowed to stay silent tends to fall into silence and stay there.

A broadcast throws away the cameras the director did not choose, but a Dota 2 replay can be filmed from any point of view. From twenty-seven professional matches the authors rendered one camera per hero, aligned them with the official broadcast, and labelled every second with the feed on air and the casters' words.

DirCast is a single causal video language model, fine-tuned from Qwen3-VL. Each second it reads one chunk laid out in the order a human crew works. First the game clock and that second's events. Then the ten cameras at low resolution, each preceded by a state token carrying its hero's health, recent damage and nearby enemies. Then a director token. After the decision, the chosen camera is appended again at high resolution, followed by a commentary token and the words spoken. The commentator always looks closely at the shot the director picked.

The director has two parts. A cut gate estimates the probability of leaving the current camera. A pointer then scores the other cameras by comparing the director's state with each camera's content and game state. Holding is the default; switching needs the gate to open. Memory works like a human crew's, which remembers the broadcast rather than every feed: low-resolution views expire after eight seconds, while on-air views, events and words stay. Training adds two fixes. In perturbed program rollouts, the on-air feed drifts from the human choice for a few seconds while the targets stay human, so the director learns to cut back and the commentator learns to speak over any shot. Event-anchored targets accept a cut to the killer or victim around a kill. At inference, the cutting and speaking thresholds follow the model's own running rates toward the human crew's.

As a director, DirCast matches the human crew's shot twenty-seven percent of the time, against twenty-five point four for a hand-written action rule. That margin is small. The striking result is the zero-shot models. None of the eight reaches the action rule, and GPT-5 matches only thirteen point five percent. Most cut between twenty and fifty-six times a minute. DirCast cuts six point seven times and covers eighty-nine point one percent of kills, more than the human director. As a commentator, it wins twenty-nine point two percent of comparisons against GPT-4o, against nineteen point four for the best baseline, though every baseline watches the human broadcast.

The comparison that most directly tests the thesis is one model against two. Trained jointly, DirCast does both roles better than single-role copies, with half the memory. To rule out a better program as the cause, both commentators were run over the human broadcast. The joint model still wins, twenty-seven point eight four percent against twenty-five point six seven. Learning to direct made it a better commentator.

The paper has no limitations section, so these come from what its evidence leaves open. Everything rests on four test matches of a single game. The protocol's official judge, GPT-5.1, was unavailable, so GPT-5 judged instead. Exact agreement with the human director sits near twenty-seven percent, only modestly above a rule. Test clips are thirty-six seconds long, so long-stream memory is tested in a separate, smaller experiment. And the data curve is still rising at twenty training matches, which also shows how little data the model has seen.

The idea to carry away is that directing and commentating are one act of attention. The director decides what the audience sees, the caster talks about it, and both depend on what has already been shown and said. Put them in one stream, make holding the default and cutting deliberate, and let the memory be the broadcast itself. A model that learns to choose the shot also learns to talk about it better.