StreamDuo: A Live Commentator and a Research Agent for Unseen Games

StreamDuo. A Live Commentator and a Research Agent for Unseen Games. By Zixian Gao and seven others, from Alaya Lab, the University of Tokyo, and three other institutions. Published on the thirty-first of October.

Streaming video language models can now commentate live gameplay, but they are trained and tested on the same games. The authors study the harder setting of cross-game streaming commentary, where a model trained on one game commentates games it has never seen. What they find is a split. Current commentators carry their timing across to a new game, but not their knowledge of its world. They know when to speak, and no longer know what they are looking at.

StreamDuo answers with a pair. A trainable commentator works alongside a frozen research agent, and that agent reads the wiki of whatever game is being played. When the commentator sees something it cannot name, it sends the current frames over. The research agent replies with a few wiki entries two seconds later, and the stream does not pause while it does.

Only the commentator is trained, on a single game, in two stages. The first uses information-gain labels, which teach it to ask only where an answer would help it say what the human caster said. The second is on-policy self-distillation from a copy of itself that is shown the ideal answer, and it teaches the commentator to make use of answers that are noisy.

Training is on Baldur's Gate 3, and testing is on ten games it never saw. Compared with Proact-VL trained on the same data, StreamDuo raises entity recall from zero point zero eight two to zero point one one six. Source-game leakage drops from nine point nine percent to one point zero percent.

Among streaming models, StreamDuo also has the best timing and the highest judged liveliness and overall quality. Read against the opening diagnosis, the results cover both halves of it. Entity recall speaks to knowledge of the game world, the half that had failed to transfer, and it rises. Leakage from the source game, Baldur's Gate 3, falls to about a tenth of where it was. And timing, the half that already transferred, comes out best among the streaming models, with a research agent in the loop whose answers arrive two seconds after the question.

The division of labour mirrors the diagnosis. Timing already transferred, so the commentator keeps that job. Game knowledge did not, so it is handed to an agent that looks it up live, and the commentator only has to learn when to ask and how to use an imperfect reply.

StreamDuo. A Live Commentator and a Research Agent for Unseen Games. By Zixian Gao and seven others, from Alaya Lab, the University of Tokyo, and three other institutions. Published on the thirty-first of October.

Streaming video LLMs can now commentate live gameplay, but they are trained and tested on the same games. This work asks what happens when a commentator trained on one game is dropped into games it has never seen. The answer is lopsided. Current commentators carry over their timing, knowing when to speak, but not their knowledge of the game world. They can tell that something is happening without being able to name it.

StreamDuo splits the job in two. A trainable commentator is paired with a frozen research agent that reads the wiki of whatever game is being played. When the commentator sees something it cannot name, it sends the current frames to the agent. About two seconds later, the agent returns a few wiki entries, and the stream keeps going the whole time. The commentator does not have to know every game. It has to know when to ask, and what to do with the reply.

Only the commentator is trained, on a single game, in two stages. The first teaches it when to ask. Information-gain labels mark the moments where an answer would help it say what the human caster actually said, so it learns to query only there. The second stage teaches it to cope with answers that are noisy. It uses on-policy self-distillation from a copy of the commentator that sees the ideal answer, so the model learns to make good use of the imperfect answers it really receives.

Training uses Baldur's Gate 3, and testing covers ten games the model has never seen. Against Proact-VL trained on the same data, StreamDuo raises entity recall from zero point zero eight two to zero point one one six. It also cuts source-game leakage, the tendency to drag in things from the training game, from nine point nine percent to one percent. Among streaming models, it achieves the best timing, along with the highest judged liveliness and overall quality.

The design keeps two things apart that a single model would have to blend: the skill of commentating, which transfers, and knowledge of a particular world, which does not. The commentator learns the first from one game, and borrows the second, on demand, from a wiki for each new one.

StreamDuo. A Live Commentator and a Research Agent for Unseen Games. By Zixian Gao and seven others, from Alaya Lab, the University of Tokyo, and three other institutions. Published on the thirty-first of October. This is the long version: the whole paper rather than the abstract.

A good game caster tells the audience who just appeared, what item was picked up, and where the fight is. Streaming video models can now decide every second whether to speak, but they are trained and tested on the same games, and the names that make commentary informative change from one game to the next. The authors retrain a strong commentator, Proact-VL, on Baldur's Gate 3 alone and point it at ten games it has never seen. It speaks at sensible moments but rarely names what is on screen, and when it does, it often reaches back into its training game, calling a character in Tears of the Kingdom Karlach. The paper's one-line diagnosis: timing transfers across games, knowledge does not.

First, the base model. Each second is one turn of a video language model, ending with a special flag token. A small head reads that token's hidden state and outputs the probability of speaking now. When to speak depends on what happens on screen, a fight starting or a menu opening, and that carries over to a new game. What to say about the entities on screen does not.

Second, why ordinary retrieval does not fix it. Knowledge gathered before the stream is mostly irrelevant to the current second. Retrieving every second floods the context. And a text query is a trap: a model that cannot recognise a character cannot describe it well enough to look it up.

StreamDuo splits the job between two agents. The commentator watches every second and is the only part that is trained. The research agent is a frozen vision language model with a keyword search over the current game's wiki, and sees nothing unless asked. A second head on the same flag token estimates whether an answer is needed, and when it crosses a threshold the commentator sends the current frames, not a text question. The research agent looks at them, proposes search keys, searches, checks the candidates against the frames again, and returns up to three short entries. That answer lands in the commentator's history two seconds later while the stream keeps going, so the commentator has to ask before it needs the answer.

Training uses the source game only. Its names are renamed at random within each clip, so none can be memorised.

Stage one teaches when to ask. The obvious label, ask just before the human names something, ignores whether the model already knows. Instead, a frozen copy of the commentator scores the human's line twice: once with an ideal answer containing the right entity in its history, once without. The difference in log probability is the information gain. Only where that gain exceeds one nat is the moment two seconds earlier labelled an ask. Lines the model can already produce, and tactical remarks no answer could help, get no label. Training answers also mix the right entry with distractors.

Stage two teaches how to use noisy answers, through on-policy self-distillation. The student is the commentator, seeing a realistic noisy answer. The teacher is the same model with its low-rank adapter switched off, seeing the ideal answer instead. It knows more only because it is told which entry is right. Because a fine-tuned model often barely reacts to an answer, the teacher's predictions are sharpened by contrasting them with and without the answer, pushing up tokens that become likely only because of the answer, such as the entity's name. The student samples its own lines, and the loss is the reverse Kullback-Leibler divergence from student to teacher over those samples. Reverse matters: it makes the student commit to what the teacher finds likely, rather than spread probability over every name in the answer.

Tested on two thousand four hundred and seventy clips from ten unseen games, StreamDuo lifts entity recall over the retrained Proact-VL from zero point zero eight two to zero point one one six, while responding slightly faster and scoring higher with the language model judge.

In the ablation, renaming plus dossier leaves recall flat, learning to ask lifts it modestly, and Stage two gives the largest gain. Replacing the information-gain labels with naive ones, ask whenever the human names an entity, makes every metric drop. At test time, asking every three seconds nearly doubles recall, but many added names do not fit the moment, so precision and judge scores fall. The learned policy asks only two hundred and sixty-four times over the whole test set, about once every six minutes, and gives the best balance.

The most telling comparison is the Stage two loss. Swap reverse for forward divergence and recall goes up further, but the student names an entity wherever the teacher gives it some probability. Source-game leakage jumps to nineteen point three percent, the highest of any model. And swapping the research agent from the smallest Qwen3-VL to larger ones changes recall little. The bottleneck is how the commentator uses entries, not who fetches them.

The paper has no limitations section, so these are drawn from what it reports. There is one training game, and no error bars or repeated runs. Quality is scored by one judge model, Qwen3-30B, and the authors note that one judge metric favours near-silent models. Even the best recall leaves most of the human's entities unnamed. The examples show the commentator sometimes repeating an unrelated entry, or reading a wiki line aloud. And the approach assumes a wiki exists for the game.

The idea to carry away is a clean division of labour. Perception, timing and language transfer across games, so they live in the trained commentator. Game knowledge does not, so a frozen agent looks it up from frames at test time. Training targets what that leaves open: asking only when an answer would change this model's line, and trusting an answer only when it fits. When no answer comes, the commentator now leaves the name out instead of borrowing one.