StreamEval: Evaluating Interactive Streaming Video Generation Models

StreamEval. Evaluating Interactive Streaming Video Generation Models. By StreamEval Team, from Tsinghua University, NVIDIA, and five other institutions. Published on the twentieth of October. Its code repository is worked out: about one hundred and forty source files, with a readme and a licence.

Streaming video generation is advancing, and with it comes the expectation of real-time interaction: a user changing what the video does while it is being generated. Measuring that requires more than judging how good the frames look. It means checking whether requests that arrive mid-stream are carried out, and whether what should stay the same actually does. A generated avatar can look flawless and still ignore the instruction it was just given, or a generated world can stay beautiful while forgetting the change it was asked to make earlier in the stream. Picture quality alone would score both as successes.

StreamEval is a unified benchmark for that setting. It assesses models along four dimensions: task execution, state consistency, visual fidelity, and audiovisual coordination. These are instantiated through ten task-specific metrics, each on a scale from zero to one hundred.

The benchmark spans five tasks: avatar dialogue, avatar performance, world generation, instruction editing, and reference editing. It comprises three hundred scripted test cases, lasting from fifteen to three hundred seconds. Each case specifies initial assets, a series of multimodal requests that arrive incrementally, and explicit requirements for what state should change and what content should be preserved.

Thirty-seven distinct models are evaluated across the three hundred cases. High visual quality and identity retention turn out not to coincide with reliable request execution or sustained state consistency. Request execution in particular remains hard. Among seven avatar-performance models, the highest Following score is thirty-six point nine.

In text-conditioned world generation, the best Following score is twenty-five point nine three. Among three reported reference-editing models, the best Edit Success is fifteen. On a scale that runs to one hundred, the strongest models on these request-execution metrics sit well below the halfway mark, and in reference editing far below it. The visual side of the ledger does not predict where a model lands on this side.

The benchmark separates two things that a viewer might assume travel together: looking right and doing what was asked. On current models they come apart, and the authors place reliable request execution under timing and preservation constraints at the centre of what omni-modal real-time interactive systems still need to solve.