Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generation

Vidu S2. Real-Time Interactive, Editable, and Spatial Video Generation. By Jintao Zhang and thirty-four others, from Tsinghua University and Shengshu Technology. Published on the tenth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Vidu S2 is about video generation that happens in real time, while someone interacts with it. The abstract measures the new release against its predecessor, Vidu S1, and names what changes: a digital character that can be interacted with live, a video stream that can be edited as it plays, and an exploration of whether spatial video can be generated in real time as well.

Vidu S2 is two models under one name. Vidu S2-Avatar is a real-time interactive digital-character model. Vidu S2-Editing is a real-time video editing model. Across both of them, the authors explore the feasibility of real-time spatial video generation, for the avatar model and for the editing model alike.

Compared with Vidu S1, the avatar model supports real-time video generation at seven hundred and twenty p. It supports generation with dynamic references, meaning references that can be updated at any moment rather than fixed at the start. And it follows instructions more strongly, with dancing given as the example. The editing model works on a video stream in real time. The edits listed are style rendering, clothing replacement, character replacement, and background replacement.

The experiments report that Vidu S2 outperforms all baselines. A playable online demo is available for anyone who wants to try the stream directly.

What the release describes is a shift in when generation happens. Not a clip rendered and handed over, but a character and a stream that change while you watch, and that can take a new reference or a new edit at any moment along the way.

Vidu S2. Real-Time Interactive, Editable, and Spatial Video Generation. By Jintao Zhang and thirty-four others, from Tsinghua University and Shengshu Technology. Published on the tenth of September. This is the long version: the whole paper rather than the abstract.

Most video generators behave like a vending machine. You enter a prompt, wait minutes or even tens of minutes, and receive the finished clip, with no way to act while it is being made. Their predecessor, Vidu S1, broke that pattern for talking heads: a character that responds to voice instructions while it streams, at up to forty-two frames per second on consumer GPUs. Its limits were plain. Resolution was capped at five hundred and forty p. The reference image defining the character was fixed once a stream began. Large body motion such as dancing was hard to follow. And it could not edit an incoming video. Vidu S2 is two models aimed at those gaps: S2-Avatar for interactive characters, and S2-Editing for editing a live stream.

The core difficulty of streaming generation is that video is produced segment by segment, each conditioned on what came before, so errors pass forward and eventually cause drift or collapse. The remedy this work builds on is Self- Forcing: train the model conditioned on segments it generated itself, so training matches what it sees at inference. The authors credit it with getting the setup right, and name two problems. The self-generated history is fed in clean rather than noised. And it is detached from the computation graph, so no gradient flows through it. Training runs in stages. A bidirectional teacher is trained with each segment supervised by its own caption, then made causal and trained on a mix of clean history and history deliberately corrupted with noise.

That fix is Self-Replay Forcing. The student first performs a long autoregressive rollout, with its blocks and key-value caches detached so the full computation graph is not kept. The whole self-generated trajectory is then re-noised, each segment independently, following Diffusion Forcing, and replayed in a single gradient-enabled causal pass. Inside that replay every segment stays in one graph, so gradients cross block boundaries without running back through the original rollout. Supervision is distribution matching distillation against the teacher, plus a perceptual loss to resist mode collapse. Resolution rises to seven hundred and twenty p without slowing the backbone, which keeps generating at low resolution while a separate refiner upsamples each segment in latent space in a single step. The two keep different histories. The backbone attends to a high-noise cache carrying coarse motion; the refiner attends to a low-noise, high-resolution cache that preserves identity and texture. Motion propagation and detail restoration are deliberately kept apart.

S2-Editing rests on frame-aligned attention. Each generated frame reads only the source frame at the same time step, so the edited video keeps exactly the input's motion and timing, while a reference image stays visible to every frame so a new appearance carries through. Paired training data is the hard part, and for style transfer the route is indirect. A model is trained to rebuild a real clip from its surface-normal video plus one of its own frames as reference. Swap that frame for a stylised image, keep the normals, and out comes a stylised clip with the original structure and motion: a ready-made training pair. For speed, linear layers run in eight-bit precision with fine-grained scaling, each layer gets its own choice among three attention approximations, and a scheduler lets the encoder, backbone, refiner and decoder share GPUs over time.

On StreamAV-Bench, S2-Avatar is best on every reported metric against thirteen other systems. The telling columns are audio-visual. Synchronization error falls to zero point six one seven, the lowest listed, and audio-visual alignment rises to zero point three five three, where the next best is zero point two seven two. Consistency scores sit near ceiling for several systems, so margins there are thin. For editing, S2-Editing leads Sparkle-Bench overall, three point seven four against three point six seven for the streaming system Decart Lucy, and on the ViViD virtual try-on test it roughly halves the distribution distance of the next best method.

Against commercial systems the paper turns to human judges, twenty trained evaluators working from a written rubric. On overall quality, S2-Avatar is preferred in every comparison against PixVerse and HeyGen, and in eighty-five point seven percent against Runway. Every share in those avatar charts is a multiple of one seventh, which is what seven judgments per comparison would produce; the count is not stated. The editing study is larger, a hundred and fifty paired cases, and there S2-Editing is preferred overall in seventy-two point seven percent of comparisons against Decart Lucy.

There is no limitations section. The one discussion of difficulty concerns spatial video. In a headset it needs higher resolution, and in camera passthrough, latency can put head motion and the view out of sync. The authors call low end-to-end latency with high visual quality a key open challenge. The spatial results themselves are framed as a feasibility exploration, built by estimating depth per frame and warping it into two views, and are shown only as example frames. The protocol says product-page claims about frame rate or latency do not replace measured results, and the internal benchmark is said to cover latency and cost, yet no latency, throughput or cost figures are reported. A ten-minute long-horizon editing set is described but its results never appear.

The idea to carry away is that in streaming generation the model's own past is the input it must learn to live with. Self-Replay Forcing lets the training signal flow back through that past without storing the whole rollout. The rest of Vidu S2 follows the same instinct: keep the fast path cheap and put quality where it does not block the stream, in a one-step refiner, a frame-aligned editor, and kernels chosen layer by layer. A playable demo is online at the Vidu website.