TaoMate-HD: Progressive-Resolution Autoregressive Audio-Video Generation

TaoMate-HD. Progressive-Resolution Autoregressive Audio-Video Generation. By Qijun Gan and five others, from Alibaba Group and Nanjing University. Published on the second of November.

A streaming digital human has two jobs at once. It produces detailed video, and it keeps speaking as new text arrives. Both strain the system. Denoising at high resolution is costly, and when the text is updated mid-stream, the speech can come out repeated or interrupted.

TaoMate-HD is an autoregressive model for high-resolution audio-video streaming, and it works through short segments, called chunks, one at a time. Within a chunk, resolution rises progressively. Two denoising steps establish the coarse structure of the video at low resolution. A final step refines the video to high resolution. Audio is not a separate pass: it is generated throughout the same trajectory as the video.

Continuity comes from history. Each stage passes its audiovisual predictions to the same stage of the next chunk, so the low-resolution steps inherit from low-resolution steps and the refinement step from the refinement step. When the text is updated, the model retains that history and uses it to infer what has already been spoken, which is aimed squarely at the repeated or interrupted speech. Continuation training uses asymmetric text conditioning, with one text window for generation and a separate one for distillation.

Against direct high-resolution generation, with the same model and the same hardware, the progressive scheme runs one point six times faster. An optimized stage-parallel deployment goes further, reaching a one point nine times overall speedup over running the progressive stages serially. The continuation training cuts character error rate by eleven point one eight percentage points relative to the baseline. And the model supports audiovisual generation an hour long.

One mechanism serves both halves of the problem. History passed stage to stage across chunks links the generation over time, and the same history is what the model consults, when the text changes, to know what has already been said.