RealtimeWAM: One-Step Asynchronous World Action Models

RealtimeWAM. One-Step Asynchronous World Action Models. By Chengtao Lv and nine others, from Nanyang Technological University, Beihang University, and two other institutions. Published on the fifth of October. Its code repository is worked out: about two thousand source files, with a readme and a licence.

World Action Models, or WAMs, take visual representations from video generation backbones and use them to guide action prediction. Recent efficient WAMs adopt a Mixture-of-Transformers architecture, computing the video representations once and reusing them in an action expert. That removes one redundancy, but two bottlenecks still hold inference back. One is iteration inside an expert: actions are denoised over multiple steps, so a single action costs several passes. The other is waiting between experts: the video expert and the action expert execute in sequence, and the action expert cannot begin until the video expert is done.

RealtimeWAM is pitched as an extremely efficient WAM variant that addresses both, with one-step action generation and asynchronous inference. Each bottleneck gets its own component. Teacher-Anchored Consistency Distillation, or TACD, removes the iteration. Cross-Expert Wavefront Pipelining, or CEWP, removes the waiting.

TACD begins from what the authors call a local-global error gap. Low local consistency error, on its own, does not guarantee that the final action is accurate. Small errors at each step can still leave the end result off target. So TACD keeps local consistency, and supplements it with explicit supervision from the endpoint of a frozen teacher's multi-step rollout. The student is anchored to where the teacher actually ends up, and that is what lets a single step produce an accurate action.

CEWP goes after waiting that is not needed. It overlaps the two experts by sharing the video key-value cache block by block, and synchronizes only at the last moment: immediately before the action attention that consumes a given block. Instead of the action expert waiting for the whole video pass, the two advance together, one block behind the other, like a wavefront moving through both.

The experiments span diverse benchmarks, among them LIBERO, LIBERO-Plus and RoboTwin, and model variants including Fast-WAM and Faster-WAM. That second point matters, because it shows the method applied to more than one existing efficient WAM. Across these benchmarks RealtimeWAM stays near-lossless, with a drop of under one percent, while delivering a significant end-to-end speedup, given as about twenty-five times on an H100. Code and checkpoints are released.

The speed here is assembled from two separate savings, each matched to one bottleneck. Distillation against the teacher's endpoint removes the steps inside the action expert. Block-wise pipelining removes the stalls between the video and action experts. One fix works within an expert and the other works across them, and the reported cost of both together is under one percent of performance.