Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

Motion-Omni. End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue. By Chengqian Ma and three others, from Peking University, Lightspeed, and two other institutions. Published on the twenty-eighth of August. Its code repository is worked out: about forty source files, with a readme.

An avatar that holds a conversation has to decide what to say and how to move while saying it. Those two abilities live in separate model families. Spoken dialogue models produce speech and no motion. Co-speech motion models produce motion, but only from audio that was handed to them. Neither family was built to do the other one's job. The standard remedy is a cascade: generate the spoken response first, then run a motion model over the finished audio. That costs a second full inference pass, and it precludes any joint optimisation between the two halves, because by the time motion is generated the speech is already fixed.

Motion-Omni collapses the cascade. It is an end-to-end framework in which a spoken dialogue model natively outputs explicit facial expression along with hand, upper-body and lower-body motion. The motion is generated directly from the hidden states that produce the speech, and that is the part which matters. Both come out of the same internal state at the same moment, rather than one being read off the other afterwards. End-to-end is meant literally: no second model reads the first one's output.

Joint training turns out not to be optional here. With the speech pathway frozen, motion remains misaligned with the audio. Alignment is recovered only by co-adapting the language model, the Speech Generator and the Motion Generator together under both objectives, and doing that while retaining spoken-dialogue ability. Supervision comes from a pipeline built to be scalable and model-agnostic: consistent-voice speech responses are pseudo-labelled by a motion teacher that is deliberately replaceable, yielding roughly four hundred and twenty-three thousand quality-ranked pairs, about fourteen hundred hours in all. The teacher being replaceable is what makes the pipeline scalable, since nothing downstream depends on which motion model did the labelling.

Instantiated with a Qwen2.5-7B-Instruct backbone, Motion-Omni-Q7 matches the same-audio teacher cascade to within two percent on reference-free motion metrics, while responding five point four times faster and running faster than real time. Against every cascade that is not the teacher, it surpasses them on beat correlation and diversity. Its word error rate is two point six two percent, the lowest among the omni-modal systems compared. Speed and speech quality were both held while the second inference pass was removed.

Alongside the model the authors release SwDA-500 and, to their knowledge, the first public evaluation protocol for stochastic open-ended full-body spoken dialogue. The protocol matches audio across motion systems and unifies rendering, automatic metrics, human evaluation and latency measurement. That is the quieter contribution: a field comparing systems which each supplied their own audio was never comparing motion alone, and matching the audio is what turns the comparison into one.

Motion-Omni. End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue. By Chengqian Ma and three others, from Peking University, Lightspeed, and two other institutions. Published on the twenty-eighth of August. This is the long version: the whole paper rather than the abstract.

Speech and the movement that accompanies it are tightly coupled in human communication, and that movement carries meaning which complements what is being said. Yet the two abilities live in separate model families. Spoken dialogue models produce speech and no motion. Co-speech motion models produce motion, but only from audio someone hands them. Run them in sequence and you get both, which is the standard remedy and the baseline throughout this paper. It has two structural costs, and the paper is an attempt to remove both at once. The motion model runs as a second full inference pass after the audio is finished, and no motion objective can ever update the speech or dialogue parameters.

Three difficulties stand in the way. The first is optimisation: making motion a native output means jointly training two objectives without degrading either, and the two outputs live at different rates, twelve and a half hertz for speech units against thirty for motion, while sharing parameters through which the losses can interfere. The second is data at a scale no corpus provides. The third is evaluation. Existing full-body motion benchmarks assume the speech is supplied and fixed, while spoken-dialogue evaluations with generated speech cover facial animation only. Neither asks the actual question: whether a model's own valid but unpredictable spoken response comes with appropriate whole-body motion.

The architecture bridges the rate gap through what the authors call a dual-input conditioning interface. The Motion Generator attends to the Speech Generator's hidden states as keys and values, and takes the embeddings of the emitted speech tokens as queries, interpolated up from the speech-unit rate to the motion rate. Motion is therefore generated from the representations that produce the speech, and never from the rendered waveform. Training runs in four stages, moving from speech recognition through text-to-speech and a text-to-speech-with-motion curriculum into a joint mixture, co-adapting the projector, the language model, the Speech Generator and the Motion Generator while keeping each a distinct module.

The ablation establishes that the return path matters rather than merely existing. With the speech pathway frozen, generated motion stays visibly misaligned with the audio; alignment is recovered only by co-adapting both. Supervision comes from a model-agnostic pipeline in which a replaceable motion teacher labels every response waveform and a dual-metric quality score orders the resulting curriculum, yielding roughly four hundred and twenty-three thousand pairs, about fourteen hundred hours. The instantiated model matches the same-audio teacher cascade to within a couple of percent on reference-free motion metrics while responding over five times faster and running faster than real time, and its word error rate of two point six two percent is the lowest among the omni-modal systems compared.

The limitations section is unusually forthcoming and repays reading. Motion quality is bounded by the teacher's codebook and pseudo-labels, so motion outside that distribution cannot be expressed at all. Only one instance was trained. The model conditions on neither speaker identity nor emotion. The training data is entirely English, which the authors say restricts generalisation to other languages and other motion cultures. It ingests the user's whole utterance before emitting a first token, making it an offline responder rather than a streaming one. And the evaluation is candid about itself: the automatic motion metrics are imperfect proxies for perceived naturalness, the human comparison used four annotators and is described as exploratory, and an attempt to scale rubric scoring with a video-input model as judge was abandoned because that judge correlated with human raters only on speech quality.

The abandoned judge is the detail to keep. Faced with a metric that agreed with people about speech and not about motion, the authors reported the failure and fell back to human ratings rather than quoting the convenient number. That is the same instinct visible in the architecture: do not read the waveform when the states are available, do not freeze the pathway you claim to be co-adapting, and do not let a proxy stand in for the thing you cannot yet measure well.