REALM: A Coarse-to-Fine Generative Framework for Embodied Reactive Listening
Peizhen Li, Longbing Cao, Yang Zhang
Macquarie University · University of North Texas
Published September 27, 2026 · on the feed September 29, 2026 · ▲ 20 · arXiv 2609.33095
The abstract
Generating responsive listener facial motion is an important task for embodied conversational AI. Two modeling challenges are central: accounting for the timing of speaker cues while maintaining continuity with the listener's ongoing motion, and capturing locally variable facial events alongside the overall motion trajectory. Listener responses may follow preceding cues with a temporal lag, while brief expressions and blinks introduce variation that is difficult to predict deterministically. These challenges motivate a framework that combines history-aware temporal alignment with stochastic expression refinement. We propose REALM (Reactive Embodied Audio-driven Listening Model), a coarse-to-fine framework for audio-driven reactive listening. A Reactive Gated Speaker-Listener Fusion module combines listener motion history with speaker audio through a delay-centered attention prior and adaptive gating. A coarse decoder predicts a base motion trajectory, which is augmented by audio-conditioned stochastic residuals in the expression subspace while retaining the coarse pose parameters. Evaluations on ViCo and L2L show improvements over the evaluated baselines across multiple motion-quality metrics. Additional analyses examine delay sensitivity, gate behavior, and blink dynamics. Finally, deployment on an Ameca humanoid robot and a perceptual user study demonstrate the applicability of the generated behavior to physical embodiment. Code: https://github.com/lipzh5/REALM Demo: https://youtu.be/Tf5mpd5S8VQ
The Brief
2:26A short podcast summary of the paper. Written and voiced by AI.
Transcript
REALM. A Coarse-to-Fine Generative Framework for Embodied Reactive Listening. By Peizhen Li and two others, from Macquarie University and the University of North Texas. Published on the twenty-seventh of September. Its code repository is public and real: thirteen source files.
A conversational agent with a face has to listen, not only speak, and listening shows on the face. Generating that responsive listener motion is the task here, and the abstract names two challenges. One is timing. A listener reacts to cues from the speaker, often after a lag, and must do so while staying continuous with the motion already underway. The other is texture. On top of the overall trajectory of the face sit brief, local events, short expressions and blinks, whose variation is hard to predict deterministically.
Those two challenges point to two ingredients: temporal alignment that is aware of history, and refinement of expression that is stochastic rather than fixed. REALM, the Reactive Embodied Audio-driven Listening Model, combines them in a coarse-to-fine framework driven by audio. The broad motion is laid down first, and the variable detail is added on top.
The first piece is a Reactive Gated Speaker-Listener Fusion module. It joins the listener's motion history with the speaker's audio, using an attention prior centred on a delay, together with adaptive gating. Next, a coarse decoder predicts a base motion trajectory. That trajectory is then augmented with stochastic residuals conditioned on the audio. These residuals act only in the expression subspace, and the coarse pose parameters are kept as they were. The head's broad movement stays put, while the expression gains its variability.
On ViCo and on L2L, REALM improves on the evaluated baselines across several motion-quality metrics. Further analyses look at sensitivity to delay, at how the gate behaves, and at the dynamics of blinks, each tied back to one of the challenges named at the start. Finally, the model is deployed on an Ameca humanoid robot, and a perceptual user study is run, to show the generated behaviour carries over to a physical body. Code and a demonstration video are linked.
Listening, as framed here, is a matter of when and of how much. The delay prior and the history-aware gate handle the when. The split between a coarse trajectory and stochastic expression residuals handles the texture of a face that is paying attention. Putting the result on a physical robot is the test of whether it reads as listening to people in the room.
Paper Quest · 7 chapters · about 37 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Listening is a performance too. Here the face has to react in time and still feel alive.
You are building a face for a conversational agent, and it has to listen, not only speak. Its nods and expressions should react to the speaker's voice.
The road ahead
- 1 · The problem ~6 min A listening face must react to a speaker's voice after a lag, stay smooth, and keep brief expressions and blinks that vary every time.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Align speaker audio softly around a typical reaction delay, gate it against the listener's history, predict a coarse trajectory, then add random detail to expressions only.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Three stages in order: gated fusion of history and audio, a coarse decoder for the base trajectory, then stochastic expression residuals.
- 6 · Field test ~3 min
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.