EvolvingAvatar: Interactive 3D Head Generation That Adapts as Conversations Unfold
Junjie Chen, Fei Wang, Kun Li, Yiqi Nie, Xun Yang, Yanbin Hao, Linfeng Zhang, Meng Wang
Hefei University of Technology · Shanghai Jiao Tong University · Hefei Comprehensive National Science Center · University of Science and Technology of China · United Arab Emirates University · Anhui University
Published September 28, 2026 · on the feed September 29, 2026 · ▲ 3 · arXiv 2609.35616
The abstract
Interactive 3D head generation requires coordinated speaking and listening motion that responds to an evolving conversation. Existing generators use incoming observations as context but keep their parameters fixed, leaving conversational patterns unused as a learning signal. We introduce EvolvingAvatar, a causal generator that uses test-time training to adapt to user face video and dyadic audio during interaction. Its dyadic context prediction objective provides a self-supervised learning signal from audiovisual context without target motion labels at test time. Persistent fast weights accumulate these updates within each conversation to guide motion generation, while transient jaw adaptation responds to current audiovisual context. Predicted speech activity controls how persistent adaptation guides motion. We also introduce InterHead-Bench, a unified 455.95-hour benchmark built from single-view and dual-view conversation videos. Experiments show improved conversational motion statistics over strong baselines. On the hardest out-of-distribution split, generation improves as conversations unfold, reducing mismatch with recorded user-avatar expression statistics by up to 11.1% from the first interval.
The Brief
2:06A short podcast summary of the paper. Written and voiced by AI.
Transcript
EvolvingAvatar. Interactive 3D Head Generation That Adapts as Conversations Unfold. By Junjie Chen and seven others, from Hefei University of Technology, Shanghai Jiao Tong University, and four other institutions. Published on the twenty-eighth of September. No code repository is linked to it.
An interactive 3D head has to both speak and listen, with motion that responds to a conversation as it evolves. Existing generators take incoming observations as context, but their parameters stay fixed. Whatever patterns a particular conversation exhibits are never used as a signal to learn from. The generator that begins a conversation is exactly the generator that ends it, however much the exchange in between might have taught it about the person on the other side.
EvolvingAvatar is a causal generator that adapts during the interaction itself, using test-time training on the user's face video and the audio of both speakers. Its learning signal is a dyadic context prediction objective, which is self-supervised: it draws on the audiovisual context and needs no target motion labels at test time.
The updates are split by timescale. Persistent fast weights accumulate them across a whole conversation and guide motion generation. A transient jaw adaptation responds to the audiovisual context of the moment. Predicted speech activity decides how strongly the persistent adaptation steers the motion. For evaluation the authors also build InterHead-Bench, a unified benchmark of four hundred and fifty-five point nine five hours, drawn from single-view and dual-view conversation videos.
Experiments show better conversational motion statistics than strong baselines. On the hardest out-of-distribution split, generation improves as the conversation goes on, cutting the mismatch with recorded user and avatar expression statistics by up to eleven point one percent relative to the first interval.
That last result is the one the title promises. On the split furthest from its training data, the avatar at the end of a conversation is measurably closer to the recorded behaviour than it was at the start. Nothing about the model was retrained offline in between. The improvement came from the conversation itself, absorbed as it happened.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A face that listens but never learns. That is the wall this quest starts at.
You are building a 3D head that talks and listens with a real person on video. Its motion has to answer the partner's nods, speech and turn-taking as the conversation goes on.
The road ahead
- 1 · The problem ~7 min A conversational 3D head must react to its partner, yet today's generators stay frozen through the chat and have no labels to learn from.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min EvolvingAvatar keeps training itself during the chat with a label-free signal, using slow-building and short-lived updates steered by predicted speech activity.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Each interval, the generator tokenises user video and dyadic audio, updates fast weights by context prediction, gates them by activity, and generates motion.
- 6 · The experiment ~6 min You’ll guess: On the hardest out-of-distribution split, by up to how much did EvolvingAvatar cut its mismatch with recorded expression statistics, compared with the first conversation interval?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.