HiPLEX: Hierarchical Policy Factorization for Full Duplex Speech Language Models

HiPLEX. Hierarchical Policy Factorization for Full Duplex Speech Language Models. By Kyudan Jung and seven others, from Qualcomm AI Research and KAIST AI. Published on the sixth of October. No code repository is linked to it.

As conversations with AI become more natural, full-duplex speech language models, which can hold real-time dialogue, matter more. Generating a good response is only part of the job. These models also have to coordinate turn-taking, backchanneling, and who holds the floor, all in real time. Reinforcement learning offers a way to refine those behaviours using direct feedback on how interactions turn out. But existing reinforcement learning methods do one of two things. They apply timing feedback to a token policy, or they optimise semantic content. Doing both together, improving when the model speaks and what it says at once, has been left unresolved.

HiPLEX is a reinforcement learning framework that splits a pretrained full-duplex text policy into two factors. A control policy decides when to emit content. A conditional content policy decides what to emit. Separating the two decisions means each can receive the feedback that suits it. Timing is judged by what happens in the conversation, such as whether the model talks over a user who has only paused. Content is judged by what was actually said. With one policy for each, neither kind of feedback has to be squeezed into a signal designed for the other.

The control factor chooses among three actions, named pad, epad, and con. The content factor selects a token only when con is chosen. In effect, at every frame the model first decides whether this is a moment to contribute content, and only if it is does it decide which token. The hierarchy describes these conditional actions within each frame, and it reuses the model's existing text head rather than adding a new one. Feedback is then routed to the right place. Timing advantages go to the token-group factor, through event-causal masks derived from generated speech episodes. A semantic advantage, scored by an LLM judge, goes to the conditional content factor.

The comparison is against GRPO. Across three Moshi seeds on Full-Duplex-Bench v1, HiPLEX reduces takeover rates during natural user pauses and during opportunities for backchannels. It also shortens response latency after an interruption, while judged quality of interruption responses stays comparable. On both Moshi and PersonaPlex, HiPLEX matches pooled human turn-timing and backchannel-rate marginals more closely than GRPO does.

What the factorisation buys is a clean division of credit. Earlier methods refined timing or refined content, one or the other. By giving the when and the what their own factors, each with its own advantage signal, HiPLEX tunes turn-taking toward human patterns while the judged quality of what gets said holds steady.