Act First, Reason Later: Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamics
Zubin Zheng, Jiahao Wu, Shaofeng Zhang, Zhirui Zhang, Yew-Soon Ong, Shengcai Liu
Southern University of Science and Technology · Zhongguancun Academy · Hong Kong Polytechnic University · DeepCybo · Nanyang Technological University
Published September 29, 2026 · on the feed September 30, 2026 · ▲ 6 · arXiv 2609.36608
The abstract
On-policy distillation (OPD) trains multi-turn language agents with dense teacher supervision on student-generated responses. However, standard think-then-act rollouts require lengthy reasoning before each short action, delaying environment transitions and experience collection. Generating actions directly reduces this delay but can degrade rollout quality. To address this, we propose ActFirst-OPD, an act-first, reason-later training framework that decouples environment interaction from full-response generation. The student infers and executes actions through reference-conditioned inverse dynamics using its current interaction context and a reference next observation, and switches to autonomous next-action prediction when the resulting transition deviates from the reference trajectory. From the collected interaction contexts, the student asynchronously generates full think-then-act responses for token-level teacher supervision. Experiments across 0.6B-, 1.7B-, and 4B-parameter Qwen3 students show that ActFirst-OPD achieves average wall-clock training speedups of 2.3times on ALFWorld, 1.8times on WebShop, and 4.9times on ScienceWorld over Vanilla OPD. It matches or exceeds all compared OPD baselines in mean task success rate across eight of nine benchmark-model settings. These results demonstrate that reasoning need not block acting during multi-turn agent distillation.
The Brief
2:21A short podcast summary of the paper. Written and voiced by AI.
Transcript
Act First, Reason Later. Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamics. By Zubin Zheng and five others, from the Southern University of Science and Technology, Zhongguancun Academy, and three other institutions. Published on the twenty-ninth of September. No code repository is linked to it.
On-policy distillation trains multi-turn language agents with dense teacher supervision on responses the student generated. The standard rollout is think then act: lengthy reasoning comes before each short action. That reasoning delays every environment transition, and with it the collection of experience. Generating actions directly would remove the delay, but it can degrade the quality of the rollouts. So the choice has looked like one between slow, careful rollouts and fast, sloppy ones.
ActFirst-OPD reverses the order. It is an act-first, reason-later training framework that decouples interaction with the environment from generation of the full response. Actions happen quickly. The reasoning that would have come before them is produced afterwards, when it no longer holds the environment up.
To choose its actions, the student uses reference-conditioned inverse dynamics. Given its current interaction context and a reference next observation, it infers the action and executes it. When the resulting transition departs from the reference trajectory, the student switches to predicting the next action on its own. Later, from the interaction contexts it has collected, the student asynchronously generates full think-then-act responses, and those receive token-level supervision from the teacher. The teacher's dense signal is untouched. What moved is the point at which the reasoning gets written.
The students are Qwen3 models at three sizes: zero point six billion, one point seven billion and four billion parameters. Against vanilla on-policy distillation, the average wall-clock training speedups are two point three times on ALFWorld, one point eight times on WebShop, and four point nine times on ScienceWorld.
Speed does not come at the cost of success. ActFirst-OPD matches or exceeds all compared distillation baselines in mean task success rate in eight of nine benchmark and model settings.
The authors turn their title into a design rule: during multi-turn agent distillation, reasoning need not block acting. Experience is collected at the pace of action, and the teacher still gets complete reasoning to supervise, just written a little later.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is an environment kept waiting by a student that thinks too long.
You are training a small student agent on multi-turn tasks, with a teacher grading every token of the responses the student writes.
The road ahead
- 1 · The problem ~6 min A student agent that reasons at length before every short action makes the environment wait, and the waiting piles up over many turns of training.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Act first by aiming at a good reference run's next observation, and write the full reasoning later, off to the side, for the teacher to grade.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Fast actions drive the run, guided by the reference until it diverges; full responses are written from the collected contexts and graded by the teacher.
- 6 · The experiment ~6 min You’ll guess: On ScienceWorld, how large was the average wall-clock training speedup of ActFirst-OPD over vanilla on-policy distillation?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.