onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction
Lei Yang, Mengyin Liu, Jia Wang, Hangyu Guo, Liang Zhao, Zheng Ge, Kang An, Binxing Jiao and 4 others
StepFun · Xiamen University
Published September 21, 2026 · on the feed September 22, 2026 · ▲ 54 · arXiv 2609.24983
The abstract
We present onPanda, an interactive tool for efficiently annotating LLM alignment data and agent trajectories. onPanda adopts token-level correction as its core interaction: while reading a model response, the annotator locates the first inappropriate token and either picks a substitute from the model's candidate tokens or types the correct text via free-form editing. The system then truncates everything after that position and continues generation from the corrected prefix, repeating this locate-correct-continue loop until a satisfactory response is obtained. This mechanism lets annotators precisely steer model outputs at low cost: a small controlled study suggests that onPanda reduces median annotation time by 52% over manual post-editing. Since the vast majority of tokens in the final response are generated by the model itself, the resulting data largely preserves the model's sampling distribution and is well suited for constructing on-policy SFT and preference data. Furthermore, the token-level corrections recorded during annotation provide fine-grained supervision with precise positions and naturally paired positive--negative samples. onPanda also connects to external tools and harnesses, enabling interactive trajectory annotation in realistic environments. In addition, we release Panda-CVL, a dataset annotated with onPanda, together with a benchmark for token-level correction.
The Brief
2:29A short podcast summary of the paper. Written and voiced by AI.
Transcript
onPanda. Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction. By Lei Yang and eleven others, from StepFun and Xiamen University. Published on the twenty-first of September. Its code repository is worked out: about one hundred source files, with a readme and a licence.
onPanda is an interactive tool for annotating alignment data for language models, and trajectories for agents, efficiently. Alongside it the authors release Panda-CVL, a dataset annotated with onPanda, together with a benchmark for token-level correction.
The core interaction is token-level correction. An annotator reads a model's response and finds the first token that is inappropriate. At that point they either pick a substitute from the model's own candidate tokens, or type the correct text freely. The system then truncates everything after that position, and continues generating from the corrected prefix. The loop repeats, locate, correct, continue, until the response is satisfactory. The annotator only writes at the points where the model went wrong, and everything between those points is the model's own continuation. That is the mechanism the authors say lets annotators steer model outputs precisely, and at low cost.
The tool also connects to external tools and harnesses, so trajectories can be annotated interactively in realistic environments.
The way the data is made determines what it is good for. Because the vast majority of tokens in a finished response are generated by the model itself, the resulting data largely preserves the model's own sampling distribution. That makes it well suited to building on-policy SFT data and preference data.
The record of corrections is a second product. Each correction marks a precise position, and it naturally pairs a positive sample with a negative one: the token that was rejected, and what replaced it. That gives fine-grained supervision with exact locations attached.
The resource is for people producing alignment data, whether for chat models or for agents working in real environments. The case for its efficiency comes from a small controlled study, which suggests onPanda cuts median annotation time by fifty-two percent compared with manual post-editing. It also serves anyone who needs token-level correction data, since the benchmark released with Panda-CVL is built around exactly that task.
The design idea is to intervene only where a response first goes wrong. Fix that one token and let the model carry on, and the annotator steers the output precisely while writing very little of it. What comes out is data that still sounds like the model, plus a map of exactly where it went wrong.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Good alignment data is slow to write, and rarely sounds like the model.
You need alignment data for a language model. Writing ideal responses by hand is slow, and the result does not look like what the model itself would write.
The road ahead
- 1 · The problem ~7 min Alignment data is expensive to make by hand, and hand-made or heavily edited responses stop looking like what the model would write.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Fix only the first token where the model goes wrong and let the model write the rest, so the data stays mostly its own.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Locate the first bad token, replace it by picking or typing, cut what follows, and let the model continue; repeat until done.
- 6 · The experiment ~6 min You’ll guess: In a small controlled study, how much did onPanda cut median annotation time compared with manual post-editing?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.