Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach

Evaluating Agentic Capabilities Before Post-Training. An End-to-End Approach. By Jiarui Qin and seven others, from Tencent Youtu Lab and Shanghai Jiao Tong University. Published on the fourteenth of October. No code repository is linked to it.

Agentic capability matters more and more for modern LLMs, but it is hard to see during pretraining. The usual evaluations at that stage are static tasks derived from trajectories, or proxies based on likelihood. They are lightweight, and that is their appeal. What they do not show is how a base model actually behaves once it is placed in an interactive environment and has to act.

ASAP, short for Agent Scaffold Adaptation for Pretraining Evaluation, sets out to measure exactly that. It is a lightweight framework for drawing out agentic behavior from pretrained base models and scoring it on end-to-end tasks, so that a checkpoint can be judged on whether it gets the job done, not only on how it scores on a proxy.

The key decision is what not to do. ASAP does not convert a base model into a chat-style agent. It keeps the model's completion interface as it is, and adapts existing agent scaffolds to fit it, through three devices: structured continuation of the trajectory, short reference prefixes, and minimal mechanisms for recovery. Because the scaffold bends toward the model rather than the other way round, a base model can be evaluated as it stands.

On software-engineering and deep-research benchmarks, ASAP produces end-to-end scores that discriminate between base models and between pretraining checkpoints. Beyond the aggregate number, it exposes the execution trajectories the models generate. Those make it possible to study exploration, editing, testing, recovery and task completion across training stages, giving a behavioral picture of how agent-relevant capabilities change during pretraining, and of how execution patterns differ after post-training.

The proposal is modest in its framing. End-to-end agent evaluation is offered as a complement to conventional pretraining benchmarks rather than a replacement, a practical diagnostic of interactive behavior that can be run before any post-training has shaped the model into an agent.