CoEvoWhen: Policy-Tool Coevolution for Ultra-Long Video Temporal Grounding
Yiduo Jia, Muzhi Zhu, Jinchuan Shi, Hao Zhong, Yuling Xi, Ke Liu, Hao Chen
Zhejiang University
Published September 30, 2026 · on the feed October 1, 2026 · ▲ 8 · arXiv 2609.40048
The abstract
Ultra-long video temporal grounding requires balancing long-range evidence search with fine-grained event understanding under a limited visual budget, yet existing agentic methods still rely largely on predefined policies and tool capabilities. Motivated by this, we propose a novel policy-tool coevolution framework that jointly evolves high-level policies and executable media tools from the agentic reasoning trajectories of a VLM, forming a reusable skill without updating model parameters. During evolution, an external skill updater distills transferable task experience in long-video temporal grounding, accordingly refining the orchestration of long-range image-based and fine-grained video-based observations. Alongside these policy updates, the updater employs its coding capabilities to upgrade existing tools or create new ones, adapting the tools to long-video evidence acquisition. Equipped with the evolved skill, the VLM autonomously orchestrates tools under the guidance of the evolved policy, coordinating image and video observations for agentic inference without relying on a separate, stronger planning model. Extensive experiments spanning five benchmarks and three VLMs show that policy-tool coevolution consistently improves temporal grounding accuracy in ultra-long videos while reducing visual token cost at inference, and that the evolved skill yields substantial performance gains on general long-video QA without additional task-specific evolution, demonstrating the effectiveness and generalizability of our framework for long-video understanding.
The Brief
2:29A short podcast summary of the paper. Written and voiced by AI.
Transcript
CoEvoWhen. Policy-Tool Coevolution for Ultra-Long Video Temporal Grounding. By Yiduo Jia and six others, from Zhejiang University. Published on the thirtieth of September. Its code repository is public but slight: two source files.
Temporal grounding in an ultra-long video means finding when something happens, and it pulls in two directions at once. You have to search for evidence across a very long range, and you have to understand individual events in fine detail, all under a limited visual budget. Existing agentic methods attack this, but they still lean largely on policies and tool capabilities that were fixed in advance. Whatever the designer wrote down is what the agent gets to work with.
CoEvoWhen lets both of those things change. It is a policy-tool coevolution framework: starting from the agentic reasoning trajectories of a vision language model, it jointly evolves the high-level policy and the executable media tools the model uses. The result is a reusable skill, and producing it does not update the model's parameters at all.
Evolution is driven by an external skill updater. On the policy side, it distils transferable task experience in long-video temporal grounding, and uses that to refine how the agent orchestrates two kinds of observation: long-range ones built from images, and fine-grained ones built from video. On the tool side, the updater uses its coding capabilities to upgrade existing tools or create new ones, adapting them to the job of acquiring evidence from long videos. Once the skill has evolved, the vision language model runs on its own. Guided by the evolved policy, it orchestrates the tools and coordinates image and video observations during inference, without relying on a separate, stronger planning model.
Experiments span five benchmarks and three vision language models. Across them, policy-tool coevolution consistently improves temporal grounding accuracy in ultra-long videos, and it does so while reducing the visual token cost at inference. The evolved skill also carries over. On general long-video question answering, it yields substantial performance gains with no additional task-specific evolution.
Two things usually treated as fixed equipment, the plan and the toolbox, are here treated as things that can be learned together from the model's own attempts. Nothing in the model's weights moves. What improves is the skill around it, and the authors read the gains on grounding, and on question answering it was never evolved for, as evidence that the framework is both effective and general for long-video understanding.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a video hours long and an agent stuck with the tools it was given.
You need an agent to find when an event happens in a video hours long. The event may fill only a tiny fraction of the timeline.
The road ahead
- 1 · The problem ~7 min Finding a brief event in an hours-long video forces a trade between cheap coarse looks that miss it and close looks that cost too much.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Evolve the agent's search policy and its media tools together, from its own graded runs, into a reusable skill while the model's weights stay frozen.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Collect the frozen model's runs, let an external updater refine policy and tools from them, package both as a skill, then run the model with it.
- 6 · The experiment ~6 min You’ll guess: Besides accuracy, what happened to visual token cost at inference once the agent used the evolved skill?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.