VepAgent: Bridging Causal-Transition via Tool-Augmented Reinforcement Learning for Video Event Prediction

VepAgent. Bridging Causal-Transition via Tool-Augmented Reinforcement Learning for Video Event Prediction. By Qiutong Chen and twelve others, from Nankai University, Carnegie Mellon University, and five other institutions. Published on the fifth of October. No code repository is linked to it.

Multimodal large language models have shown real potential in understanding video. Video Event Prediction asks something different of them: given what has been seen, say what happens next. Here the authors identify two habits that get in the way. These models lean on retrospective summarization, and on priors drawn from text. Both limit their ability to bridge causal transitions that were never observed, the steps between the last thing on screen and the event still to come. Prior methods, in this account, passively project future trajectories from historical dependencies.

VepAgent is an agentic framework that combines causal-transition reasoning with tool-augmented reinforcement learning. Its central move is to model the logical progression explicitly, starting from the terminal observed state and working forward to the future event, rather than extrapolating from the history as a whole.

There are three parts. First, a dataset: futurebench-4K, a chain-of-thought collection for supervised fine-tuning, which structures the deduction of unobserved intermediate states so the model learns to fill the causal gap. Second, a diagnostic tool library offering state tracking, frame retrieval, and region magnification. During inference the agent can call on these dynamically, to recover spatio-temporal evidence it is missing and to resolve visual ambiguities.

Third, a composite reward for the reinforcement learning stage. It jointly optimizes prediction accuracy, causal coherence, and what the authors call a reliable prior. The stated aim of that combination is to push the agent to rely on genuine visual grounding rather than on superficial textual similarities. The reward is what holds the other parts to the video, so that a plausible sounding answer is not enough on its own.

Evaluations use the FutureBench and NEPBench datasets. The authors report state-of-the-art performance, significantly outperforming larger multimodal models. They present this as validating the empirical effectiveness of an agentic, future-oriented reasoning paradigm. The abstract gives no specific figures beyond that.

The argument of the paper is about where prediction should start. A model that summarizes the past and then guesses tends to lean on what text makes likely. VepAgent anchors the guess at the final observed state, trains on explicit chains through the unseen middle, lets the agent look again at the frames when the picture is ambiguous, and rewards it for staying grounded in what it actually saw.