Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs

Before It Fades. Reinforcing Temporal Representations at Inference Time in VideoLLMs. By Youngwoo Shin and three others, from KAIST. Published on the first of October. Its code repository is public but slight: nine source files.

VideoLLMs receive frames in sequential order and interpret how visual content evolves along the temporal axis. Even so, temporal reasoning remains a persistent weakness across architectures. One simple probe exposes it. Reverse the frame order of a video, a transformation that should invert any temporal answer, and the final prediction often stays exactly the same. A model that truly followed the order of events would have to change its answer when the order is flipped. One that answers the same way either way has, at least by the output, lost track of which came first.

Before fixing the failure, the authors locate it. They define a temporal divergence vector, called tau sub l: the layer-wise difference in representation induced by reversing temporal order. Tracking its magnitude across layers reveals a consistent profile. Divergence peaks at intermediate layers, then progressively diminishes toward the output. They confirm that the peak is specific to temporal reasoning, and that it is functionally critical for predictions. The picture that emerges is that VideoLLMs do acquire temporal information at intermediate layers. They fail to maintain it to the output. That distinction matters for what a fix should look like. A model that never perceived temporal order would need to learn it. A model that perceives it and then lets it fade needs something closer to preservation.

That progressive fading motivates the method, Temporal Activation Injection, or TAI. For each input, TAI extracts the divergence vector at the peak of the profile, and reinjects it into subsequent layers, following the decay that was measured. No training is involved. The signal the model already computed is carried forward to where it had been lost.

TAI consistently improves temporal reasoning across three VideoLLMs and four benchmarks, with negligible impact on non-temporal tasks. Both halves of that sentence carry weight. The gain shows up across several models and several benchmarks rather than one pairing. And the lack of damage elsewhere is consistent with the earlier check, that the peak being amplified is specific to temporal reasoning.

A diagnosis does most of the work here. Once the profile shows temporal information arriving in the middle of the network and fading before the end, the repair writes itself: take the signal from where it is strongest and put it back where it was disappearing.