RTAformer: Relation-Transition-Aware Transformer for Video Scene Graph Generation

RTAformer. Relation-Transition-Aware Transformer for Video Scene Graph Generation. By Jaeyeon Heo and two others, from Sejong University. Published on the fourteenth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Video scene graph generation tries to capture the objects in a video and the relationships between them as those relationships evolve. That asks for two things at once. Object identities persist from frame to frame, while relationships can change abruptly. Existing methods aggregate information across frames to make use of temporal context, but leaning on consistency can hide exactly those changes. A model may keep its previous prediction even after the underlying relation has changed. The authors give this failure mode a name: relational semantic inertia.

RTAformer, the Relation-Transition-Aware Transformer, is a one-stage framework that models the two things separately. Entity persistence is handled on one side, and relation changes on the other, so stability in who is in the scene no longer drags along stability in what they are doing.

Two components carry this. First, RTAformer injects calibrated differences between the relation queries of adjacent frames, and uses them to guide updates to the predicate. In other words, the change from one frame to the next is handed to the model directly, as a signal about the relation, rather than being something it has to notice despite its pull toward consistency. Second, during training, cross-frame instance alignment helps reduce representational drift from one frame to the next. That component serves the persistence side: the same object should keep looking like the same object as the video goes on.

The experiments are run on Action Genome, where RTAformer improves both recall and mean recall over existing methods. Ablation studies, which remove the proposed components to see what each one contributes, support the contribution of each. Comparisons using matched backbones show larger gains on relation-transition frames than on frames without a transition.

That last comparison is the one that speaks to the diagnosis. If the gains concentrate on the moments where a relation actually changes, the model is doing better precisely where inertia would have held a stale prediction in place.