RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs

RelateAnything. Real-Time Open-Vocabulary Relation Prediction From Any Inputs. By Maëlic Neau, working independently. Published on the eleventh of September. Its code repository is worked out: about two hundred source files, with a readme and a licence.

Object detection has already let go of its fixed taxonomy. Open-vocabulary detectors accept any class list at inference, and promptable segmentation returns regions with no class names at all. In both cases the list of things the model can talk about has left the model and become an input, something a user hands over at the moment of use. Relation prediction has not made that move. Scene-graph models are still trained and evaluated on the fifty or fifty-six predicates of a single annotation style. Their relation head is conditioned on object labels, and that ties the whole model to whichever detector produced those labels.

The paper argues that three obstacles explain this, and that none of them is primarily about modelling. First, there is no relation corpus that is both free-text and verified. Second, an architecture conditioned on labels cannot accept a vocabulary it was never trained on. Third, the standard metric rewards agreement with the training corpus, so a model that knows a larger vocabulary shows up as a regression rather than an improvement. Each obstacle blocks progress on its own, which is why fixing the model alone would not be enough.

RelateAnything is a fifty-three million parameter model built to remove all three at once. It takes an image, plus regions from any source, and returns scored relations over a predicate vocabulary handed to it at inference as plain strings. Object labels are never an input, so the region source can change without retraining. The vocabulary is a bank of text embeddings rather than a learned classifier, which is what lets new predicates arrive as text instead of as new output units. It runs at twenty milliseconds per frame.

Training over nineteen thousand one hundred and three predicates needs two things. It needs positive-unlabeled supervision. And it needs a text encoder that separates antonyms, which contrastive encoders place at a cosine similarity of zero point nine five, close enough to be nearly indistinguishable.

For the supervision, the authors build RA-4M: four hundred and seventy-four thousand images and four point three million relations over ten thousand one hundred and two free-text predicates. The relations are generated against numbered box markers and then verified geometrically, which answers the first obstacle, a corpus that is free-text and checked. For measurement, they build OV-SGG-Bench. It scores six axes across datasets, and it is designed so that the priors standard recall rewards cannot satisfy it, which answers the third obstacle, the metric.

The evaluation covers three cross-dataset benchmarks and a fourth that is zero-shot. There, RelateAnything reaches two point three to three point five times the mean recall of the strongest open-vocabulary method of comparable scale. Those margins survive when a real detector supplies the regions, rather than clean ones. It also leads a three billion parameter VLM scene-graph model on both metrics, while using under two percent of that model's parameters. One further finding concerns measurement itself: in-domain evaluation overstates transfer gains by about five times, which is the metric problem from the start of the paper, now given a size.

The argument runs against a modelling-first reading. Relation prediction stayed closed, on this account, because of data, architecture and metric together, and the paper answers all three in one package: a verified free-text corpus, a model that never sees object labels, and a benchmark that does not reward memorising one annotation style. Model, corpus and benchmark are all public.