ARC-SORT: Adaptive, Robust, Compensated Multi-Object Tracking with Interacting Multiple Models

ARC-SORT. Adaptive, Robust, Compensated Multi-Object Tracking with Interacting Multiple Models. By Girish Kumar Adari, from Athlitix. Published on the fifteenth of October. Its code repository is worked out: about fifty-five source files, with a readme and a licence.

Multi-object tracking in sports broadcasts is hard for two reasons at once. Athletes move fast and change direction abruptly. And while they do, the camera is panning and zooming. A tracker's prediction of where someone will be next is therefore sometimes dependable and sometimes badly off, and the tracker cannot simply assume one or the other. Two players run toward each other and cross. The camera swings to follow the ball. A winger steps out of shot and returns a few seconds later. Each of those moments asks a different question of the tracker, and a single fixed rule for matching detections to tracks answers all of them the same way.

ARC-SORT, short for Adaptive, Robust, Compensated SORT, is a tracking-by-detection tracker that trains nothing and runs in real time. Its design rule fits in one sentence: the tracker measures its own uncertainty, and that measurement decides how it acts. The components that follow each apply the rule, or its limits, to a different moment in tracking. None of them needs training. What changes from one situation to the next is not a learned weight, but how much the tracker trusts what it currently believes.

The first moment is matching. An Interacting Multiple Model Kalman filter gives every track a predictive covariance, and that covariance becomes a per-track search region. The less reliable a track's motion prediction, the larger the region in which a detection may match it. This is an uncertainty-adaptive Expansion IoU, or EIoU. On identical inputs, it is this continuous coupling that matters. The same covariance used as a hard Mahalanobis gate hurts at every threshold the author swept, and richer summaries of the covariance add nothing measurable.

The second moment is a crossing. When two tracks cross, the assignment cost itself reports whether the decision was close. ARC-SORT keeps such pairs under review, and applies at most one corrective identity swap once appearance evidence from after the crossing shows the first decision was wrong. That adds one point of HOTA, higher order tracking accuracy, on both the SportsMOT and DanceTrack test sets, at under a millisecond per frame. The third moment is absence. When a player leaves the frame for seconds, motion extrapolation carries no information, so the offline re-identification drops the usual motion gate and links tracklets by appearance alone.

On the SportsMOT test set, ARC-SORT reaches eighty-one point eight two HOTA. It has the fewest identity switches of the sixty-nine leaderboard entries as of August two thousand twenty-six, and its multi-object tracking accuracy, MOTA, is ninety-seven point zero nine, tied for third.

On the DanceTrack test set it reaches sixty-two point one HOTA, with the same association settings. Given Deep-EIoU's public detections and embeddings, it scores eighty point three one HOTA, three point one above Deep-EIoU itself. And it runs at about fifty frames per second on a laptop RTX GPU.

Code will be released. The thread through the design is self-knowledge. How reliable the tracker believes its own prediction to be sets how far it searches and whether it revisits a decision, and where motion carries no information at all, motion is simply set aside.

ARC-SORT. Adaptive, Robust, Compensated Multi-Object Tracking with Interacting Multiple Models. By Girish Kumar Adari, from Athlitix. Published on the fifteenth of October. This is the long version: the whole paper rather than the abstract.

Picture a basketball broadcast through a tracker's eyes. A guard sprints, cuts, collides with a defender, and the camera pans and zooms over all of it. A tracking-by-detection system links the boxes a detector finds in every frame into tracks that keep their identities. The SORT family does that with a Kalman filter, which predicts where each track will be, and an overlap cost between predicted and detected boxes. Sport breaks both halves: athletes violate the constant-velocity assumption, and the camera moves on top. The usual response is a fixed tolerance margin, or motion learned per domain. Girish Kumar Adari, of Athlitix, points somewhere else. Two quantities that say how far a tracker should trust itself are computed every frame and then thrown away.

The first is the filter's predictive covariance: how far its motion prediction can be trusted. Overlap-based pipelines either ignore it when matching, or spend it as a hard Mahalanobis veto that rejects any pair lying too many standard deviations away. The second is the assignment's own cost margin: how nearly it preferred the swapped pairing. Nothing in a standard tracker separates a confident match from a coin flip. ARC-SORT is built on one principle: a tracker should let its own uncertainty govern its own decisions. It applies that rule to how widely to search, to how firmly to commit at a crossing, and to long gaps where motion carries no information at all. It trains nothing, and works only on the association step.

Motion comes from an Interacting Multiple Model filter running three hypotheses side by side: constant velocity, constant acceleration for the box centre, and a coordinated turn with its own turn rate. Each frame the models are mixed and reweighted by how well they explain the data, and the combined covariance includes the spread between them, so disagreement counts as uncertainty. A camera-motion estimate warps every prediction into the current frame. The core step reads the positional standard deviation in each direction, divides by the predicted box width and height, and keeps the larger. Doubled and clipped between a floor and a ceiling, that single number is the track's expansion factor. Both boxes grow by it before overlap is computed, so the matching region covers about two standard deviations of centre error in units of box size. A track mid-manoeuvre casts a wide net. A confident one keeps a tight gate. Appearance similarity joins the score, but only where boxes already overlap: it refines plausible matches and never creates one.

Crossings get a second mechanism. For any two accepted matches, the tracker reads off how much total score the assignment would lose if the two identities were exchanged. When that margin is small, the pair is entangled. The tracker snapshots both tracks' appearance before any post-crossing update can contaminate it, then on each later frame adds up how much each track's new embedding resembles its own old snapshot rather than the other's. The episode closes thirty frames after the last entanglement. If the evidence is persistently negative, the two tracks exchange identities, at most once per episode, without touching any filter state. The third stage is offline. On football validation sequences, sixty-four percent of identity switches follow gaps of more than twenty frames, almost all of them players leaving the shot entirely. Over such gaps the covariance has diverged, so the merge links tracklets by average appearance alone and drops the customary gate that extrapolates terminal velocity across the gap.

The central experiment is a ladder on SportsMOT validation: identical cached inputs, the same covariance offered to every way of using it. As a strict veto, it collapses HOTA, the higher order tracking accuracy, to seventy-one point two six, and doubles identity switches. A fixed expansion reaches seventy-seven point zero four. A binary velocity switch recovers about a fifth of the gap. Continuous adaptation reaches seventy-seven point nine seven. Richer versions add nothing: separate factors per axis are byte-identical on forty-one of forty-five sequences, and a distributional blend is statistically indistinguishable. One scalar, spent as tolerance and not as veto, captures what the covariance offers.

The comparison that best tests the long-horizon claim holds detections fixed. On Deep-EIoU's exact public detections and embeddings, with the merge still gated, ARC-SORT scores seventy-five point six seven on the test server, below Deep-EIoU's own seventy-seven point two. In-view machinery cannot reach an out-of-view failure. Removing the gate lifts it to eighty point three one. Deferred commitment then adds one full HOTA point on both the SportsMOT and DanceTrack test servers. On its own detector stack the full tracker reaches eighty-one point eight two, sixth of sixty-nine entries, with the fewest identity switches on the leaderboard despite the lowest detection accuracy in its comparison table.

The paper has a short limitations paragraph. Deferred commitment handles pairs only and makes a single swap, and its gain depends on the embedding model, carrying over at between fifty-three and one hundred and twenty percent of the validation gain. The long-horizon stage is offline. Public model-zoo detection caps the score, and on both benchmarks trained or foundation-model systems still score higher. The appendix adds something the summary does not. A ground-truth audit of swaps on SportsMOT found eleven verified correct and none verified wrong, but on the denser DanceTrack it found eleven correct, twenty-four incorrect and fifty-three undecidable, so there the verdict rests on sequence-level gains alone.

The idea to carry away is that a tracker already knows when it is guessing. Spend that uncertainty as tolerance: widen the search exactly when the prediction is weak. Spend it as deferral: when a crossing was decided on a coin flip, keep the case open and let clean evidence settle it. And recognise where it runs out: after a multi-second camera exit, motion knows nothing, so stop letting it veto appearance. None of it is learned, and the whole thing runs at about fifty frames per second on a laptop graphics card.