TIDES: Implicit Time-Awareness in Selective State Space Models

TIDES. Implicit Time-Awareness in Selective State Space Models. By Taylan Soydan and three others, from ETH Zurich and Brown University. Published on the fifth of October. Its code repository is public and real: about thirty source files.

Selective state space models, Mamba being the example the abstract names, earn their strong per-token expressivity in a particular way. The time discretization step is made a learned function of the input. That choice has a side effect. Once the step is learned from the input, it no longer equals the physical time gap between consecutive observations. For irregular time series, where the gap between readings is part of the data, that limits what these models can do. Continuous time state space models, S5 among them, sit on the other side of the trade. They keep the step equal to the physical gap, and so handle irregular timestamps natively, but their dynamics stay linear and time invariant, which limits expressivity per token. One family has the expressivity, the other has the time, and neither has both.

TIDES is a selective state space variant that reconciles the two architectures by changing where the input dependence lives. Instead of putting it on the step size, it moves it onto the diagonal state matrix. The step is then free to go back to meaning what it meant in the first place. It equals the physical time gap, as in S5, so irregular timestamps are handled natively, and the per-token expressivity that makes selective models effective is kept rather than sacrificed. The input dependence has not been removed. It has only been relocated.

The abstract demonstrates this first on a new benchmark of the authors' own, called Fading Flash. It is a compact, controlled diagnostic for sequence models, and it tests two things jointly: whether a model is input dependent, and whether it extrapolates to time gaps outside the distribution it saw. Testing both at once is what lets it isolate the distinct failure modes of current state of the art architectures. TIDES is said to avoid those failure modes by construction, since the step size no longer has to carry both jobs.

On large-scale benchmarks, TIDES sets the new best average rank on UEA time series classification, and on the Physiome ODE regression benchmark. It was also tested on eight datasets that are irregular by nature, drawn from astronomy, agriculture, neuromorphic sensing and climate events. On six of those eight, it matches or exceeds the reference baseline model.

Stated plainly, each family was paying for one property with the other. Selective models bought input dependence with the meaning of the step size. Continuous models bought physical time with linear, time invariant dynamics. Shifting the input dependence onto the state matrix lets the step keep its physical meaning, and the claim is that nothing is given up in exchange: irregular timestamps handled natively, with the expressivity of a selective model still intact.