Fractional State Space Transition for Long Sequence Modeling
Ivan Kobyzev, Abbas Ghaddar, Ali Nasiri-Sarvi, Lifeng Shang, Yufei Cui
Huawei Noah's Ark Lab
Published September 28, 2026 · on the feed September 30, 2026 · ▲ 10 · arXiv 2609.36314
The abstract
State Space Models (SSMs) compress sequence history into a bounded recurrent state, making the resulting memory law a central architectural choice for long-context performance. Most modern SSMs rely on ODE-based dynamics that lead to exponential forgetting, limiting their ability to retain information over broad temporal ranges. We introduce FRAC, a selective SSM architecture derived from fractional dynamics that replaces this exponential decay with power-law long memory. To make fractional dynamics practical, FRAC approximates the heavy-tailed target kernel with a finite-state, log-spaced sum of exponential modes. This construction turns fractional memory into an efficient recurrent module with parallel training and prefill, while retaining bounded-state autoregressive decoding. Extensive experiments, including 1.3B-parameter language modeling, demonstrate that FRAC consistently improves long-context performance over state-of-the-art SSM baselines while staying competitive on short-context. These results show that fractional dynamics provide a practical and effective prior for long-context SSMs.
The Brief
1:51A short podcast summary of the paper. Written and voiced by AI.
Transcript
Fractional State Space Transition for Long Sequence Modeling. By Ivan Kobyzev and four others, from Huawei Noah's Ark Lab. Published on the twenty-eighth of September. Its code repository is public and real: nineteen source files.
State space models compress the history of a sequence into a bounded recurrent state. That makes the memory law, the rule by which old information fades, a central architectural choice for long-context performance. Most modern state space models rely on dynamics derived from ordinary differential equations, and those dynamics lead to exponential forgetting. Information decays quickly, which limits how much the model can retain across broad temporal ranges.
FRAC is a selective state space architecture derived instead from fractional dynamics. Its change is to the memory law itself. Exponential decay is replaced with power-law long memory, so that older information fades along a heavy tail rather than dropping away.
Fractional dynamics are not practical as they stand, so FRAC approximates the heavy-tailed target kernel with a finite set of states: a sum of exponential modes, spaced logarithmically. Many exponentials at different scales, added together, stand in for the one heavy tail. That construction turns fractional memory into an efficient recurrent module. Training and prefill can run in parallel, and autoregressive decoding still keeps a bounded state.
The experiments are extensive and reach language modeling at one point three billion parameters. Across them, FRAC consistently improves long-context performance over state-of-the-art state space baselines, while staying competitive on short-context tasks.
The authors read these results as showing that fractional dynamics are a practical and effective prior for long-context state space models. What changes is the shape of forgetting, not the size of the state, and FRAC makes that change without giving up bounded-state decoding.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a memory that fades too fast.
You are designing a sequence model that squeezes all of its history into a fixed-size state, so it runs fast on very long inputs.
The road ahead
- 1 · The problem ~7 min A model that squeezes its history into a fixed-size state forgets old events exponentially fast, even when long documents still need them.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Swap exponential forgetting for power-law memory from fractional dynamics, and fit it into a fixed state with many exponential decays at log-spaced timescales.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Build power-law memory from log-spaced exponential modes, train and prefill with a parallel scan, and decode step by step with a bounded state.
- 6 · The experiment ~6 min You’ll guess: FRAC improves long-context performance over state-of-the-art SSM baselines. How does it fare against them on short-context tasks?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.