Flide: Accelerating Decision Inference with Exact State Forking
Bobo Li, Jiajia Song, Hao Fei, Wynne Hsu, Mong Li Lee
National University of Singapore · University of Oxford
Published October 30, 2026 · on the feed November 10, 2026 · ▲ 35 · arXiv 2610.42148
The abstract
Decision models such as Jev have emerged as a highly promising paradigm for LLM agent systems, providing lightweight and reliable categorical reasoning for rapid System One execution. However, existing generation-oriented inference engines fail to support these workloads effectively, suffering from severe prefix recomputation overheads or distorted output probabilities. To address this challenge, we propose Flide, an efficient and exact inference engine that decouples the logical tree structure of a decision request from its physical execution within a single flattened token stream. Through ForkAttention, Flide navigates request dependencies using GPU-resident indices across both standard attention and recurrent Gated DeltaNet layers, guaranteeing in exact arithmetic that every query branch evaluates identically to an isolated forward pass. By expressing branch topologies directly in GPU-resident metadata, Flide accommodates diverse request mixtures with a compact set of pre-recorded execution graphs while enabling expanding agent histories to resume from checkpoints saved in earlier passes. Extensive evaluations on public hybrid and decision models demonstrate that Flide answers queries up to 9.5 times faster than vLLM and SGLang while staying closest to a float32 reference in total variation distance. By unifying physical execution throughput with mathematical exactness, Flide establishes a principled foundation for scalable, latency-critical decision intelligence in agent systems.
The Brief
2:12A short podcast summary of the paper. Written and voiced by AI.
Transcript
Flide. Accelerating Decision Inference with Exact State Forking. By Bobo Li and four others, from the National University of Singapore and the University of Oxford. Published on the thirtieth of October.
Decision models such as Jev have become a promising paradigm for agent systems built on language models. They provide lightweight, reliable categorical reasoning for rapid System One execution: picking among options rather than writing out long text. The engines that serve these models, though, were built for generation. On decision workloads they either suffer severe overheads from recomputing shared prefixes, or they distort the output probabilities. One path is slow, and the other is wrong, and a decision model needs its probabilities to be right.
Flide is an inference engine meant to be both efficient and exact. Its central move is to decouple two things that existing engines tie together: the logical tree structure of a decision request, and the physical execution of that request. Logically a request branches. Physically, Flide runs it within a single flattened token stream.
The component that makes this work is ForkAttention. It navigates the dependencies between parts of a request using indices that live on the GPU, and it does so across both standard attention layers and recurrent Gated DeltaNet layers. The guarantee is strong: in exact arithmetic, every query branch evaluates identically to an isolated forward pass. Because branch topologies are expressed directly in GPU-resident metadata, Flide can handle diverse mixtures of requests with a compact set of pre-recorded execution graphs. It also lets agent histories that keep growing resume from checkpoints saved in earlier passes, rather than starting over.
Evaluations cover public hybrid models and decision models. Flide answers queries up to nine point five times faster than vLLM and SGLang. It also stays closest to a float32 reference, measured by total variation distance, so the speed does not come at the expense of the probabilities.
Prefix recomputation and probability distortion had looked like a trade-off, one or the other. Flide treats them as a single problem with a single cause: the logical shape of a request being forced into the physical shape of the execution. Separate the two, and the authors argue that throughput and mathematical exactness can be had together, laying a foundation for latency-critical decision intelligence in agent systems.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Decision models answer in one pass. The engines serving them were built for something else.
You serve a model that reads a long agent trajectory and answers several multiple-choice questions about it, returning a probability for each option. Strict thresholds act on those probabilities, so they must be both fast and precise.
The road ahead
- 1 · The problem ~6 min Decision models answer several multiple-choice questions about one long context, and engines built for text generation make that either slow or subtly wrong.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Treat a request as a tree: compute the shared trunk once, run all branches in one flat stream, and give each branch exactly what it would see alone.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Flide flattens a round into one stream, computes each trunk once, forks every question from it, and resumes growing histories from checkpoints.
- 6 · The experiment ~6 min You’ll guess: Against vLLM and SGLang, how did Flide do on speed and on matching a float32 reference distribution?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.