SpecKCast: Spectral Kalman Forecasting for Diffusion Acceleration

SpecKCast. Spectral Kalman Forecasting for Diffusion Acceleration. By Javad Rajabi and four others, from the University of Toronto, the Vector Institute, and one other institution. Published on the nineteenth of October. No code repository is linked to it.

Diffusion Transformers generate high-fidelity images and video, but they are expensive to run, because the transformer is evaluated again at every one of many denoising steps. Feature caching cuts the cost by reusing features from one step at another. More recent forecasting methods go further: instead of reusing a feature, they extrapolate it to the steps being skipped.

The authors identify two mismatches in how that forecasting is done. First, it extrapolates the predicted features uniformly, even though different spatial-frequency components of those features change most at different stages of denoising. Second, it overlooks that denoising does not progress at an even pace, so two equal step intervals can contain very different amounts of change. Both mismatches make forecast errors grow as more steps are skipped, and that caps how much acceleration is possible.

SpecKCast is a training-free method that forecasts the model's output in the frequency domain rather than directly. Breaking the output into frequency components answers the first mismatch, because each component can be followed on its own. Measuring time in a way tied to the noise schedule answers the second.

Concretely, SpecKCast applies a two-D discrete cosine transform to the output. Each resulting coefficient is tracked by a lightweight Kalman filter, which follows the coefficient's value and its rate of change. The filter runs along a time coordinate that is specific to each frequency band and derived from the noise schedule, rather than along raw step counts. The transformer itself is called only when the filter's forecast uncertainty rises above a threshold. Otherwise the forecast stands in for it.

Experiments cover four generators: FLUX one dev, Qwen-Image, Wan two point one at fourteen billion parameters, and HunyuanVideo. Across them, SpecKCast achieves a better trade-off between acceleration and fidelity than existing caching and forecasting methods. It reaches up to a six times speedup, and the extra memory it needs is under zero point zero five gibibytes.

Skipping steps is a bet that the skipped computation can be predicted. This method makes the bet more carefully: it predicts each frequency separately, on a clock that follows how denoising actually proceeds, and it lets its own uncertainty decide when the real model has to be consulted. The prediction stops being a fixed schedule and becomes something the filter monitors as it goes.