BudgetPix: Compute-Adaptive Tokenization for Pixel-Space Image Diffusion

BudgetPix. Compute-Adaptive Tokenization for Pixel-Space Image Diffusion. By Ozgur Kara and six others, from the University of Illinois Urbana-Champaign and Google. Published on the eighth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Most image generation models tokenize uniformly. Every equally sized patch of the image gets exactly the same computational budget, whatever it contains. Two things follow from that static arrangement. The model cannot adapt when the resources available at inference time change, because the cost is fixed. And the tradeoff between quality and cost is worse than it needs to be, since a plain stretch of background receives the same effort as a region full of intricate detail. The effort is spent evenly, and an image is rarely uneven in the same way twice.

BudgetPix replaces that with adaptive tokenization. Compute is allocated dynamically according to visual complexity and spatial layout, so busy regions can receive more and simple ones less. Because the allocation is not fixed, the budget itself becomes something chosen at inference time rather than decided once in training. Both complaints about uniform tokenization are answered by the same change: the effort follows the content, and the total effort can be turned up or down when the image is generated.

The framework has three components, and each one handles a different consequence of letting the token count vary. The first is an adaptive encoder. It maps an image of fixed size to a token sequence of variable length, using an entropy-guided quadtree together with a multi-scale patch embedder. So the image going in has a set size, but the number of tokens describing it does not. The second is a scale-aware decoder, which reconstructs images at a fixed resolution from those multi-scale token sets. The third is a flexible schedule for training and for sampling, which lets pixel-space denoisers operate across varying token counts. Together these are designed to slot into existing pixel-space diffusion architectures, and the result is a single checkpoint that can be run at a wide range of compute budgets.

On text-to-image generation, BudgetPix matches the fidelity of MiniT2I-L at five hundred and twelve pixels square, and of PixelDiT at one thousand and twenty-four pixels square. In both cases it uses just a quarter of the original compute budget.

In class-conditional generation on a MeanFlow backbone, BudgetPix needs only sixty percent of the full compute budget to produce images with almost no loss in quality: the FID rises by a marginal zero point eight points. Assessments by human judges and by VLM judges agree that BudgetPix gives a significantly better tradeoff between quality and efficiency than earlier budget-adaptive baselines.

Cost, in this design, stops being a property of the architecture and becomes a dial. The headline figures make the point from two directions: the same fidelity as full-budget models for a quarter of the compute in one setting, and a barely measurable change in FID at sixty percent in another. One trained model, one checkpoint, and a budget set at the moment of generation, with the compute going where the image is complicated rather than spread evenly across every patch.