Diffs vs. Whole Files: An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models

Diffs vs. Whole Files. An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models. By Andrej Andrejev, working independently. Published on the fifth of September. Its code repository is worked out: about sixty source files, with a readme and a licence.

A language model that edits code can produce its answer in one of two ways. In direct generation it writes out the whole modified file in one shot. In iterative diff-based generation, which the authors call steps, it emits a sequence of small search-and-replace edits, applied one at a time, until it signals that it is done or a step budget runs out. The diff-based regime has obvious appeal. It mirrors how developers actually edit, and it ought to need far fewer generated tokens per turn.

The finding runs against that intuition. For the models and data studied here, direct generation substantially outperforms diff-based generation, on every metric measured.

Two code models were trained in both regimes on a shared Flutter and Dart dataset. One is a one hundred million parameter model trained from scratch, named Rainbow-Pony-100M. The other is a fine-tuned Qwen2.5-Coder at zero point five billion parameters. That makes four models, each evaluated on a held-out set of roughly one thousand seven hundred and ninety tasks.

Direct generation wins on the pass rate for compilation and static analysis, on bits per byte, on character-level similarity to the reference, and in blinded ratings by an LLM judge of goal fulfilment, correctness and code quality. The gap survives two controls. It persists when task difficulty is matched through a matched-ID comparison, and it persists when the comparison is restricted to code that compiles on both sides.

The authors then look at the conditions where diffs do win, and trace them to one mechanism that does not depend on architecture. Diff-based generation is competitive on edits that are short and spatially localized. Its wins at the level of task categories concentrate in exactly two categories, refactoring and fixes for error handling or edge cases, which are the two with the lowest mean number of edit steps in the dataset. The authors name this property task locality.

The scope is bounded by the setup. The evidence covers two small models, one language pairing in Flutter and Dart, and one dataset. Within that, the result is broad, holding across every metric and both controls, and the exception is precise.

Task locality is offered as a way to decide between the regimes. When the change a task needs is small and confined to one place, the edit-based approach can hold its own. When it is not, writing the whole file wins. The authors discuss what this implies for when an edit-based training regime is, and is not, the right choice for a code-editing model. The question it leaves a practitioner with is not which regime is better in general, but how local the edits are that a model will be asked to make. Where they are local, diffs remain an option. Everywhere else in this study, the whole file was the stronger output.