OmniTaskonomy: When Does Visual Generation Improve Visual Understanding?

OmniTaskonomy. When Does Visual Generation Improve Visual Understanding?. By Jiaxin Ge and fifteen others, from the University of California, Berkeley, Duke University, and four other institutions. Published on the twenty-ninth of September. Its code repository is worked out: about seven hundred source files, with a readme.

Training a model to generate images could plausibly teach it rich perceptual skills: geometry, spatial relationships, what counts as an object. Whether that actually helps it understand images has been unclear. This work asks when, and how, supervision from visual generation improves visual understanding. Its answer has two parts. Under the correct recipe, generation training does improve understanding, and the gains grow as more generation data is used. But the benefit is selective. Which understanding skills improve depends on which generation task was trained.

The first part comes from controlled pairs. Each pair poses the same underlying problem twice, once as image-to-image generation and once as image-to-text understanding, so the only thing that differs is the output modality. Training on the image-to-image side improves downstream image-to-text performance, and the improvement increases with the amount of image-to-image training data. The design matters here. Because each pair shares its problem and differs only in what the model is asked to output, a gain on the understanding side can be traced to the generation training rather than to a change in the task itself. And the trend with data size means the effect is not a one-off: feeding in more generation examples brings a larger benefit downstream.

To look beyond matched pairs, the authors build OmniTaskonomy, a unified taxonomy of nineteen generation tasks and twenty-five understanding capabilities, and map transfer between them. Some links are intuitive. Depth prediction improves metric 3D reasoning. Object pointing improves counting. Jigsaw reconstruction improves 2D ordering. Others are surprising. Two point five D segmentation improves category recognition, and Z-depth prediction improves localisation.

One further analysis probes why. The authors measure gradient alignment between generation and understanding tasks, and find that stronger alignment goes with larger transfer gains downstream. In other words, where training on a generation task pushes the model's parameters in a direction similar to training on an understanding task, the generation task tends to help that understanding capability more.

Several limits come with the claim. The positive result for paired tasks holds under the correct recipe, which means generation training is not a benefit automatically; how it is done matters. The transfer map is selective and depends on the task, so a given generation task helps some capabilities and not others. And the gradient-alignment finding is an association. It says that stronger alignment comes with bigger gains, not that alignment alone produces them.

Rather than a yes or a no, the result is a map. Generation is presented as a rich source of supervision for understanding, and the taxonomy is offered as a roadmap: pick the generation tasks whose benefits land where you need them, and put them in the right training curriculum.