T-DIG: Text-Conditioned Diversity for Image Generation

T-DIG. Text-Conditioned Diversity for Image Generation. By Simran Khanuja and three others, from Carnegie Mellon University and Northeastern University. Published on the twenty-first of October.

Text-to-image platforms usually return several images for each prompt. The point is to let users explore different visual readings of what they typed and pick the one that matches what they meant. Yet the models behind them are usually optimized for the quality of a single image, and that comes at the cost of diversity across images, which limits the exploring. Diversity is also contextual. One user wants variation in the background, another wants it in artistic style, and the same user might later want variation in clothing. The images should differ along whichever axis the user cares about, and a fixed notion of diversity cannot know which one that is.

T-DIG, short for text-conditioned diversity for image generation, lets users say which axis they mean. It is a framework for both generation and editing. Users type a prompt, look at the default generations, ask for variations in natural language, and keep refining until they are satisfied. Under the hood, the core is an LLM policy. Given a request, it produces K prompts or edit instructions, each conditioned on the requested axis, and hands them to a generator or editor that stays frozen.

The framework is iterative and built from multiple modules, so a session is a loop rather than a single request: prompt, inspect, ask for variation, refine, and repeat. The policy at its centre is trained with GRPO. Its reward is for text-conditioned diversity, meaning the images should differ along the axis the user named, and it carries alignment penalties, so that the variations do not drift away from the original prompt. The frozen generator is what makes the approach broad. Because nothing inside the image model changes, T-DIG can work with open-weight models, with distilled models, and with proprietary ones.

The authors also release what they describe as the first dataset for text-conditioned prompt diversity in image generation. Its training and validation splits are built from axes written by humans. Its test set is drawn from real user interactions with T-DIG itself.

On text-conditioned diversity, T-DIG far exceeds the baselines. It does so while preserving alignment with the base prompt, across a variety of metrics and in human judgements. Those are the two pressures the training balanced: spread the images along the requested axis, and keep each one faithful to what the user first asked for. The broader demonstration is that user-directed exploration of diversity can be added to a wide range of image generation models, open and closed, without needing access to their weights.

The design choice that carries this work is keeping the image model untouched. All the steering happens in text, in the prompts a trained policy writes, so the same layer can sit in front of almost any generator. Diversity stops being a single number to maximize and becomes something the user asks for by name.