Personalized Image Generation with Reasoning and Reflection

Personalized Image Generation with Reasoning and Reflection. By Bo Ni and twelve others, from Vanderbilt University, Adobe Systems, and two other institutions. Published on the thirtieth of September. No code repository is linked to it.

Personalized image generation has stayed narrowly focused on one thing: conditional synthesis from curated visual exemplars. Give the model a few chosen pictures, and it makes more in their image. What that leaves out is who the user actually is. A user's real personal context is much richer than a few pictures. It takes in reviews, posts, images, captions and metadata, accumulated over time. A generator that is truly personalized should draw on that whole history, and produce images aligned with the user's lifestyle and aesthetic preferences. The paper sets out to build a way of testing whether one does.

The authors introduce what they describe as the first unified benchmark for personalized image generation from user histories. Its evaluation protocol runs along several axes at once. Fidelity to the target. Visual quality. User distinguishability, meaning whether one user's output can be told apart from another's. Semantic alignment with the user's history. And utility specific to each task. Taken together, the axes ask whether an image is good, and also whether it is this person's.

The benchmark is grounded in real-world e-commerce and social media settings, and has two complementary tasks. Personalized Scene Generation places a given object in a scene that reflects a user's preferences and lifestyle. Its motivation is personalized product presentation: showing the same item differently to different people. Personalized Creative Generation produces a novel image on a specified topic that stays faithful to a user's aesthetic and visual identity. Its motivation is content creation on social media.

Alongside the benchmark comes a method, PEARL. It couples a multimodal reasoner with a frozen image generator, in an interleaved loop of reasoning and reflection that is optimized with a differential data reward. The image generator stays frozen throughout. Across both tasks, PEARL outperforms strong baselines, with an average improvement of fifteen percent across the personalization metrics.

What changes is the thing personalization is conditioned on. In place of a handful of chosen exemplars, the input becomes a person's accumulated history: what they reviewed, what they posted, how they captioned it. The benchmark then checks, among other things, whether the images made for one user are recognisably theirs and not someone else's. PEARL is the method the authors put up against that bar, and the two tasks give it room to be tested in a shop window and on a social feed alike.