LongBanana: An Expert-Verified Benchmark for Long-Context Multi-Reference Image Synthesis

LongBanana. An Expert-Verified Benchmark for Long-Context Multi-Reference Image Synthesis. By Haoxiang Cao and eighteen others, from South China Normal University, the University of British Columbia, and nine other institutions. Published on the fourth of November.

Image generation is entering an age where agents drive it, and this benchmark targets a particular job in that setting: long-context multi-reference image synthesis. A system must retrieve visual evidence that is scattered across many references, reject the distractors, bind entities and their relations correctly, and compose a single coherent image from all of it. To the authors' knowledge, LongBanana is the first benchmark for this task that experts both designed and verified.

It measures how well a system uses dense visual evidence. Scoring runs along three dimensions. Content Fidelity asks whether the things that should be in the image are there. Relational Binding asks whether they stand in the right relations to one another. Global Constraint Satisfaction asks whether the image as a whole meets the requirements placed on it. Each dimension lines up with one of the demands of the task: retrieving the right evidence, binding it correctly, and composing one coherent result.

There are ninety carefully designed tasks, organised by a three-by-three taxonomy. Each task comes with about thirty-two reference images, twenty constraints, and fifty evaluation items. Across the whole benchmark that adds up to four thousand six hundred and seventy atomic evaluation items, every one specified by an expert.

The evaluation is grounded in a graph. Each task is instantiated with its reference images and an entity-relation graph. The obligations on that graph, at the level of a node, an edge, and the whole graph, are then converted into checklist items, which is how the three scoring dimensions get filled. The expert labels were reviewed, and a fifteen percent subset was labelled twice. On it, exact agreement is eighty-nine point four percent, and the quadratic weighted kappa is zero point eight three.

The authors also introduce LongBanana-Gen, an agent-harness framework for long-horizon image generation. Experiments show that even minimal agent harnesses improve over direct generation. A ReAct-Plan-Exec-Reflect workflow, called RPR, yields gains that depend on the harness, by strengthening evidence grounding and repair. Read together, the two results say that structure around the generator matters for this task: wrapping it in a harness at all helps, and how much a more deliberate workflow adds depends on the harness it runs in.

What LongBanana offers is an auditable testbed. Every score traces back through a checklist item to a specific node, edge or graph-level obligation, and those items were set by experts whose agreement is reported. That makes it both a way to measure dense visual evidence use and a shared target for building stronger agent-driven image generation.