IntReal: A Real-World Dataset and Benchmark for Multimodal Interleaved Generation

IntReal. A Real-World Dataset and Benchmark for Multimodal Interleaved Generation. By Shaohan Yu and six others, from Nanjing University, Meituan, and two other institutions. Published on the twenty-eighth of October.

People routinely consume content where text and images are naturally interleaved. Unified multimodal models can already generate text and images together, yet generating that kind of interleaved content remains largely underexplored. Two things hold progress back. There are no well-defined interleaved generation tasks, and no training data to match. And there is no fine-grained evaluation protocol that can reliably diagnose how a model performs across different capability dimensions.

IntReal answers both with two complementary components, one for each limitation: a dataset for the missing tasks and training data, and an evaluation suite for the missing diagnosis. The evaluation half is the IntReal Evaluation Suite. It scores a model along seven complementary dimensions, using specialized expert models to do the judging, which allows diagnosis from coarse to fine. A single overall score can say that a model falls short. Seven separate dimensions are meant to say where.

The data half is IntReal-100K, a real-world dataset of multimodal interleaved samples spanning four major categories. It serves as training data, which is the other gap the authors identified.

The suite itself is validated on three properties. Its metrics should not measure the same thing twice, so redundancy between them is checked. They should match people, so agreement with human judgments is checked. And they should not depend on which expert model does the scoring, so consistency across different expert evaluators is checked.

On all three the suite holds up. Inter-metric redundancy is low: the largest pairwise absolute Spearman correlation is at most zero point four two. Agreement with human judgments reaches eighty-three point three percent accuracy. And consistency across expert evaluators is strong, with Spearman correlation above zero point nine. Taken together, the authors read these as support for both the complementarity and the reliability of the metrics: seven dimensions that mostly measure different things, that track human judgment, and that do not hinge on one particular expert evaluator.

Benchmarking four families of unified multimodal models, the suite reveals distinct strengths and weaknesses across them. Post-training experiments then show that IntReal-100K improves interleaved generation. They also show a second effect: training unimodal generation can benefit interleaved generation too.

That last result is the one that reaches past the benchmark. If training one modality's generation already helps the interleaved case, then, as the authors put it, interleaved generation can benefit from modality-specific optimization, rather than needing joint text-image training at every stage.