Real In, Real Out: What If We Only Use Real Data for Scene Text Editing?

Real In, Real Out. What If We Only Use Real Data for Scene Text Editing?. By Xingsong Ye and four others, from Fudan University, the Shanghai Key Laboratory of Multimodal Embodied AI, and one other institution. Published on the twentieth of October. Its code repository is public but slight: eight source files.

Scene Text Editing changes the words in an image while keeping their original style intact. It has real practical value across many applications. Training it normally needs paired images, the same scene before and after the edit, and pairs like that are hard to find in the wild. Recent approaches have leaned heavily on synthetic paired data and on complex disentanglement strategies. The result struggles to capture how diverse and complex natural scenes are, and often produces artifacts or hallucinations. The weakness, in other words, sits in the training data itself. A model that learns its editing from synthetic pairs carries the gap between synthetic and natural images with it when it meets a real scene, and a more elaborate disentanglement strategy inside the model does not change where its pairs came from.

The paper challenges that practice with a direct question. What if every real sample were turned into its own paired supervision? Take a real text image, disrupt it in a way that destroys what the text says but keeps how it looks, and train the model to restore the original. The original image then serves as the target, and no synthetic pair is needed.

The authors explore several ways to disrupt the original image, including cropping, shuffling and masking, to build self-paired training data. On top of a conditional diffusion architecture they build a strong baseline called TextRIRO. The disruption that works best combines character-column shuffling with image cropping, called Shuffle and Crop. It effectively destroys text semantics while preserving key style cues, which lets the model learn editing behaviour directly from the real text image. The balance matters in both directions. Disrupt too little and the model can simply copy the text back. Disrupt too much and the style it is meant to preserve is gone as well. Shuffle and Crop is the combination the experiments found to sit between those two failures.

With this concise, real-oriented design, TextRIRO achieves promising performance in both accuracy and visual realism across real-world benchmarks, substantially narrowing the gap between synthetic and real domains.

The method also produces a resource. Using TextRIRO, the authors build TextRIRO-3M, a large-scale realistic dataset for long-text recognition, made by concatenating edited images with original ones. Training recognition models on TextRIRO-3M significantly improves recognition accuracy, which the authors take as evidence that the editor yields useful training material for downstream tasks.

The whole approach turns on one substitution. In place of synthetic pairs that never quite look like the real world, each real image is broken in a controlled way and becomes its own training pair. Real data goes in, and realistic edits come out. The authors also show that the edits themselves are good enough to train other models on, which turns the editor into a data source in its own right.