FlowTranslator: Training-Free In-Image Machine Translation with Flow-Based Editing

FlowTranslator. Training-Free In-Image Machine Translation with Flow-Based Editing. By Zhixian Li and eight others, from Fudan University and the Shanghai Key Laboratory of Multimodal Embodied AI. Published on the third of November.

In-image machine translation takes the text that appears inside an image and replaces it with its translation, while keeping everything around it intact: the background, the layout, the look of the scene. Task-specific models for this already perform convincingly. Their cost is data. They need massive collections of paired images, one before translation and one after, to learn from.

The authors start from an observation about text-to-image models. Advanced ones have reached a practically usable level of multilingual text generation, and the argument is that this capability can be transferred to in-image translation instead of training a dedicated model. FlowTranslator is the result: a framework for the task that needs no training at all.

The pipeline begins outside the image generator. An OCR model reads the text, and an LLM produces a translation prompt, which is injected into a diffusion model built on text-to-image generation. Getting the translated text to come out accurate and natural is then handled by two controls over the generation trajectory. The first is Region Coupling Release, or RCR. Early in the trajectory, it partially decorrelates the noise of the target branch inside the text region. The second is Coupling Expansion, or CE. In the middle of the trajectory, it amplifies the edit displacement that is beginning to emerge. One acts early and locally to loosen the text region; the other acts later to strengthen the change that is forming.

Scoring follows the EasyOCR read-back protocol, which reads the generated text back out of the image. Under it, FlowTranslator reaches average PRIM BLEU and COMET scores of twenty point six and fifty-seven point two respectively, substantially ahead of UniTranslator, a strong baseline that is trained for the task. The harder test uses ScenePair, whole-image photographs that are more realistic, translated into Chinese. There it reaches forty point one percent exact-match accuracy and surpasses existing editing systems. Some of those systems, the authors note, only handle the text-rendering stage rather than the complete translation task.

The conclusion drawn is that flow-based editing makes a strong baseline for in-image translation, and one that is fully training-free. The shift is in where the capability comes from. Instead of learning translation-in-place from paired examples, the system borrows the multilingual text rendering a general image generator already has, and steers its trajectory so that the new text lands where the old text was.