VibeEdit: Image Editing with Canvas Instructions

VibeEdit. Image Editing with Canvas Instructions. By Jinjing Zhao and eleven others, from the University of Sydney, Microsoft Research, and six other institutions. Published on the eighth of October.

Text-guided image editing splits a request into two parts, and only one of them suits words. Saying what should change is usually easy. Saying which object or region should change is the cumbersome half, and it gets worse when several objects in the picture look alike. A sentence has to single out the third cup from the left, or the person nearest the window, and hope the model reads it the same way. The difficulty is not describing the edit. It is pointing at its target.

The paper answers with a different interface. Instead of a separate text prompt, users place spatial marks directly on the image, and can add optional short notes beside them. Together those annotations form what the authors call a canvas instruction: one object that says both where to edit and what to change. The editor built to follow them, VibeEdit, handles object addition, removal, replacement, attribute modification and movement, all from the canvas alone.

Training starts from data. The authors construct one point five five million source and target edit pairs, each with object masks and structured edit descriptions, and canvas instructions are rendered from those pairs during training. The model itself is adapted from Qwen-Image-Edit with layer-decoupled conditioning, which encodes the source image and the canvas instruction separately rather than as one input.

Optimisation then runs in two stages. First comes region-weighted supervised fine-tuning. After it comes rubric-guided reinforcement learning, aimed at three things at once: completing the requested edit, making the local edit good, and preserving the regions that were never meant to change.

Evaluation uses an independently constructed, human-curated benchmark of four hundred and nineteen cases, built to emphasise picking the right target among similar objects. On it, VibeEdit reaches a VLM rubric score of seventy-nine point nine. Its PSNR outside the edited region is thirty-two point eight decibels.

The comparison is FireRed, the highest-scoring text-instructed baseline in the authors' evaluation. FireRed scores sixty-seven point four on the rubric, and twenty-four decibels outside the region. The second number is the one that speaks to preservation: the canvas-instructed editor leaves far more of the untouched image as it was.

Underneath the results sits a simple division of labour. Words stay in charge of what the edit is, where they were always comfortable, and marks on the image take over where it happens. Once the target is drawn rather than described, the hardest case for a text prompt, a crowd of lookalike objects, becomes the case this benchmark was designed to test.