Painting Change Map as guidance for generating painting processes

Painting Change Map as guidance for generating painting processes. By Markus Pobitzer and three others, from the University of Trento. Published on the eighteenth of October. Its code repository is public but slight: six source files.

Digital art generation has shifted from synthesizing a static image to generating video that mimics the human painting process. Existing models can produce visually stunning time-lapses of a painting being created, or being unpainted in reverse. But they largely treat the creative process as a black box. The finished time-lapse may look right, while the person using the model has little say in how it unfolds.

The trouble lies in the conditioning. Current mechanisms tend to fail in one of two directions. Some are too rigid, relying on static geometric cues such as Canny edges or depth maps, which describe a fixed structure rather than a process. Others are too holistic, using a simple scalar progress variable that says how far along the painting is but offers no localized control. Either way, users lack the agency to steer how a painting evolves over time.

The authors argue that real control over generating artwork as video requires a joint understanding of two dimensions. One is where an artist acts. The other is when those actions take place. Their answer is the Painting Change Map, a spatio-temporal control mechanism designed to carry both at once. Edges and depth maps answer only the first question, and a progress scalar answers only the second. The change map is meant to answer them together, for every region of the canvas and every moment of the process.

The Painting Change Map explicitly tracks normalized structural changes across a continuum of one thousand timesteps. That resolution is chosen to be fine enough to capture changes at the level of individual strokes, while remaining computationally tractable. Localized visual differences are mapped into an encoded RGB space, so the map itself can be read as an image of where and when the painting changes. That representation provides fine-grained guidance in both directions: for the forward painting process, and for the reverse, unpainting one.

Extensive benchmarking is reported in support. The method outperforms existing control baselines, and it achieves competitive or superior results when compared with state-of-the-art painting methods. The strongest showing comes under oracle conditions, where the method surpasses all of the baselines by a significant margin. Those results cover a method that offers the user more control than the alternatives, and still holds its own on quality against the dedicated painting methods. The code is publicly available.

Underneath the results is a reframing of what a painting video should be conditioned on. Neither a fixed outline nor a single progress dial describes a process. A map that records where change happens and when it happens does, and that is what hands the steering back to the user. The same map serves a painting being built up and a painting being taken apart.