JLD: Perceptual Distance Through A Jacobian Lens

JLD. Perceptual Distance Through A Jacobian Lens. By Shreshth Saini and two others, from the University of Texas at Austin, Google, and one other institution. Published on the fifth of October. Its code repository is public and real: about thirty source files.

Image compression, restoration, and generation all need a way to measure how different two images look to a person. Pixel error is the simple answer, and it ignores how people see. The most accurate perceptual distances take the opposite route: they are fitted to human judgments. That ties them to a fixed set of data and a fixed resolution. The abstract gives a concrete case. Double the image resolution, and the correlation of DISTS with human scores on TID2013 drops from zero point eight one five to zero point seven one seven. A metric fitted to people's judgments at one size of image carries that size with it, and becomes less faithful when the images change.

The Jacobian Lens Distance, or JLD, takes its perceptual geometry from a frozen vision encoder instead of from human labels. It pairs two things the encoder already has. Early patch features are local. Later representations capture perceptual sensitivity. JLD uses the later stages to decide which directions in the early features matter. The result keeps the spatial precision of the first layers while borrowing the judgment of the last ones, and no human score enters at any point.

Concretely, it uses the encoder's Jacobian to find the directions in early feature space that most strongly affect the encoder's output. Averaging the product of the Jacobian's transpose with itself gives a fixed metric tensor, which the authors call the Jacobian lens. It is fitted once, from one hundred unlabeled images, in about thirty-five seconds. Locally, this construction defines a pullback metric in pixel space. That gives JLD a clear geometric interpretation, one that can be analyzed directly on real images rather than taken on trust.

Across four standard perceptual databases, JLD reaches state-of-the-art performance and consistently outperforms LPIPS, DISTS, PieAPP, and DreamSim. The resolution test that troubled DISTS is repeated. On TID2013, when the resolution doubles, the correlation of JLD's lens term barely moves: from zero point eight five zero to zero point eight four five. Set beside the earlier drop for DISTS, from zero point eight one five to zero point seven one seven, the contrast is the point of the design: a geometry taken from the encoder is not tied to the data or the resolution of a human study.

A faster variant, JLD-fast, runs four times faster than LPIPS-VGG while reaching a mean correlation of zero point nine one one. And the approach extends naturally to video. On Waterloo IVC 4K, JLD reaches a correlation of zero point seven eight six, against zero point six one one for VMAF.

The shift is in where perceptual judgment comes from. Instead of learning it from people at one resolution, JLD reads it off the sensitivities of an encoder that already exists, using a lens built from a hundred images and no labels at all. Because the result is a metric with a geometric meaning, it can be inspected, not only scored.