Sparse Autoencoders Are a Shared Coordinate System for Symbolic Matching

Sparse Autoencoders Are a Shared Coordinate System for Symbolic Matching. By Xiaocong Yang and four others, from the University of Illinois Urbana-Champaign. Published on the fourteenth of October. Its code repository is public and real: about fifty-five source files.

Large language models pack broad knowledge into one set of weights, but they are hard to interpret and their reasoning is hard to trust, a real concern in high-stakes uses. Neuro-symbolic models offer an interpretable architecture and a capacity for System 2 reasoning, and could be a promising alternative if they were built at the scale of language models. Their symbolic backbone, though, is usually constructed by human effort. That bottlenecks them at the scope of their symbol component, and tends to keep them domain-specific.

A data-driven route would sidestep the hand building: construct a larger symbolic system by aligning and merging existing ones. Before anything can be merged, though, its parts have to be matched. This work targets that first, foundational step: matching units across symbolic systems that were built differently from one another.

SAE-IDF uses a sparse autoencoder as a shared coordinate system. Symbols from different systems are mapped into the same sparse feature space, and matched there. However the original systems were organised, their symbols end up described in one common set of coordinates.

Each symbol is encoded in its native context into a sparse code by the autoencoder. Every dimension of that code is then re-weighted by its rarity, and the re-weighted codes are compared across systems to find matches.

Evaluation covers twelve tasks across six kinds of symbolic system. With Gemma-2-9B, SAE-IDF beats the same model's dense baseline on every task. With Llama-3.1-8B, it wins on all but one. Margins reach as high as forty-five points, and the advantage is consistent across autoencoder layers and configurations, so it does not hinge on one lucky setting.

The matches are also interpretable. Every dimension of the sparse code carries a human-readable meaning, so each match breaks down into named concepts, the features. A match is not just a similarity score: it comes with a list of reasons that a person can read.

Matching is a modest first step toward a bigger goal. The authors see it as the opening move toward a universal symbolic backbone, and the sparse autoencoder supplies what that requires: a common space where symbols can be compared, and where every comparison can be read.