Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens
Seongtae Hong, Youngjoon Jang, Jungseob Lee, Chanjun Park, Heuiseok Lim
Korea University · Soongsil University
Published November 5, 2026 · on the feed November 6, 2026 · ▲ 7 · arXiv 2611.07298
The abstract
Embedding models compress variable-length text into fixed-size vector representations, but this compression can entangle semantic content with residual token-level information, such as artifacts arising from attention sink behavior. We propose a simple method that prepends trainable [DUMP] tokens to the input during embedding model training. The method keeps the model architecture, pooling strategy, and training loss unchanged, while optimizing the added tokens through the standard contrastive retrieval objective. By introducing trainable prefix positions that interact with regular tokens during encoding, [DUMP] tokens help reduce the influence of residual or uninformative token-level signals on the final retrieval representation. Experiments on standard retrieval and long-context benchmarks show that our method consistently improves over finetuned baselines across encoder and decoder embedding models. Further analysis shows that [DUMP] tokens alter token-level activation and attention patterns associated with low-information tokens. These results suggest that trainable prefix tokens provide a lightweight way to improve retrieval representations.
The Brief
1:46A short podcast summary of the paper. Written and voiced by AI.
Transcript
Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens. By Seongtae Hong and four others, from Korea University and Soongsil University. Published on the fifth of November.
An embedding model takes text of any length and squeezes it into a vector of fixed size. Compression of that kind has a side effect. Alongside the meaning of the text, the vector can end up carrying residual information from individual tokens, such as artifacts that arise from attention sink behaviour. Semantic content and that token-level residue get entangled, and the residue ends up shaping a representation that is supposed to be about meaning.
The proposal is small on purpose. During embedding model training, prepend trainable tokens, called DUMP tokens, to the input. Nothing else about the setup moves: the architecture, the pooling strategy, and the training loss all stay as they were. The added tokens are optimized through the same standard contrastive retrieval objective as everything else.
What the tokens provide is a set of trainable prefix positions that interact with the regular tokens during encoding. Through that interaction, the DUMP tokens help reduce how much residual or uninformative token-level signal reaches the final retrieval representation. The aim is an embedding that speaks more about what the text means and less about the quirks of the individual tokens inside it.
The method is tested on standard retrieval benchmarks and on long-context benchmarks. It consistently improves over finetuned baselines, and it does so for both encoder and decoder embedding models. Further analysis looks inside the models and finds that the DUMP tokens alter token-level activation and attention patterns associated with low-information tokens.
Because the change touches only the input, and leaves the model, the pooling and the loss alone, it slots into an existing training setup without redesign. The authors' conclusion is correspondingly modest: trainable prefix tokens offer a lightweight way to improve retrieval representations.
More with an account
Listen to every day’s papers
Daily playlists, deep dives, quests for the papers, and quizzes on their key terms.
Join free