Self-Evolving Search Index
Sangam Lee, Wonjae Lee, Sunghwan Kim, Deogyong Kim, Jaehoon Kim, Daye Nam, SeongKu Kang, Dongha Lee
Yonsei University · Samsung Research · University of California, Irvine · Korea University
Published September 17, 2026 · on the feed September 18, 2026 · ▲ 60 · arXiv 2609.19656
The abstract
Information retrieval is increasingly important as LLM agents tackle complex tasks involving diverse information needs. Because retrieval relies on an index that represents each document through index keys, retrieval quality depends heavily on how effectively these keys expose the knowledge contained in each document. However, effective index representations vary across retrieval environments, making it difficult for any fixed optimization strategy to perform consistently. Yet evolving an index to its retrieval environment remains largely human-driven, requiring humans to diagnose retrieval failures, refine the optimization strategy, and reprocess the index accordingly. We propose SELF-INDEX, a framework that enables an index to self-evolve without human intervention. Its Optimizer autonomously diagnoses retrieval shortfalls, selectively revises the responsible index keys, and validates each revision before updating the index. Beyond reacting to observed retrieval demands, SELF-INDEX proactively explores additional demands through a Query Simulator, allowing the index to evolve beyond the queries already available for optimization. Across diverse corpora and retrievers, SELF-INDEX consistently improves retrieval performance while outperforming existing index optimization methods. We further show that these benefits extend to downstream applications, improving the effectiveness and efficiency of search agents and helping agent memory systems retrieve useful past interactions.
The Brief
2:29A short podcast summary of the paper. Written and voiced by AI.
Transcript
Self-Evolving Search Index. By Sangam Lee and seven others, from Yonsei University, Samsung Research, and two other institutions. Published on the seventeenth of September. Its code repository is a placeholder: a readme, and nothing to run yet.
As agents built on language models tackle complex tasks with diverse information needs, retrieval matters more. Retrieval relies on an index that represents each document through index keys, so its quality depends heavily on how well those keys expose the knowledge inside each document. But what makes a good index representation varies from one retrieval environment to another, so no fixed optimization strategy performs consistently. Adapting an index to its environment is still largely done by people: diagnosing retrieval failures, refining the strategy, and reprocessing the index to match. An index that suits one environment can suit another poorly, and every time the environment changes, the same manual cycle has to run again.
SELF-INDEX is a framework that lets an index evolve by itself, with no human intervention. The work people were doing, diagnosing and repairing, is handed to two components: an Optimizer that repairs the index, and a Query Simulator that explores what it may be asked. The Optimizer handles demand already seen; the Query Simulator anticipates demand not yet seen.
The Optimizer autonomously diagnoses retrieval shortfalls, selectively revises the index keys responsible for them, and validates each revision before the index is updated. Only changes that pass validation are kept. Validation is the safeguard: a revision that would make retrieval worse is caught before it reaches the index. The Query Simulator adds a proactive side. Rather than reacting only to retrieval demands already observed, it explores additional ones, so the index can evolve beyond the queries that happened to be available for optimization.
Across diverse corpora and retrievers, SELF-INDEX consistently improves retrieval performance, and outperforms existing index optimization methods. The benefits carry into downstream applications. Search agents become both more effective and more efficient, and agent memory systems get better at retrieving useful past interactions. The index, in other words, improves the systems that sit on top of it, not only the retrieval measures.
The shift is from an index as a fixed artifact, built once and rebuilt by hand, to an index that maintains itself. It notices where it fails, repairs only those keys, checks the repair, and goes looking for questions it has not yet been asked.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
An index is only as good as its keys, and the right keys depend on where you are.
You run a search system for language-model agents. Each document sits in an index under keys, short representations the retriever matches against queries.
The road ahead
- 1 · The problem ~7 min Search keys that suit one corpus and retriever fail on another, and people must fix the index by hand each time.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min A language-model loop takes over the human work: it diagnoses weak keys from retrieval results, rewrites them, and keeps only rewrites that pass checks.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Each round, the Optimizer diagnoses shortfalls, rewrites the responsible documents' key sets as a whole, and validates every key before updating the index.
- 6 · The experiment ~6 min You’ll guess: When search agents used an index that had evolved itself, what happened to their performance and their efficiency?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.