Self-Evolving Search Index

Sangam Lee1*, Wonjae Lee1,2*, Sunghwan Kim1, Deogyong Kim1, Jaehoon Kim1,
Daye Nam3, SeongKu Kang4, Dongha Lee1†
1Yonsei University 2Samsung Research 3University of California, Irvine 4Korea University

*Equal contribution. †Corresponding author.

Paper Code
BRIGHT Biology example: for a query about independent ring-finger movement, a relevant document rises from rank 63 at iteration 0 to rank 40 at iteration 6 and rank 2 at iteration 10 as its index keys clarify the functional relevance of tendon connections.
A BRIGHT Biology example. Select the figure to view it at full size.

Effective index representations vary across retrieval environments, making it challenging to evolve an index to fit a given environment. Existing approaches make this evolution costly in both human effort and computation. Self-Index enables an index to self-evolve without human intervention.

Why Self-Index?

Retrieval quality depends on how well index keys expose document knowledge. Because effective representations vary across corpora and retrievers, the index needs to evolve to fit its retrieval environment. Existing approaches make this evolution costly in two ways.

Human Effort

Humans must diagnose retrieval failures and refine the optimization strategy, through manual changes or additional relevance annotations and retraining.

Computational Cost

Applying each revised strategy requires reprocessing all index keys. This cost recurs as new retrieval failures call for further refinement.

Self-Index addresses these bottlenecks by automating index evolution and selectively updating the affected keys.

Self-Index Framework

Self-Index loop: the Query Simulator samples documents, generates queries, and filters duplicates. The Optimizer retrieves index keys, diagnoses shortfalls using co-retrieval profiles, revises the affected key sets, and validates faithfulness, specificity, and separation before updating the index.
The Optimizer evolves index keys using observed or simulated queries. The Query Simulator explores new demands, and validated improvements accumulate as the loop repeats.

Self-Diagnosis

Uses retrieval outcomes and co-retrieval patterns to identify index shortfalls, without requiring query-document relevance annotations.

Self-Revision

Autonomously decides how to refine the affected key sets, while keeping the underlying documents unchanged.

Self-Validation

Checks generated keys for faithfulness, specificity, and separation, accepting only validated revisions into the index.

Self-Exploration

A Query Simulator explores retrieval demands not yet covered, allowing the index to evolve beyond the queries already available.

🚀 Performance Highlights

Self-Index improves retrieval across natural language, code, math, and tables, with benefits extending to search agents and agent memory.

Stronger Retrieval

+57.0% (BGE)
+40.4% (BM25)

Relative gains in average nDCG@10 on BRIGHT over the corresponding base index.

More Efficient Search Agents

+41.1% accuracy
−24.0% online cost

Relative changes on BrowseComp-Plus for Kimi-K2.5 with BM25, compared with the base index.

Better Agent Memory

+13.9% overall
0.415 → 0.472

Relative overall score gain on LongMemEval-V2 for the Query → Slice baseline.

Reported results from the paper. Online cost excludes offline index construction.