A Three-Layer Caching Architecture for Low-Latency LLM Web Search on Commodity CPU Hardware
Ayushman Bhattacharya, Nihal Gazi
pollinations.ai
Published August 12, 2026 · on the feed September 10, 2026 · ▲ 6 · arXiv 2609.05463
The abstract
AI-powered search products such as ChatGPT search, Google's AI Overviews, and Perplexity provide LLM-synthesized answers grounded in live web results. We developed OreoLook (formerly lixSearch), an open-source answer engine using automated browser agents and provider-routed LLM inference. Its local search, caching, session-management, and embedding stack runs on commodity CPU hardware; answer synthesis is performed by a remote inference provider. As usage grew, sessions lost context, equivalent queries triggered redundant work, and URLs were repeatedly embedded across sessions. We present a three-layer caching architecture: (1) a Session Context Window maintaining a rolling window of recent messages in Redis with automatic overflow to Huffman-compressed disk archives; (2) a Semantic Query Cache catches rephrasings via cosine similarity on embedding vectors, eliminating redundant LLM invocations; and (3) a URL Embedding Cache that deduplicates embedding computations across sessions. Deployed on a single 8-vCPU Intel Cascade Lake server (2 GHz, 32 GB RAM) running 30 Hypercorn worker processes across three containerized replicas, the evaluated system reported an 89.3% aggregate Redis keyspace hit rate with 0.1 ms read latency and just 1.38 MB of memory overhead. A background LRU eviction daemon migrates idle sessions from Redis to disk and re-hydrates them on demand, enabling conversations that can be resumed hours or days later under the configured retention policy.
The Brief
2:43A short podcast summary of the paper. Written and voiced by AI.
Transcript
A Three-Layer Caching Architecture for Low-Latency LLM Web Search on Commodity CPU Hardware. By Ayushman Bhattacharya and one other, from pollinations dot A I. Published on the twelfth of August. Its code repository is worked out: about two hundred and fifty source files, with a readme and a licence.
AI-powered search products, ChatGPT search, Google's AI Overviews and Perplexity among them, answer questions with text an LLM synthesizes from live web results. The authors built an open-source answer engine of this kind, OreoLook, formerly called lixSearch. It uses automated browser agents and routes LLM inference to a provider. Its local stack, covering search, caching, session management and embeddings, runs on commodity CPU hardware, while answer synthesis happens at a remote inference provider. As usage grew, three problems appeared. Sessions lost context. Queries that meant the same thing triggered redundant work. And the same URLs were embedded again and again across sessions.
Each of those problems gets its own caching layer, and the three together form the architecture the paper presents. Lost context calls for somewhere to keep a session. Redundant queries call for recognising a question already answered. Repeated embeddings call for remembering pages already processed.
The first layer is a Session Context Window. It keeps a rolling window of recent messages in Redis, and when that window overflows, the older material goes automatically to disk archives compressed with Huffman coding. The second layer is a Semantic Query Cache. It catches rephrasings of a question by comparing embedding vectors with cosine similarity, so an equivalent query does not trigger another LLM call. The third is a URL Embedding Cache, which deduplicates embedding computations across sessions, so a page embedded once is not embedded again.
A background daemon handles eviction on a least-recently-used basis. It moves idle sessions out of Redis to disk and rehydrates them when they are needed, so a conversation can be picked up again hours or days later, within the configured retention policy.
The deployment is modest by design. One server, an eight-vCPU Intel Cascade Lake machine running at two gigahertz with thirty-two gigabytes of RAM, runs thirty Hypercorn worker processes across three containerized replicas.
On that setup, the evaluated system reported an aggregate Redis keyspace hit rate of eighty-nine point three percent. Read latency was zero point one milliseconds, and the memory overhead just one point three eight megabytes.
This is an engineering report, and its argument is that the pain points of a growing answer engine can each be traced to repeated work, and that each can be absorbed by a cache aimed squarely at it. Context, queries and embeddings each get one layer, and all of it fits on ordinary CPU hardware.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A cheap answer engine starts creaking under its own users. Here is where it hurts.
You run a small open-source answer engine: browser agents search the web, a remote LLM writes the answer, and everything else runs on one ordinary CPU server.
The road ahead
- 1 · The problem ~7 min A cheap answer engine on one CPU server starts wasting work and forgetting conversations as more people use it.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Give each of the three wastes its own cache layer: recent messages, rephrased questions, and page embeddings, each stored and shared in its own way.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Messages enter a rolling Redis window, overflow to compressed disk, queries check a similarity cache, pages reuse stored embeddings, and idle sessions move to disk.
- 6 · The experiment ~6 min You’ll guess: On a single commodity CPU server, what aggregate Redis keyspace hit rate did the deployed caching system report?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.