MemPilot: Orchestrating On-Demand Multimodal Memory Curation for LLM Agents

MemPilot. Orchestrating On-Demand Multimodal Memory Curation for LLM Agents. By Haozhen Zhang and eight others, from Nanyang Technological University, Tsinghua University, and one other institution. Published on the fifth of October. Its code repository is worked out: about sixty source files, with a readme and a licence.

Memory lets a language model agent retain information and reuse it across interactions. Most agent memory systems build that memory in a query-agnostic way, before anyone knows what will be asked. That has two costs. It can spend preprocessing effort that turns out to be unnecessary, and it can throw away details that later prove essential. Recent studies have started moving memory processing to runtime, adapting it to the query. But they typically specialize in particular operations or fixed processing schemes, and flexible control over the balance of performance, cost, and latency has gone largely unexplored.

MemPilot is a framework that orchestrates memory curation on demand, under whatever preference among performance, cost, and latency the user sets. At its centre is a multi-step language model policy, optimized with reinforcement learning. At each step it chooses between two moves: retrieve from the query-agnostic memory that already exists, or delegate query-specific curation of the raw multimodal history to one of a set of heterogeneous language and vision-language models.

The policy controls several things jointly: how much evidence to use, what curation instructions to give, which model to select, and whether to access visual content. That is what allows fine-grained allocation of computation at runtime. Training it means juggling competing objectives, so the authors adapt objective-wise advantage decoupling, estimating each objective's advantage separately before aggregating them. And because rollouts are multi-step, they introduce prefix-based marginal utility estimation to assign credit to individual steps within a rollout.

Experiments run on five multimodal agent-memory benchmarks. MemPilot shows favorable trade-offs among performance, cost, and latency across the different optimization preferences. Sweeping the preference produces broader frontiers than existing baselines that are themselves aware of trade-offs.

Memory here stops being a fixed archive built once and becomes a decision made per query, about how much to spend and on what. The breadth of the frontier is the point: the same framework can be steered toward cheaper, faster, or more accurate behaviour by setting the preference, rather than by committing to one fixed processing scheme in advance.