EpiCon: Collective Agent Learning through Co-Evolving Multimodal Memory
Ziyun Zeng, Hang Hua, Shaden Alshammari, Rogerio Feris, William T. Freeman, Jiebo Luo
MIT-IBM Computing Research Lab · University of Rochester · Massachusetts Institute of Technology
Published September 29, 2026 · on the feed September 30, 2026 · ▲ 8 · arXiv 2609.37923
The abstract
Agents can learn from past executions, but enabling different agents to reuse and build on one another's experience remains challenging. We introduce EpiCon, a shared multimodal memory framework for agent collective learning without updating host model parameters. EpiCon links question-level memory evolution to a persistent experience bank through two independently trained 2B models: a memory controller and a tree self-organizer. The controller jointly refines textual guidance and visual evidence across attempts and selectively includes visual memory. The self-organizer consolidates lessons hierarchically and retrieves experience and rules for new problems. We evaluate EpiCon on eleven benchmarks spanning four multimodal task domains, using two harnesses and multiple backbones. A frozen bank improves other systems even with a single solving attempt. A second harness raises the original system's macro-average score by 2.6 points across eleven benchmarks. Across four host configurations, EpiCon improves macro-average scores by 1.7 to 4.9 points over No Memory and reduces memory-operation time by 67\% to 74\% relative to backbone-sized memory models.
The Brief
1:59A short podcast summary of the paper. Written and voiced by AI.
Transcript
EpiCon. Collective Agent Learning through Co-Evolving Multimodal Memory. By Ziyun Zeng and five others, from the MIT-IBM Computing Research Lab, the University of Rochester, and one other institution. Published on the twenty-ninth of September. No code repository is linked to it.
An agent can learn from its own past executions. Getting different agents to reuse and build on one another's experience is harder, and doing it without touching the parameters of the model each agent runs on is harder still.
EpiCon is a shared multimodal memory framework for collective learning among agents, and it leaves the host model's parameters alone. It connects two kinds of memory. One evolves at the level of a single question, across repeated attempts at it. The other is a persistent experience bank that outlives any one question. Two small models, each with two billion parameters and trained independently, link the two.
The first small model is a memory controller. Across attempts, it refines textual guidance and visual evidence together, and it decides selectively when visual memory should be included at all. The second is a tree self-organizer. It consolidates lessons hierarchically into the bank, and when a new problem arrives, it retrieves the relevant experience and rules.
The evaluation spans eleven benchmarks across four multimodal task domains, using two harnesses and multiple backbones. A frozen bank improves other systems even when they get only a single solving attempt. Moving to a second harness raises the original system's macro-average score by two point six points across the eleven benchmarks. Over four host configurations, EpiCon improves macro-average scores by between one point seven and four point nine points over having no memory. It also cuts memory-operation time by sixty-seven to seventy-four percent compared with memory models the size of the backbone.
The detail to hold onto is the frozen bank. Experience gathered by one system, stored and left untouched, still helps a different system on its first try. That is the sense in which the learning here is collective: what is shared is not weights but memory.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every agent learns something, and every lesson dies with the run. That is the wall.
You run several different agent systems on visual tasks: charts, documents, diagrams. Each learns something from its failures, then forgets it, and the others never hear about it.
The road ahead
- 1 · The problem ~7 min Agent systems on visual tasks learn from failures, then forget, cannot share, cannot be retrained, and lose the image part of each lesson.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Keep lessons in an external shared bank: refine text and image evidence within a question, and organize lessons into a tree across questions.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min The method runs in three steps: refine text and image memory across attempts, consolidate lessons into a tree bank, then retrieve for new problems.
- 6 · The experiment ~6 min You’ll guess: Compared with memory models the size of the backbone, how did EpiCon's small 2B memory models change memory-operation time?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.