Building AI Research Teams for Autonomous Discovery of Linear Attention Architectures

Building AI Research Teams for Autonomous Discovery of Linear Attention Architectures. By Yujian Liu and seven others, from UC Santa Barbara, Pennsylvania State University, and one other institution. Published on the twentieth of October. No code repository is linked to it.

Autonomous agents can increasingly propose ideas, modify code, and run experiments. That opens a possibility: accelerating scientific discovery with teams of collaborating AI researchers. A search of this kind can get stuck in local minima, and the authors' starting proposal for escaping them is orthogonal exploration, built by staffing the team with agents of varying expertise.

Diverse backgrounds alone do not realise that design. When experiments are costly, a team can test only a limited number of ideas. Someone has to decide which subset of researchers gets to propose and run the next experiments, which is a balance between exploring and exploiting. And each result ought to shape what the team tries next. The question the paper poses is how AI researchers from different scientific backgrounds can build on one another's findings and keep making progress under a limited experimental budget.

The answer is a team of persistent AI researchers, each grounded in a different discipline. Three organisational choices define it. Each researcher keeps a private research history. All of them share experimental evidence. And research opportunities reach them through adaptive allocation.

Put together, that gives the team two kinds of memory and one kind of steering. The shared evidence is common ground: every researcher can see what the experiments have shown. The private history is individual: what this particular researcher has tried and learned. Adaptive allocation is the steering, deciding where the limited budget of experiments goes, which is the exploration and exploitation balance the problem called for. Persistence ties the three together: the same researchers stay on the problem over time, so each new result can shape what the team tries next rather than being lost between rounds. Disciplinary grounding is what supplies the orthogonal exploration. Researchers who come from different fields are meant to look in different directions, which is the original bet against local minima. The team was deployed on one concrete search problem, the design of linear attention architectures.

There, the architecture the team discovered outperforms GDN2 at the one point three billion parameter scale. The comparison is broad rather than a single score. It covers language modeling, common-sense reasoning, and retrieval tasks of two kinds, synthetic and real-world.

Controlled comparisons then remove one organisational component at a time, and every removal yields markedly weaker discoveries. Without disciplinary backgrounds, the search plateaus early. Without private histories, researchers make less progress, even though the shared evidence they see is identical. And uniform selection of researchers falls short of adaptive allocation within the same budget. The middle result is the sharpest one: holding the shared evidence fixed, what each researcher privately remembers still changes how far the team gets.

The authors draw the conclusion in organisational terms. Sustained autonomous discovery, on this evidence, depends not only on stronger agents. It also depends on who explores, what they remember, and how research effort is allocated. Each of those three was tested by taking it away, and each one mattered.