BEND 2.0: Benchmarking the Capabilities Needed to Design DNA

BEND 2.0. Benchmarking the Capabilities Needed to Design DNA. By Andreas Bjerregaard and eight others, from the University of Copenhagen, the Technical University of Denmark, and one other institution. Published on the second of November.

Genomic foundation models are more and more often used to propose DNA sequences and edits to them. Successful design, though, needs capabilities that current benchmarks do not measure. A model can do well on the existing tests and still not be the tool a designer needs. BEND 2.0 is built to measure that missing part.

BEND 2.0, short for BENchmark for DNA language models, targets two questions. How well do models predict the effects of changes to DNA? And can those predictions actually guide the design of sequences? The second question is the one that matters to anyone using these models to design rather than to score. A prediction is only useful for design if acting on it gets you to the sequence you wanted with fewer tries. So the benchmark looks not only at whether models predict well, but at whether their predictions can be put to work.

The benchmark has eight tasks, built on five experimental datasets. Some test prediction of the effects of changes, and some are design tasks, where the question is how many candidates have to be tried before a desired response is found. The authors also compare ways of accessing a model, including its native outputs and linear readouts fitted to its frozen representations.

Two choices matter for performance: which model is used, and how its representations are accessed. Linear readouts fitted to frozen representations can match or even exceed a model's native outputs. Functional post-training substantially improves the models' capabilities. The linear readouts also transfer well across experimental assays and across genes, which suggests the models capture functional patterns beyond the data used to fit any individual predictor.

There are cautions too. Strong prediction of average effects can hide weaker prediction of effects that depend on context. And accurate prediction of combined mutation effects does not mean the interactions between those mutations are predicted accurately. In the design tasks, model predictions can cut the number of candidates needed to reach a desired response, but they still leave substantial room for better design choices, especially for context-specific objectives and objectives spanning multiple cells.

The picture the benchmark paints is mixed in a specific way. Genomic models now show signs of general utility, since a simple readout on their representations transfers across assays and genes. Yet they have not saturated performance even on simple tasks, and an averaged score can mask exactly the context-specific and interaction effects a designer cares about.