DAGFormer: Global Topology Prediction for Transformer Cross-Layer Connections

DAGFormer. Global Topology Prediction for Transformer Cross-Layer Connections. By Xiaocong Yang and two others, from the University of Illinois Urbana-Champaign. Published on the fourteenth of October. Its code repository is worked out: about three hundred and fifty source files, with a readme.

A standard Transformer passes information between layers through residual connections. Existing cross-layer routing architectures go further and add other connections, but they decide each layer's connections separately, one layer at a time. No part of the model looks at the wiring as a whole.

DAGFormer plans the routing of the whole model at once. A topology predictor, trained jointly with the language model, decides the cross-layer connections for every layer together. The connection pattern becomes a single global plan instead of a collection of local decisions.

On top of the usual residual path, DAGFormer adds direct connections from earlier layers to later ones. The predictor sets the strength of every one of those connections, and it does so based on the input tokens. The wiring of the network is therefore not fixed in advance: it can differ from one input to the next.

Comparisons use matched OLMo-2 pairs ranging from seventy-five million to one billion parameters. Against the vanilla Transformer, DAGFormer lowers WikiText-2 perplexity by seventeen to twenty-five percent. A one billion parameter DAGFormer trained on five billion tokens beats the vanilla model trained on ten billion, which is half the data for a better result. Against four routing baselines with the same data and budget, it reaches the lowest perplexity at seventy-five million, at three hundred million, and at one billion parameters.

The gains carry beyond perplexity. Across thirteen evaluation tasks in commonsense, science, reading and math, mean accuracy rises by two point zero to two point five points at the three hundred million and one billion scales. After instruction finetuning, DAGFormer wins sixty-three to seventy percent of AlpacaEval comparisons against the vanilla Transformer, ahead of every routing baseline. Under structured pruning it degrades the least in five of six settings.

One change in viewpoint ties the results together. Treating the connections of the whole network as a single plan, chosen per input, rather than a series of separate local choices, pays off in perplexity, in downstream tasks, and in how gracefully the model survives pruning.