Scaling Parameter-Efficient Fine-Tuning for Brain Encoding

Scaling Parameter-Efficient Fine-Tuning for Brain Encoding. By Zhuofan Josh Ying and eight others, from Columbia University, Mind Lab, and two other institutions. Published on the nineteenth of October. Its code repository is public and real: about thirty-five source files.

Brain encoding models aim to predict neural responses to stimuli: given what a person sees or hears, what will their brain activity look like. The most widely used approach takes a pretrained backbone, keeps it frozen, and fits a readout on top of its features. The alternatives have been hard to make work. Training large models from scratch on neural data has proven difficult, and so has fully fine-tuning them after task-pretraining. The reason is the data. Neural data is costly to collect and limited in amount, which is not enough to move a very large model all the way. So the backbone usually stays exactly as it came, and only the thin layer above it ever learns from the brain.

The paper shows that parameter-efficient fine-tuning, or PEFT, opens a path between those two options. With a small number of trainable parameters, very large pretrained backbones can be effectively adapted for brain encoding. The method, called BrainLoRA, trains a low-rank adapter, a LoRA, together with a lightweight readout. The backbone itself now changes, but only through a small set of added parameters, which keeps the amount being learned in proportion to the limited neural data available.

Making this computationally practical takes a second step. Rather than using sliding windows, the approach exploits the backbone's causal structure to predict multiple timepoints in a single pass, which significantly reduces the cost of feature extraction. That saving is what makes efficient fine-tuning possible for multimodal backbones of up to one hundred and four billion parameters, which to the authors' knowledge is the largest model yet adapted for brain encoding. A further piece of engineering scales out the number of adapted models: a multi-LoRA kernel that accelerates the joint training of multiple LoRAs on one shared backbone.

Across nine diverse multimodal backbones, the approach consistently improves brain encoding performance. On the same Qwen3-Omni backbone used by the prior state of the art, BrainLoRA achieves the best published performance on the Algonauts Challenge of two thousand twenty-five. Scaling out with the multi-LoRA kernel produces an ensemble of more than two hundred adapted models, and that ensemble ranks first on the public Algonauts post-challenge leaderboard at the time of writing.

The adapted backbones also remain interpretable. Modality attribution recovers the canonical sensory topography, a pattern that prior work often obtains by construction, by building separate unimodal encoders. Here it emerges from a single multimodal backbone. Neural adaptation also shifts what happens inside that backbone: it increases the contribution of audio inputs, and it increases cross-modal mixing within the model.

What the paper argues for is a change of default. The field has largely frozen its backbones and trained only readouts. The authors put forward PEFT as an efficient and powerful paradigm for brain encoding instead: adapting the backbone itself, even at the largest scale tried, without the volume of data that full fine-tuning would demand. And it does so without giving up the ability to look inside the adapted model and read what it is doing with sight and sound.