Not All Ranks Are Equal: Budget-Aware LoRA Merging Across Tasks

Not All Ranks Are Equal. Budget-Aware LoRA Merging Across Tasks. By Avinash Amballa and four others, from Samsung Research America. Published on the third of September. No code repository is linked to it.

When each task has its own low-rank adapter, or LoRA, serving several tasks means swapping task-specific weights at inference time. Merging the adapters promises to eliminate that overhead. The authors point to an assumption built into existing merging methods: that every layer needs the same rank budget. Some methods go further and assume the budget must also be split equally among the tasks. The paper shows that this uniform-budget assumption is a major source of the performance gap between a merged adapter and the separate per-task LoRAs it replaces.

Dropping the assumption is not free, though. Deciding which ranks to keep is a selection problem, and rank selection is NP hard. So the question becomes how to allocate the budget well without solving that problem exactly.

The answer is Net Utility, a data free metric. It asks, for every direction a task's adapter contributes, how useful that direction is for its own task and how much it interferes with the other tasks. Directions that help a lot and collide little score high. Directions that help little or collide a lot score low.

Concretely, each task LoRA is first decomposed by its Singular Value Decomposition, so that it becomes a set of singular directions. Each of those directions is scored by its task utility and by its interference with the other tasks' directions. The scores are then pooled globally, across layers and tasks together, and the directions with the highest values are selected, subject to a constraint on the total number of directions chosen. Pooling globally is what lets the budget fall unevenly: a layer or a task that needs more directions can receive them.

Net Utility is applied on top of five different merging methods, across three different merging spaces. Merging is carried out over two sets of tasks, one of vision tasks and one of language tasks. Rank allocation based on Net Utility outperforms the same methods without that allocation. On average the improvement is two point one percent over the vision tasks, and two point two percent over the language tasks.

Because it sits on top of existing merging methods, the contribution is less a new way to merge than a new way to spend the budget. The title says it directly: once ranks are allowed to be unequal, deciding where they go becomes the lever.