Not All Objectives Are Born Equal: Priority-Constrained Descent for Hierarchical Multi-Objective Optimization
Dara Varam, Mohamed I. Alhajri
American University of Sharjah · Massachusetts Institute of Technology
Published September 21, 2026 · on the feed September 29, 2026 · ▲ 4 · arXiv 2606.29521
The abstract
Deep learning problems rarely involve objectives that are equal in importance. A primary objective defines the goal, whilst secondary objectives, such as sparsity, compression, or robustness constrain the solution. While existing multi-objective methods have proven effective in practice, they have a clear symmetry problem and neglect the inherent objective hierarchy built into these objective spaces. We introduce Priority-Constrained Descent (PCD), a gradient-based optimization framework designed to explicitly exploit hierarchical objective structures. PCD preserves the direction of primary descent whilst allowing for the minimal distortion necessary to guarantee progress on secondary objectives, controlled by a single τin [0, 1] that dictates the strength of the distortion. The resulting formulation is invariant to objective scaling and admits exact closed-form solutions for problems with two and three objectives. We evaluate PCD within structured network compression settings, unstructured sparsity and low-rankness, and across a variety of synthetic experiments, showing Pareto dominance and better per-objective performance with secondary progress guarantees over existing methods, further exhibiting the interpretable trade-off that τ provides.
The Brief
2:08A short podcast summary of the paper. Written and voiced by AI.
Transcript
Not All Objectives Are Born Equal. Priority-Constrained Descent for Hierarchical Multi-Objective Optimization. By Dara Varam and one other, from the American University of Sharjah and the Massachusetts Institute of Technology. Published on the twenty-first of September. Its code repository is public and real: thirteen source files.
Most deep learning problems do not carry objectives of equal weight. One objective is primary and defines the goal. Others, like sparsity, compression or robustness, are secondary: they constrain the solution rather than define it. Existing multi-objective methods work well in practice, but the paper argues they share a symmetry problem. They treat the objectives as peers, and so they ignore the hierarchy that is already built into the objective space.
Priority-Constrained Descent, or PCD, is a gradient-based optimization framework built to exploit that hierarchy explicitly. The primary objective sets the direction of descent, and PCD preserves that direction. The secondary objectives are allowed to bend it, but only by the minimal distortion needed to guarantee progress on them.
How much bending is permitted comes down to one number, a parameter called tau that runs from zero to one and sets the strength of the distortion. That single dial is the whole of the trade-off. The formulation that results has two useful properties. It is invariant to how the objectives are scaled, so multiplying one loss by a constant does not change the answer. And for problems with two or three objectives, it admits exact closed-form solutions: the answer can be written down directly.
The evaluation covers structured network compression, unstructured sparsity and low-rankness, and a range of synthetic experiments. Across these, PCD shows Pareto dominance over existing methods and better performance on each individual objective, while keeping its guarantee of progress on the secondary ones. The experiments also show the trade-off controlled by tau behaving in an interpretable way.
Where symmetric methods ask how to balance several losses, PCD starts from a different question: which loss is in charge? Once that is answered, the remaining objectives become constraints on a direction rather than rivals for it, and a single parameter decides how far they may push.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is an optimizer that treats a goal and its constraints as equals.
You are pruning a network. Accuracy is what you actually care about; sparsity is a condition the solution must meet, not the goal itself.
The road ahead
- 1 · The problem ~6 min When you prune a network, accuracy is the goal and sparsity is only a condition, yet standard methods treat the two losses as equals.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Let the primary loss's gradient set the direction, and let secondary losses bend it only as much as needed, with one number tau setting how much.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Each PCD step takes the primary gradient, sets the progress bar for secondaries with tau, and minimally bends the direction to clear it.
- 6 · The experiment ~6 min You’ll guess: In compression, sparsity and synthetic tests, how did Priority-Constrained Descent, which favours a primary objective, fare against existing symmetric multi-objective methods?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.