Nereus: Adaptive Parallelism for LLM Post-Training

Nereus. Adaptive Parallelism for LLM Post-Training. By Songlin Jiang and five others, from Aalto University, Shenzhen University of Advanced Technology, and one other institution. Published on the twenty-eighth of September. No code repository is linked to it.

Reinforcement learning post-training for large language models coordinates several models across generation, inference and training, all on GPU clusters. Over the course of a run, a lot can change: resource availability, sequence length, memory pressure, and which stage is the bottleneck. An execution plan that suited the job at the start can become slow later on, or even infeasible. Adapting a live job is hard when its models share GPUs. Three questions have to be answered together. Is a new plan worth the cost of the transition? How can the distributed state the job already holds be reused rather than rebuilt? And how should GPU transfers be coordinated across models and stages?

Nereus is a cost-aware runtime that answers those questions, adapting post-training jobs into efficient execution plans as conditions shift. A low-overhead controller selects a global plan that fits in memory. It then decides whether to admit the transition using a cost model calibrated against the job that is actually running, rather than against assumptions made before the run began.

Estimating a transition and executing it both depend on a handle for distributed state. Nereus represents the state of each replica of a model-stage, meaning one model in one stage, as an Elastic Model Unit. A global transition graph then orders the transformations of those units and the GPU transfers between them. The job's existing state becomes a set of pieces that can be reshaped and moved in a coordinated order, instead of something torn down and started again.

In a trace built from real data, adapting tensor and pipeline parallelism online cuts average step latency by twenty-seven point seven percent, relative to the initial fixed layout with data-parallel scaling. In a run of one thousand steps that reaches one thousand and twenty-four GPUs, six transitions consume zero point zero seven nine percent of total run time.

End to end, on eight-billion-parameter PPO, Nereus improves throughput by between two point one four and seven point two seven times over OpenRLHF, and by between one point one and one point four seven times over Verl, across diverse clusters.

Post-training jobs are usually planned once and then lived with. Nereus treats the plan as something to revisit while the run is in progress, and the figure that makes that affordable is the overhead: six changes of plan across a thousand steps, together costing well under a tenth of one percent of the run.