SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs
Ruijia Yang, Shiyuan Lin, Yulong Ao, Zhiyu Li, Yingli Zhao, Xianduo Li, Yonghua Lin, Zeyi Wen
The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence
Published September 28, 2026 · on the feed October 1, 2026 · ▲ 2 · arXiv 2609.34162
The abstract
Host-resident layer streaming enables full-parameter LLM fine-tuning beyond GPU memory, but data-parallel ranks compete for shared host resources. Replicated transfers amplify traffic, while strong scaling can expose host work as computation windows shrink. We present SlideDP, a synchronous data-parallel runtime for shared-host multi-GPU systems. It maintains one authoritative host state, decouples communication routes from state layout, and pipelines parameter delivery, gradient aggregation, and CPU updates across ranks and chunks. An analytical step-time model characterizes resource bottlenecks and pipeline exposure; runtime measurements guide communication, chunking, and activation policies under a GPU memory budget. In matched-batch sweeps, SlideDP achieves geometric-mean throughput ratios of 1.46-2.64times over SlideFormer, MegaTrain, and ZeRO-Offload. On four H100s, SlideDP approaches GPU-resident FSDP2 throughput for Qwen3-14B at a smaller batch size. With a larger batch, it processes over 1M tokens per step and exceeds FSDP2's measured peak throughput by 11.2%. Separately, it supports 256K-token sequences for the same model and fine-tunes Qwen2.5-72B on four RTX 4090 GPUs. Project page: https://github.com/RegiaYoung/SlideDP.
The Brief
2:11A short podcast summary of the paper. Written and voiced by AI.
Transcript
SlideDP. Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs. By Ruijia Yang and seven others, from the Hong Kong University of Science and Technology, Guangzhou and the Beijing Academy of Artificial Intelligence. Published on the twenty-eighth of September. Its code repository is public and real: about twenty-five source files.
Streaming layers from host memory lets full-parameter fine-tuning of a large language model go beyond what fits on the GPU. Add more GPUs, though, and the data-parallel ranks start competing for the same shared host resources. Replicated transfers multiply the traffic. And under strong scaling, as each GPU's computation window shrinks, host-side work that used to be hidden starts to show.
SlideDP is a synchronous data-parallel runtime built for that shared-host, multi-GPU setting. Its central choice is to keep one authoritative copy of the state on the host, and to decouple the routes that communication takes from the way that state is laid out.
Parameter delivery, gradient aggregation, and the updates on the CPU are pipelined across ranks and across chunks. An analytical step-time model characterizes where the resource bottlenecks lie and how much of the pipeline is exposed. Measurements taken at runtime then guide the policies for communication, chunking, and activations, all within a GPU memory budget.
In matched-batch sweeps, SlideDP's geometric-mean throughput comes out between one point four six and two point six four times that of SlideFormer, MegaTrain, and ZeRO-Offload. On four H100s, it approaches the throughput of GPU-resident FSDP2 for Qwen3-14B at a smaller batch size.
With a larger batch it processes over a million tokens per step, beating the peak throughput measured for FSDP2 by eleven point two percent. Separately, it handles sequences of two hundred fifty-six thousand tokens for the same model, and fine-tunes Qwen2.5-72B on four RTX 4090 GPUs.
Running out of GPU memory used to be the limit. Once the model lives on the host, the limit moves to the host itself, shared by every GPU that needs it. SlideDP's answer is a single source of truth there, and a pipeline that keeps the host busy without letting it be seen.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
One GPU was fine. Add more, and they start fighting over the same host.
You fine-tune every weight of a large language model that does not fit on a GPU. Its training states live in CPU memory, and layers are sent through a small GPU window one at a time.
The road ahead
- 1 · The problem ~7 min Training a model too big for one GPU from CPU memory works, but adding GPUs on the same machine makes them fight over the host.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Keep one master copy on the host, overlap all host work with GPU computing, and let a model plus measurements pick the routes.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min The runtime keeps one host state, delivers parameters to all GPUs, combines their gradients, and updates the host state on the CPU.
- 6 · The experiment ~6 min You’ll guess: On four H100s with a larger batch, how did SlideDP's throughput for Qwen3-14B compare with the measured peak of GPU-resident FSDP2?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.