A GPU-Parallel Framework for Heterogeneous Multi-Task Reinforcement Learning

A GPU-Parallel Framework for Heterogeneous Multi-Task Reinforcement Learning. By Rui Zhang and nine others, from NVIDIA, the Hong Kong University of Science and Technology, Guangzhou, and one other institution. Published on the sixth of October. Its code repository is worked out: about seven hundred source files, with a readme.

Simulation running in parallel on a GPU can supply a robot with a great deal of interaction. Yet the benchmarks that exist rarely put that scale together with two other things: manipulation tasks that differ substantially from one another, and a standard way of evaluating reinforcement learning across many tasks at once. One of these tends to come without the others.

Hebero, the Heterogeneous Benchmark for Robot Learning, is built to close that gap. It runs in Isaac Lab on the GPU, and it lets a single policy be trained and evaluated jointly across all forty of its heterogeneous tasks, efficiently. The question it poses is whether one policy can learn the whole set together, rather than one specialist per task.

A first result concerns scale. Under a fixed wall-clock budget, increasing the number of parallel replicas per task improves success. The authors then address a harder setting, with sparse rewards and only a few demonstrations. Their framework, Demonstration-Guided Policy Optimization or DGPO, reuses demonstrations in two ways: to provide dense tracking rewards, and for asymmetric value learning. Because the stack is shared, it allows controlled comparisons of how different learners use demonstrations, all within PPO.

Inside DGPO sits IW-ABC. It relies on a lightweight signal of learning progress for each task, and uses it to coordinate two mechanisms. Adaptive behavior cloning relaxes the pull of the demonstrations as a task progresses. Importance weighting puts more emphasis on the tasks that are lagging during the PPO updates.

With fifty demonstrations per task, IW-ABC reaches a mean success of ninety point one percent from state inputs. That is seven point eight percentage points above the strongest baseline, FAMO-ABC. The visual version of the method reaches a mean success of ninety-three point five percent. Beyond simulation, real-world experiments show a single multi-task policy trained in simulation carrying out four tasks on a physical Piper robot.

The contribution has two layers. Underneath is a benchmark that makes large-scale, many-task training and evaluation routine. On top is a way to spend scarce demonstrations wisely, steering attention toward whichever tasks are falling behind and loosening guidance as each one is learned.