Horizon-Scalable Q-Learning

Horizon-Scalable Q-Learning. By Haoran Xu and four others, from the University of Texas at Austin. Published on the seventh of November.

Learning an accurate off-policy Q-function from an offline dataset gets harder the longer the task runs. One-step temporal-difference learning bootstraps after every single transition, so small errors at each step pile up along the horizon. Early states can end up badly misvalued even while the TD loss looks small. Multi-step TD shortens that chain by following the dataset's own trajectory for several steps before bootstrapping. But the behaviour policy that produced the data is often suboptimal, so every action followed adds a bias, and no fixed number of steps suits a whole trajectory. Too few and errors compound; too many and bias accumulates.

Horizon-Scalable Q-Learning, or HQL, stops choosing a fixed number and lets the target decide for itself, step by step, balancing automatically between one-step and multi-step returns. The question it asks at each point is whether the action in the dataset is good enough to keep following.

The rule is concrete. Sample N actions from the BC policy and take the best of them according to the critic. If the critic values the dataset action at least as highly as that best-of-N action, the target extends through it, one step further along the trajectory. If not, the target falls back to a one-step backup. The authors show that this rule amounts to a principled off-policy correction which needs no action densities. Not needing them is what makes HQL compatible with expressive policies of the flow and diffusion kind.

On a combination-lock task, tested across a range of horizons, HQL avoids the failures of both one-step and multi-step TD. On long-horizon tasks from OGBench and OCBench, it outperforms prior horizon-reduction methods.

Seen from the side of the dataset, the method draws a line through every trajectory. Wherever the logged action holds up against the policy's own best guess, the value estimate is allowed to travel further along the recorded path before bootstrapping. Wherever it does not, the estimate stops and bootstraps after a single step, so the horizon each target spans is set by the data rather than fixed in advance.