ScaleNav: Scalable Internet Video Pretraining for Language-Conditioned Local Navigation
Yihui Yang, Yuanhong Yu, Jiawei Wang, Zhongrui Yu, Jia Shi, Kun Zhan, Ruizhen Hu, Xiaowei Zhou and 2 others
Zhejiang University · Li Auto Inc. · Shenzhen University
Published October 19, 2026 · on the feed October 20, 2026 · ▲ 91 · arXiv 2610.25690
The abstract
We study language-conditioned local navigation, where a robot must navigate to a text-specified target visible in its initial observation and stop at the described location. End-to-end navigation policies require diverse training data to generalize across scenes, yet real-robot data is costly to collect and simulated data leaves a sim-to-real gap. Internet egocentric videos offer a scalable source of real-world navigation experience. Our key insight is to learn goal-directed visual dynamics from Internet egocentric videos, and use these dynamics to support robot action generation. To this end, we propose ScaleNav, a unified world-action framework with three-stage training. We first perform instruction-conditioned video pretraining on these videos to learn how observations evolve as the camera approaches instructed targets across diverse real-world scenes. Action alignment then uses action-labeled simulated data to train the model to predict executable action chunks. Joint fine-tuning subsequently optimizes all trainable components together for navigation. ScaleNav achieves success rates of 86.5% on a single-view adaptation of Short-Horizon OVON and 94.9% on our HabitatGS-LN benchmark, outperforming the evaluated baselines on both benchmarks. We further deploy and evaluate ScaleNav on a real quadruped robot, with results supporting the model's ability to generalize to real-world navigation. Code and videos are available at https://zju3dv.github.io/ScaleNav/
The Brief
2:28A short podcast summary of the paper. Written and voiced by AI.
Transcript
ScaleNav. Scalable Internet Video Pretraining for Language-Conditioned Local Navigation. By Yihui Yang and nine others, from Zhejiang University, Li Auto, and one other institution. Published on the nineteenth of October. Its code repository is worked out: about eighty-five source files, with a readme and a licence.
The task is language-conditioned local navigation. A robot is given a text description of a target that is already visible in its first observation, and it has to move there and stop at the described spot. End-to-end navigation policies can learn this, but to generalise across scenes they need diverse training data. That is where the difficulty sits. Data from real robots is costly to collect, and simulated data leaves a gap between simulation and the real world.
There is a third source that scales: egocentric videos from the Internet, which are full of real-world navigation experience. The key insight is to use those videos to learn goal-directed visual dynamics, meaning how the view changes as a camera heads toward a target, and then to let those dynamics support the generation of robot actions. ScaleNav puts this into a unified world-action framework trained in three stages.
The first stage is instruction-conditioned video pretraining on the Internet videos. Here the model learns how observations evolve as the camera approaches an instructed target, across a wide range of real scenes. No robot actions are involved yet.
The second stage is action alignment. Simulated data that carries action labels teaches the model to predict action chunks a robot can execute.
The third stage is joint fine-tuning, which optimises all the trainable components together for the navigation task itself. The order matters to the design. Real video supplies the variety of scenes, simulation supplies the actions, and the final stage fits the two to each other.
ScaleNav reaches a success rate of eighty-six point five percent on a single-view adaptation of Short-Horizon OVON, and ninety-four point nine percent on the authors' own HabitatGS-LN benchmark. On both benchmarks it outperforms the baselines that were evaluated. The authors also deploy it on a real quadruped robot, and those results support the model's ability to generalise to navigation in the real world.
Robot data is scarce, but first-person footage of moving toward things is not. ScaleNav's bet is that the visual side of reaching a goal can be learned from that footage, leaving only the translation into actions to be taught with labelled simulation. The real-robot deployment suggests the transfer survives the trip off the screen.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A robot can see the target. Getting there without real-robot data is the hard part.
You are teaching a robot to walk to a target it can already see, like the desk by the window, and stop there.
The road ahead
- 1 · The problem ~7 min A robot must walk to a target it can already see and stop there, but the data it needs is costly, unrealistic or missing actions.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Learn from videos how the view changes while approaching a named target, then teach actions with simulated data, then tune everything together.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Three stages in order: video pretraining on instruction-paired Internet videos, action alignment on simulated data, then joint fine-tuning of all trainable components.
- 6 · The experiment ~6 min You’ll guess: What success rate did ScaleNav reach on the authors' own HabitatGS-LN benchmark?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.