Holi-World: Building Holistic World Intelligence from Human-Ego Experience
Hao Li, Xiyuan Zheng, Yuanyuan Gao, Yalun Dai, Xinhao Ji, Xuan-Li Chen, Jing Zhao, Fangzhou Hong and 2 others
Ropedia · NTU · HKUST · PKU
Published October 19, 2026 · on the feed October 20, 2026 · ▲ 172 · arXiv 2610.26549
The abstract
Human egocentric experience offers a rich foundation for learning how the world is organized and how it changes through interaction. Realizing this potential at scale requires turning growing volumes of recordings into learning resources that connect scene structure, object semantics, and interaction histories across viewpoints and time, while reducing dependence on costly manual annotation. We introduce HOLI-WORLD, an integrated ecosystem connecting experience capture, world representation, automated data construction, model learning, and independent evaluation. To sup- port learning at scale, HOLIWORLD-Foundry combines complementary static and dynamic perception to automatically construct world records that connect persistent scene structure and evolving object states with interaction histories and supporting observations. HOLIWORLD-Data organizes these records from approximately 100 self-collected scenes into grounded learning resources, including approximately 80K QA pairs used to train HOLIWORLD-VL-4B. Independent human annotations of heldout recordings form HOLIWORLD-Bench, supporting evaluation of both annotation quality and learned spatial and temporal understanding. Experiments on ScanNet, ScanNet++, Aria Digital Twin, and HOLIWORLD-Bench demonstrate strong performance in static perception and dynamic tracking, while training on HOLIWORLD-Data substantially improves HOLIWORLD-VL-4B's spatial and temporal reasoning on HOLI- WORLD-Bench and EgoDynamic4D. HOLIWORLD provides a practical foundation for expanding learning resources from the continuous flow of human experience and advancing machine understanding of the dynamic world.
The Brief
2:37A short podcast summary of the paper. Written and voiced by AI.
Transcript
Holi-World. Building Holistic World Intelligence from Human-Ego Experience. By Hao Li and nine others, from Ropedia, Nanyang Technological University, and two other institutions. Published on the nineteenth of October. Its code repository is a placeholder: a readme, and nothing to run yet.
Human egocentric experience, the world as seen from a person's own point of view, is a rich foundation for learning how the world is organised and how it changes through interaction. Using it at scale means turning a growing volume of recordings into learning resources, ones that link scene structure, object semantics and histories of interaction across viewpoints and across time, and doing so without leaning heavily on costly manual annotation.
HOLI-WORLD is the authors' answer: an integrated ecosystem that connects capturing experience, representing the world, constructing data automatically, learning models, and evaluating them independently.
It has four named parts. HOLIWORLD-Foundry is the construction engine. HOLIWORLD-Data is the organised learning resource it feeds. HOLIWORLD-VL-4B is a model trained on that data. And HOLIWORLD-Bench is the evaluation set.
The data is built from roughly one hundred scenes that the authors collected themselves. Out of those come grounded learning resources, including about eighty thousand question and answer pairs, which are what train HOLIWORLD-VL-4B.
The Foundry is what makes scale possible. It combines complementary static and dynamic perception to build world records automatically. Each record connects the persistent structure of a scene and the changing states of the objects in it with the history of interactions and the observations that support them.
Evaluation is kept apart from that pipeline. HOLIWORLD-Bench comes from independent human annotations of recordings that were held out, so it can check both the quality of the automatic annotation and the spatial and temporal understanding a model has learned.
The ecosystem is aimed at learning machine understanding of a dynamic world from human experience. In experiments on ScanNet, ScanNet++, Aria Digital Twin and HOLIWORLD-Bench, it shows strong performance in static perception and in dynamic tracking. Training on HOLIWORLD-Data substantially improves the spatial and temporal reasoning of HOLIWORLD-VL-4B, on HOLIWORLD-Bench and on EgoDynamic4D.
The contribution is less a single dataset than a pipeline for producing them. Recordings go in, structured world records come out, and an independent human benchmark checks the result. The authors offer it as a practical foundation for turning the continuous flow of human experience into learning resources.
Paper Quest · 7 chapters · about 42 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Human experience is plentiful. Labelling it is not. Start there.
You have hours of head-camera recordings of people moving through rooms and handling objects. They could teach a model how the world is laid out and how it changes.
The road ahead
- 1 · The problem ~7 min Head-camera recordings could teach a model how rooms are laid out and how objects change, but labelling them by hand costs too much.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min An automatic pipeline turns recordings into linked world records, which become training questions, while separate human labels check the results.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Capture recordings, build world records, derive grounded questions, train a model, then test on independent human labels.
- 6 · The experiment ~7 min You’ll guess: Did fine-tuning a small vision-language model on the automatically built data improve its spatial and temporal reasoning?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.