TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning

TouchScale. 500 Hours of Human Vision and Touch for Visual-Tactile Learning. By Dayou Li and twenty-five others, from Texas A and M University, Google DeepMind, and thirteen other institutions. Published on the seventh of October. Its code repository is worked out: about one hundred and forty source files, with a readme and a licence.

Large-scale egocentric video of people interacting with the world is becoming an important source of physical supervision for embodied learning. Video, though, leaves out the very thing that characterizes physical interaction. Contact and pressure go unrecorded. Visual-tactile datasets supply that missing supervision, but their synchronized tactile data remain far smaller in volume than human video. And the largest of them often merge recordings from different sensors or different annotation procedures, which makes it difficult to isolate what data scale on its own contributes. TouchScale is the answer to both problems: a five hundred hour dataset of contact-rich human interaction, recorded with a single unified wearable setup.

The recordings are organised around approximately two thousand predefined task descriptions, spanning everyday activities on one side and structured manipulation on the other. Each recording temporally aligns three streams. There is egocentric RGB-D video, seen from the wearer's point of view. There is wrist RGB video. And there are dense, full-hand, bimanual tactile measurements, which is to say touch recorded across the whole of both hands. The planned public release covers all of the synchronized visual-tactile recordings, together with reconstructed object models.

The defining choice in how it was made is consistency. Every hour comes from the same wearable setup, rather than from a merger of sensors and procedures, and that is what lets the authors study scale directly. With the sensor and the collection protocol held fixed, they vary how much TouchScale data is used. Both zero-shot tactile prediction and robot success show an overall upward trend as more of it goes in. The effect of scale, in other words, is no longer tangled up with the effect of changing equipment.

Three sets of results sketch who it serves. For tactile prediction: compared with prior tactile data, training on the full TouchScale raises zero-shot contact IoU, measured on data from a tactile sensor never seen in training, from zero point one three four to zero point three eight three. For visual perception: pretraining a visual encoder on TouchScale yields the highest action recognition accuracy on three benchmarks among the compared visual-tactile datasets.

For robotics: used for visual-tactile mid-training of a robot policy, TouchScale improves the average real-world success rate across four contact-rich manipulation tasks from twenty-two point five percent to fifty-seven point five percent. The audience therefore spans tactile perception, visual representation learning and robot manipulation, and anyone working on scalable visual-tactile learning more broadly.

The authors read these results as evidence that human visual-tactile data, collected at scale with consistent sensing, benefits both perception and robot manipulation. Five hundred hours is the headline figure, but the dataset's argument rests as much on its uniformity as on its size. One setup across every recording is what allows the scaling trend to be read cleanly.