PyRUA-DataFarm: LLM Agent as a Cost-Efficient Teleoperator for Real-Ready Robot Data

PyRUA-DataFarm. LLM Agent as a Cost-Efficient Teleoperator for Real-Ready Robot Data. By Kewei Zhang and thirteen others, from Peking University and NVIDIA. Published on the third of November.

Robot demonstrations are still mostly collected by teleoperation, with a person driving the robot through the task. That has two costs that never go away. Every episode consumes both robot time and operator time. And every new task starts from scratch, so nothing learned collecting one dataset makes the next one cheaper. The question the authors put is direct: can a coding agent do the teleoperator's job instead?

Their answer is the DataFarm. An agent is handed a data request and a single calibration episode of the robot in the target scene, and from those it rebuilds the task in simulation. It writes three things: the scene, an expert controller that performs the task, and a success judge that decides whether an attempt worked. Simulation then does what a human operator used to do, and it does it with exact actions. Notice how little the agent is given to start from. One calibration episode in the real scene is the only robot time spent on a new task, and everything after it is built in software: the environment, the behaviour that solves the task, and the test that grades it.

The simulator supplies the actions, and a video model supplies the visuals. Conditioned on real images, it renders each rollout as photographic video, generating all the camera views together so that they stay consistent with each other. What comes out is what the authors call real-ready data: synchronized camera views labelled with the robot's actions, on which a policy can be trained directly and then deployed on the real robot. Before anything is admitted, independent judges certify the expert controller and every rendered episode.

The test covers five real-robot tasks. Policies trained only on farm data match policies trained on the same number of teleoperated episodes: ninety-three successes in a hundred trials, against eighty-four. Under an unseen tablecloth and coloured light, the farm-trained policies keep eighty-seven successes, while the teleoperation-trained ones fall to forty-six.

Then the costs. Adapting a task is fully automatic. It costs on average twenty-one in API usage, paid once per task, plus five for a render contract. Each further episode costs on average zero point eight for simulation, rendering and review, with no robot and no operator involved.

Put together, one hour of three-view data for a task costs on average one hundred and seven, with all of those costs included. That sits slightly below a quote of one hundred and seventeen for teleoperation in China, and below two hundred and ninety for labour alone in North America.

The economics change shape as much as size. Teleoperation charges for every episode in robot and human time. The farm charges once to set a task up, and after that each episode costs a small average amount for simulation, rendering and review. On the five tasks reported, the farm's data was not only cheaper per hour than the teleoperation quotes. Policies trained on it matched the teleoperated ones in the original scene, and held up far better when the tablecloth and the light changed.