Encore: Few-Shot Agentic Discovery of Manipulation Strategies

Encore. Few-Shot Agentic Discovery of Manipulation Strategies. By Yifan Kang and three others, from the Massachusetts Institute of Technology, the twenty seventy-seven AI Open Source Foundation, and two other institutions. Published on the twenty-ninth of September. Its code repository is worked out: about five hundred and fifty source files, with a readme.

Coding agents can now write, run and debug programs with little human help, so pointing one at a robot seems natural. The trouble is how robot tasks are usually specified: by a single sentence. That sentence leaves out how to grasp the object, in what order to make contact, and what the finished result should look like. An agent handed only the sentence has to discover all of that by trial and error. The missing information is not about programming at all. It is about the task, and the sentence simply does not contain it.

ENCORE gives the agent a few demonstrations, but not as training data. They are evidence to read. Instead of fitting a policy to the demonstrations, the agent studies them the way a programmer studies an example, and then writes the policy itself as a program.

A deterministic builder first distils each demonstration into a pack: keyframes from multiple views, gripper events, frame strips, and the full trajectory. A coding agent studies that pack and writes a policy program against a fixed perception and action API. It refines the program iteratively over a few development rollouts, and then freezes it. Only after freezing does the program meet a sealed evaluation, and that evaluation never reveals the success signal, so the agent cannot keep tuning against the test.

On LIBERO-PRO, the agent's very first program already succeeds in half of the perturbed tasks when demonstrations are available, and in just one task without them. After refinement, the frozen programs outperform the strongest prior agentic system run with the same language model: ninety-six point three percent against eighty-nine point three percent.

On RoboDojo tasks whose instructions leave the goal unstated, no program succeeds without demonstrations at all. ENCORE also runs on a real bimanual robot, where it learns a cube handover and a cup inversion from five demonstrations each.

The RoboDojo result is the clearest statement of the premise. When the sentence does not say what success looks like, the agent's trial and error on those tasks does not recover it, and no program succeeds. A handful of demonstrations, read as documents rather than fitted as data, supplies the details the instruction left out: how to grasp, in what order to make contact, and what the end state should be. The demonstrations stop being examples to imitate and become a specification to read, and the coding agent's existing ability to write, run and debug programs does the rest.