In-Context Learning for Robots: Methods and Applications

In-Context Learning for Robots. Methods and Applications. By Haojian Huang and thirty-eight others, from Knowin AI, the Hong Kong University of Science and Technology, Guangzhou, and six other institutions. Published on the twenty-eighth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

A general-purpose robot has to work out what a new task requires and then turn that understanding into the right physical action. In-context learning for robots supports this. It uses demonstrations and interaction to direct competence the robot already has, while its neural parameters stay fixed during deployment. The robot is taught by being shown, not by being retrained. This review gathers that literature, across both manipulation and navigation.

The organising principle is the interface: how contextual evidence, the demonstrations and interactions, gets connected to execution. Four families follow from it. Context-conditioned policies. Geometric demonstration transfer. World-model-based control. And skill- and agent-based execution. Setting these interfaces side by side makes their transfer assumptions explicit, and clarifies the different roles that training, correspondence and memory play in making context useful.

One thread runs through all four: whether what was taught survives change. The review examines how each mechanism preserves taught requirements as objects, environments and execution conditions shift. A second thread is evaluation. The analysis links method design to evaluation practices that separate three things: responsiveness to teaching, physical transfer, and benefit from retained experience. A method can look good on one of those and still fail another, which is why keeping them apart matters.

The review ends with an agenda. It connects learning tasks compositionally and transferring them faithfully with something larger, physical recursive self-improvement, in which experience improves the robot's ability to learn the tasks that come after. On that view, the point of a demonstration is not only the task it teaches but how much easier it makes the next one.

In-Context Learning for Robots. Methods and Applications. By Haojian Huang and thirty-eight others, from Knowin AI, the Hong Kong University of Science and Technology, Guangzhou, and six other institutions. Published on the twenty-eighth of September. This is the long version: the whole paper rather than the abstract.

A packing robot may already grasp every part in a kit and still not know what to do. It may need a demonstration to fix the order the parts go in, the history of the attempt to know which placements are done, and contact evidence to cope with a tighter fit. The same visible scene can call for different actions. In-context learning for robots, as the authors define it, is how demonstrations, corrections and interaction change deployed behavior without another task-specific update to the neural weights. Fine-tuning writes new evidence into the weights. In-context learning makes it available at the moment of decision.

Two ideas come before the methods. The first is where adaptation lives. The survey separates five carriers: supplied context, a recurrent state, an external memory of episodes, fast weights that change during deployment while the base stays fixed, and weight updates that persist into later tasks. Only the first three keep the neural parameters fixed. The split matters because each carrier has its own test. Remove the supplied evidence, clear the recurrent state, suppress retrieval, or restore the weights from before adaptation, and see which change explains the behavior.

The second idea is a ladder of six learning horizons. The first two are programmed control and learned policies. The third and fourth are the in-context ones: adapting from the history of interaction, and inferring a new task from teaching. The fifth and sixth are stated as research objectives rather than achievements. Physical recursive self-improvement is experience that makes the next task easier to learn. Collective knowledge evolution extends that loop to robots with different bodies exchanging verified lessons.

The core taxonomy follows the deployed path from context to action, and sorts methods by the intermediate that execution consumes. Context-conditioned policies turn evidence into an action distribution, either by retrieving and replaying recorded actions or by encoding the example and generating new ones. Geometric demonstration transfer reconstructs a motion or contact reference, aligns or retargets it to the current scene, and tracks it. In the rigid case, the demonstrated gripper motion is expressed relative to a reference object, then placed around that object's new pose. World-model-based control predicts future observations or states, then either decodes actions from a desired future or scores candidate actions by the futures they would cause. Skill and agent-based execution produces a specification, a skill sequence, a program or a tool call, which a separate executor carries out.

Each intermediate carries its own transfer assumption. A policy's representation must preserve the taught distinction. A geometric reference needs the matched interaction to still apply. A predicted future must be physically realizable. A program can only pass on what the skills can express: if interpretation drops a handle-grasp requirement, the executor cannot recover it. Across all four sit shared mechanisms, correspondence and memory, plus what the authors call model-external adaptation, where a fixed model proposes revised guidance, code or retrieval rules, kept only if a validation step accepts them.

A survey runs no experiments of its own. What it offers is a census and a set of reported comparisons. The census covers four hundred and twelve references, seventy-two point one percent first released since twenty twenty-four. Of two hundred and sixty method references, a hundred and twenty-three are context-conditioned policies and ninety-one are skill and agent systems; geometric transfer has twenty-four and world models twenty-two.

The comparisons that test the thesis most directly concern what makes teaching informative. In Behavior Prompting Policy's drawing study, the budget is held at ten thousand demonstrations. Spread over five hundred tasks, the error is nine point five pixels. Spread over two thousand tasks with five each, it falls to three point four. Diversity in what examples teach beats repetition. Skild AI's report scales training data while comparing two kinds of prompt. On unseen tasks, demonstration prompting climbs from one percent to sixty-six percent; language prompting climbs from zero to nine. Similar performance on familiar tasks can hide very different returns to new teaching. Counterweights are reported too. LMAct finds that more expert episodes often add little, and in REGENT's unseen-embodiment comparison, plain Retrieve-and-Play beats the frozen contextual transformer.

The paper has no limitations section. It describes its method as a mechanism-oriented narrative synthesis with targeted updates through the twenty-ninth of September, and its counts describe the reviewed corpus rather than the field. The numbers it collects are within-study comparisons on different tasks, so they cannot be ranked against one another. Some of the most quotable results come from industrial blog posts, and the survey notes that Skild's per-step metric includes human recovery, mainly for the language baseline. Much of the evaluation chapter is proposal: the control tables and the drawer-reuse figure describe candidate tests, not completed ones. One disclosure is also on the page: GLOW, which sits in the taxonomy and in the production table, is a technical report from Knowin AI, the authors' own first affiliation.

The idea to carry away is that contextual learning succeeds or fails on whether a task-relevant distinction survives from evidence to execution. A demonstration that says plates first, or grasp by the handle, passes through selection, interpretation, an intermediate and an executor, and can be lost at any of them. Where it is lost says what to fix: data, representation, or motor competence. For the longer game, the authors' measure is not a stronger solver today but a successor that learns the next task from less teaching. Their concrete case is LMPC, where training on successful teaching sessions raised held-out success from thirty-nine point four to sixty-six point three percent, while sessions needed fewer turns.