Agent Priors-guided Policy Learning

Agent Priors-guided Policy Learning. By Puming Jiang and six others, from the National University of Singapore. Published on the twenty-eighth of September. Its code repository is worked out: about five hundred and fifty source files, with a readme and a licence.

A robot that learns from only a few demonstrations usually needs to generalize in two different ways at once. Compositional generalization is about recombining skills to solve tasks it has never seen. Skill generalization is about the policy behind each individual skill continuing to work in situations it was not shown. These two depend on each other, and yet information gets lost in the hand-off between them.

Here is where it leaks. Where a skill works is determined by the structure its policy was trained with. Composition, though, never sees that structure. It sees the skill only through a separate description: a name, an instruction, or a symbolic operator. Whatever that description is, it omits the very thing that decides where the skill can be trusted. So the planner choosing which skill to call is choosing partly blind.

The key idea is to make each policy's structural prior part of the interface between composition and the skill. A structural prior is a statement of what a behavior depends on. The abstract's example is a grasp that depends only on the gripper's pose relative to the object. That one statement does two jobs. Built into training, it shapes where the policy generalizes. Stated in language, it tells composition where the policy applies. The same assumption that governs the skill is the one the planner gets to read.

The idea is instantiated as Agent Priors-guided Policy Learning, or APPL, and it splits the work between two agents. A construction agent takes complete demonstrations and segments them into reusable skills. For each skill it proposes several structural priors, rather than committing to one, and it trains and verifies one policy per prior. The result is a small family of policies for each skill, each carrying its own stated assumption about what matters.

A runtime agent then takes over when a new task goal arrives. It selects among these prior-specific policies and composes them toward the goal, and it does that using their interfaces, meaning the stated priors. Choice of skill and choice of which version of the skill both run through the same description of what each policy depends on.

The evaluation covers MetaWorld and long-horizon ManiSkill tasks. Across them, APPL improves out-of-distribution skill generalization, so individual skills hold up better outside the conditions they were trained in. It also enables skill compositions that had not been seen before. One ablation speaks directly to the central idea: removing the interface information substantially reduces performance. The prior is not just helping during training, then. Composition is using what it says.

Skill learning and skill composition are usually treated as separate problems with a thin label passed between them. APPL's proposal is that the label should carry the assumption the skill was built on. A training-time structural assumption, written down, becomes the bridge: it decides where a policy generalizes, and it tells the planner exactly that.