UniSkill: Learning Actor-Aligned Skill Proposals for an Evolving Policy

UniSkill. Learning Actor-Aligned Skill Proposals for an Evolving Policy. By Yifei Lu and six others, from Qiantang Credit, Ant Group, and one other institution. Published on the seventh of October.

Language model agents can get better across tasks by keeping reusable skills distilled from earlier interactions. Recent work goes further and optimises task execution and skill extraction together, so that the policy and the skillbank co-evolve. That creates a credit assignment problem. While the actor is still learning, rewarding a proposed skill by how it fares when reused in later training steps can conflate the benefit of the skill with the actor's own improvement. If the agent does better a few steps later, nobody can say whether the new skill helped or whether the actor simply got better in the meantime. The direct alternative, testing each proposed skill on its own, answers that question cleanly, but it requires extra actor rollouts for every proposal, and that is costly. One route is noisy and the other is expensive.

UniSkill uses one shared policy for both jobs. It interacts with the environment, and from the resulting trajectories it proposes edits to the skillbank: Add, Update, or No Edit. The actor learns from environment rewards as usual. Skill proposals learn from a different signal, called contrastive action feedback, which is designed to say whether a proposed skill suits the actor as it currently stands.

The feedback works on trajectories that have already been collected. For a given task there are earlier successful trajectories and earlier failed ones, and the current actor assigns each a log-likelihood. The feedback measures how the gap between those two changes when the retrieved skill is replaced with the proposed one. Put plainly, it asks how swapping in the new skill shifts the current actor's preference between what succeeded and what failed on that task. That gives an actor-alignment signal without running new rollouts for each proposal. One complication remains. Feedback at the level of a whole proposal can suppress an edit operation that was appropriate, simply because the skill content attached to it scored poorly. To keep exploring, UniSkill adds a skill-edit support regularisation.

UniSkill reaches ninety-eight point four percent success on ALFWorld and eighty-four point seven percent on WebShop, while joint training stays stable. Further ALFWorld experiments show that it remains effective when the shared policy runs on a smaller backbone. The implementation is publicly available.

The design answers a measurement question as much as a learning one. When the actor and the skillbank change together, a skill's apparent value is contaminated by everything else that changed. Scoring a proposal against the current actor's own preference between success and failure, using data already in hand, separates the two without paying for fresh rollouts.