MorphAct: Robot Policies Morphing with the Worlds They Act In

MorphAct. Robot Policies Morphing with the Worlds They Act In. By Ziyan Liu and nine others, from the University of Science and Technology of China, Shanghai Jiao Tong University, and four other institutions. Published on the fourteenth of October. Its code repository is worked out: about one hundred and twenty source files, with a readme and a licence.

Scaling robot learning across many environments and interactions is part of the road to general intelligence in the physical world. Yet robot policies today share one set of parameters, and those parameters stay fixed across every interaction context. When different operations favour competing adjustments to those parameters, fitting the whole mixture can leave individual operations insufficiently adapted. Every task gets a compromise, and no task gets exactly what it needs.

The authors back this with atomic-task diagnostics. They find competing update directions in both the vision-language parameters and the action parameters. An update that improves one source task raises the loss on other tasks. The conflict is real, and it reaches both halves of the policy.

MorphAct responds with contextual parameterization. Instead of fixed weights, the policy's parameters are adjusted dynamically as the interaction context changes. The policy, in the words of the title, morphs with the world it acts in.

Two coupled generators produce context-dependent low-rank updates, one for the vision-language module and one for the action module. The coupling mirrors how the policy computes: vision-language updates shape the representations that condition how the action-side parameters are generated. Both generators are trained together through the policy's native action objective, with no operation labels and no expert-adapter targets. Nobody has to tell the system which operation it is performing.

MorphAct improves performance across three vision-language-action backbones on LIBERO and Meta-World, with gains of up to sixteen point three percent. In real-world evaluation, it gains up to fourteen percent over pi zero point five. Ablations support both contextual parameter generation and the coupling of representations.

Rather than asking one fixed set of weights to compromise among tasks that disagree, MorphAct lets the weights shift with the situation, and keeps the action side listening to what the vision side has learned.

MorphAct. Robot Policies Morphing with the Worlds They Act In. By Ziyan Liu and nine others, from the University of Science and Technology of China, Shanghai Jiao Tong University, and four other institutions. Published on the fourteenth of October. This is the long version: the whole paper rather than the abstract.

Placing a block in a tray is one instruction and four different jobs. The robot moves through free space, makes contact, transports the block, and then positions it precisely. Each of those wants different visual cues and different control. Yet a vision-language-action policy such as X-VLA, pi zero or pi zero point five runs all of it through one fixed set of weights. Context changes the activations, but the effective parameters stay where they are, and that remains true after full fine-tuning or low-rank adaptation. If the training signals from different operations pull those shared weights in opposite directions, fitting their mixture leaves each one under-served. The authors first measure whether that pull is real.

The diagnostics use the sixty-five atomic tasks of RoboCasa and one fixed pi zero point five checkpoint. First, gradient direction. Comparing gradients from batches of different tasks, close to half the comparisons point in opposing directions, about forty-eight percent in both the vision-language module and the action module. Batches from the same task disagree too, but less often: about thirty-nine and forty-three percent.

Second, an actual update. One gradient step on a task's batch always lowers that task's own loss, in all two hundred and sixty updates tried. But every one of those steps also raises the loss on between twenty-two and forty-six of the other sixty-four tasks. Third, inside each module, positive and negative contributions from different parameter groups cancel each other, so a near-zero module-level score can hide a real tug of war. Adaptation demands differ by context, in both modules, not just the action head.

MorphAct keeps the backbone frozen and adds two generators whose weights are shared across all contexts, while their outputs change with each context. At every replanning step, the context is the camera observation, the instruction and the robot state. The first generator reads the multimodal embeddings and writes low-rank updates for selected projections of the vision-language module. That adapted module then runs once and produces two things: features for the second generator, and the usual conditioning signal the action module already expects. The second generator reads those adapted features, plus a token carrying the projected robot state, and writes low-rank updates for the action module. The order mirrors the policy itself: perception shapes what control sees.

Both generators reuse a structured synthesis design from HY-WU. Queries encode which layer, which projection and which rank slot is being written, and attention runs across layers, within a layer, and onto the condition tokens, before separate readouts produce the two low-rank factors. The updates are applied as residuals without ever building a full weight matrix. Training uses only the backbone's native action objective: flow matching for the pi models, clean actions for X-VLA. No operation labels, no expert adapters to imitate. The loss reaches the first generator by two routes, through the conditioning signal and through the second generator's dependence on adapted features, and neither route is cut. At inference the updates are generated once per context, then reused across every solver step of the action chunk. The generators never see noisy actions or solver time, and there is no test-time optimization. One small detail: the factor readouts start small but not zero, because a zero readout would block gradients to the generator entirely.

On LIBERO, mean success rises on all three backbones: by four point one five points on X-VLA, two and a half on pi zero, and one point seven five on pi zero point five, whose baseline was already near ninety-seven percent. The Long suite gains most on every backbone. Meta-World, trained jointly on all fifty tasks, shows bigger moves: pi zero climbs from about fifty-one to sixty-seven percent average success, and on its Very Hard group from twenty to fifty-eight. On four real tasks with pi zero point five, every task improves; Sort Hardware goes from eight to twenty-two percent.

The comparison that tests the thesis is in the ablations, on X-VLA and LIBERO. The Unadapted variant adapts both modules but feeds the action generator features from before the vision-language update. Switching to the adapted features lifts mean success from eighty-six to ninety-seven point two percent, and on Long, coupling adds twenty-one point six points where adapting the upstream module alone added two point two. Replacing the live context with a fixed condition, or one condition per task, drops mean success to eighty-seven point five and eighty-eight point two five percent. The cost: chunk latency rises by roughly eighteen to twenty-one percent over shared LoRA.

There is no limitations section, so the gaps have to be read from what is reported. MorphAct's scores average three seeds, but the baseline scores are retained from an earlier comparison with no variances, so the smallest gains, like pi zero point five on LIBERO, carry no measure of spread against them. All ablations use one backbone on one benchmark. The diagnostics come from a single checkpoint, with repeated batches rather than independent seeds and no confidence intervals, and the largest cross-task loss increase from one step is under one percent: local loss effects, as the appendix says, not success rates. Gains are uneven too: X-VLA does not improve on Very Hard, and pi zero point five slips slightly on Spatial. Finally, the generators carry weights of their own that shared LoRA does not, plus more memory and a lower control rate, even though the factors active in any one context are matched in size.

The idea to carry away is a change in what a policy's weights are. Instead of one compromise fitted to every operation at once, the weights become an output: a shared rule produces a fresh low-rank adjustment for each moment, first for perception, then for action conditioned on that adapted perception. The ablations say the gain comes from updates that follow the live context, not from a label or from extra capacity.