HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games

HiSTEVE 1.0. Harnessing Human-Interface VLA Agents for Open-World Games. By Zecong Tang and five others, from the University of Hong Kong, Zhejiang University, and three other institutions. Published on the twenty-second of October. It links to a code repository that is no longer there.

A long-standing goal is a virtual embodied agent that plays an open-ended game the way a human player does. That needs two things at once: autonomy over a long horizon, and reliable control through the native interface. Minecraft, a procedurally generated open world, is a natural place to study putting them together. Existing agents tend to have one or the other. Some offer native visuomotor control but no long-horizon autonomy. Others plan at a high level, but carry out their plans through environment APIs or predefined skills rather than through the screen, keyboard and mouse.

HiSTEVE is a hierarchical vision-language-action agent, a VLA, wrapped in what the authors call a capability-aware agent harness. The harness handles orchestration above, and a single VLA handles control below. The planner is meant to hand down only subtasks that sit within what the VLA can actually do.

At the orchestration level, the harness has structured memory for tracking progress, and a VLM planner that selects subtasks within the VLA's capabilities. The memory keeps track of where the agent is in a long task, and the planner decides what the VLA should attempt next. Knowing when a subtask is finished gets its own component, introduced so that completion can be determined reliably: a policy-coupled task completion state detector. As the name says, it is coupled to the policy: it conditions on the VLA's own policy context, and from that it signals a candidate completion.

At the control level, one VLA executes every subtask, perceiving through vision only and acting through native keyboard-and-mouse actions. Some of those interactions are with a GUI and some are not, and the two can interfere. To reduce that interference, the model gets regime-specific RMSNorm branches and separate mouse-movement token embeddings. Training proceeds through world knowledge injection, imitation learning, and reinforcement learning with environment feedback, aimed at reliability and at recovery.

On atomic tasks HiSTEVE reports state-of-the-art performance: eighty-five point zero percent on MineStudio Bench, and an average success rate of thirty-seven point four percent on OpenHA. Beyond single tasks, it completes long-horizon ones, such as crafting an iron pickaxe from scratch, the way a human player would. That last result is the one the harness exists for: a sequence of subtasks, each executed through the native interface, carried through to the end.

Agents in this setting have tended to choose between steering with a mouse and planning far ahead. HiSTEVE splits the job: a harness that remembers, plans and judges completion, over one VLA that only ever sees pixels and only ever presses keys.

HiSTEVE 1.0. Harnessing Human-Interface VLA Agents for Open-World Games. By Zecong Tang and five others, from the University of Hong Kong, Zhejiang University, and three other institutions. Published on the twenty-second of October. This is the long version: the whole paper rather than the abstract.

Picture a Minecraft player who starts with nothing and is asked for an iron sword. They chop a tree, craft a wooden pickaxe, mine stone, make a stone sword, mine iron ore, smelt it, and only then craft the sword. Seven stages, all done through screen, keys and mouse. Vision-language-action models map first-person pixels straight to keyboard and mouse actions, but keep no persistent record of where the task stands. Language-model planners reason well over long tasks, but many act through environment APIs or predefined skills, which most games do not offer. HiSTEVE tries to do both. Stop a subtask too early and it is unfinished. Stop it too late and the agent wastes actions and may undo progress. And because a long task calls the low-level policy again and again, small failure rates per call compound.

Two ideas come first. The first is a hierarchy with two clocks. On the slow clock, a planner picks one atomic subtask at a time, such as mine stone. On the fast clock, a single VLA built on a seven-billion-parameter Qwen model emits a keyboard-and-mouse action twenty times a second, from the current frame and a fixed instruction. Planning is reference-free: no expert route, no predefined sequence, only the final goal. The planner may choose only from a library of subtask schemas restricted to what the VLA can actually do. Each schema states when it applies, what visible completion and failure look like, and its execution budget. None contains a scripted implementation. Every physical step still comes from the policy.

The second idea is a conflict inside the policy. Walking the world and working an inventory screen are different regimes. Outside a menu, moving the mouse turns the camera. Inside one, the same motion moves a cursor. One action, two meanings, two kinds of image, and training a single network on both invites negative transfer.

Before fixing the conflict, the authors measured it. They trained a fully shared model on both regimes and compared gradients from interface batches with gradients from world batches, against a reference of two halves of the same regime. The gap stayed positive through training and at every layer, and was largest in the two RMSNorm modules: about zero point six for the input norm, against roughly zero point zero four for the feed-forward network. Those norms are just per-channel scale vectors, so the fix is cheap. Regime-specific RMSNorm keeps two scale vectors in each norm of every decoder layer, one per regime, chosen by the current subtask. Everything else is shared. Dual-Mouse gives cursor movement its own action tokens, separate from camera tokens. Both sets decode to the same physical mouse bins, so the split lives only in the representation. Training runs in three stages: world knowledge, imitation, and reinforcement learning, which scores several branches from the same start on environment-verified completion, and pairs successful and failed segments from matched states, so a correct prefix inside a failed run is not punished.

Then the handoff. A task completion state detector watches each invocation. It shares the VLA's image encoder, looks at four earlier frames from the last eight hundred milliseconds, and through cross-attention every fourth layer reads the VLA's own final hidden state for the current frame and instruction. When its completion probability clears a threshold on two consecutive queries, control returns to the harness. But a handoff is not a success. The harness checks the visual evidence itself: a resource counts only once the inventory change is seen, and a crafted item only once it sits in the player's inventory. It labels the outcome success, failure or interrupted, then advances, recovers locally, or replans. Only verified transitions enter memory, which has working, episodic, procedural and world stores.

The headline is eighty-five percent on MineStudio Bench, twenty points above the strongest baseline, and thirty-seven point four percent on OpenHA, barely above it. On OpenHA the gain is in embodied tasks; interface and combat tasks sit slightly below the baseline. The comparison that tests the thesis is the harness ablation. With the VLA and the planner held fixed, the full harness is swapped for fixed-horizon handoff, where each instruction simply runs for a set duration. Atomic scores barely move: MineStudio goes from eighty-one point two percent to eighty-five. Goal success on LongBench, the authors' long-horizon suite, goes from ten point seven percent to twenty-five.

The training ablation points the same way. Reinforcement learning lifts MineStudio from sixty-five point one percent to eighty point seven, yet leaves LongBench goal success flat at fourteen point three. The detector, tested on four thousand six hundred and seventy-four balanced samples, reaches zero point eight six three accuracy. GPT five point five, prompted with the same frames, reaches zero point seven eight, and the open Qwen models almost never call a task complete.

The paper names two limits in its discussion appendix. The atomic interfaces are defined by hand, and capability is bounded by the existing VLA: a new skill needs new data and retraining, and nothing learns continually from deployment. Planning also relies on an off-the-shelf model, GPT-4o, so the whole system depends on that backend. The evidence has its own edges. LongBench is fourteen tasks of the authors' own design, and success stays low: recovery and open-ended tasks reach fourteen point three percent. The best long-horizon family, at forty-two point nine percent, carries a standard deviation of fourteen point four. And the authors say the gradient analysis does not establish a causal source.

The idea to carry away is to separate stopping from succeeding. A learned detector decides when to hand back control. The harness, from visual evidence, decides whether anything was achieved, and only that verified state is remembered. Pair that with a policy that splits its normalization where the two regimes actually conflict, and the same controller goes further, because it is organized, not because it got stronger.