ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds

ExplorationBench. Measuring AI Systems' Exploration in Verifiable Alien Worlds. By Ming Zhang and nineteen others, from Fudan University, the Hunyuan Team at Tencent, and one other institution. Published on the twenty-fourth of September. No code repository is linked to it.

Scientific discovery begins where known problems end. Past that edge, an AI system has to explore: frame hypotheses, design experiments, and iterate on what comes back. Measuring that ability runs into two obstacles. The first is verification. If a hypothesis is genuinely new, how would anyone check whether it holds? The second is attribution. Even when a system reaches the right answer, it is hard to tell whether it discovered it through exploration or simply recalled related knowledge from its pre-training data.

ExplorationBench measures whether a system can acquire unfamiliar rules by exploring, and then apply them. The authors describe it as turning a wicked problem, the evaluation of scientific exploration, into a concrete and tractable framework. The move that makes it tractable is to build the test inside verifiable Alien Worlds, and each of their two defining features answers one of the obstacles. Their rules are executable, so every answer can be checked exactly. And their rules conflict with familiar knowledge, so recall alone cannot solve the tasks.

There are two sandboxes. AlienCode holds thirty-one discovery targets and seventy tasks. AlienLogic holds twenty-four discovery targets, and also seventy tasks. Each sandbox comes with three resources: a manual that is flawed, environmental feedback specific to the task, and a dedicated schema for tool calls. A system uses these to explore the sandbox first. Only afterwards does it face held-out tasks, which it has to solve with whatever it learned.

Ten AI systems were evaluated. The strongest of them can acquire unfamiliar rules and put them to use. That is the encouraging half. The other half is about stability. Performance varies substantially from one exploration trajectory to another, so the same system can fare well or poorly depending on how its exploration happened to unfold. And more exploration is not reliably better: continued exploration can stall, or even reverse gains made earlier.

The authors present the benchmark as a step toward systems that can acquire and apply genuinely new knowledge by exploring unknown environments. Its design carries that aim in a single constraint. By making the worlds alien and executable at once, it closes off memory as a shortcut and leaves exploration as the only way through, and then shows that the way through is uneven even for the systems that manage it.