GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations

GlobeBench. Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations. By Nicholas Saban and three others, from Patronus AI. Published on the fifteenth of October. Its code repository is public but slight: eight source files.

Text-based world models have recently shown they can simulate the state transitions of complex reinforcement learning environments, for a diverse set of uses. An agent acts, and the model, instead of a real environment, says what happens next. What has been missing is any standardized benchmark to compare these models, study them, and improve them. Without one, a claim that a model simulates an environment well has nothing shared to be measured against, and progress across different models cannot be compared.

GlobeBench is meant to be that benchmark. Each task asks whether a model can predict the environment's response to a specific action, given the full trajectory of the agent so far. Getting it right means understanding earlier state changes, following the environment's rules, and predicting the effects of the action on objects related to it and on objects that are not. That last part cuts both ways. A good simulator has to change what the action touches, and it has to leave everything else alone.

There are one thousand six hundred and thirty-two tasks, across five separate domains and nineteen simulation skills. Scoring compares the predicted response with a reference derived from actually executing the action. The comparison runs through a hybrid architecture that combines deterministic gates with a semantic judge. The reference is what really happened when the action was run, so a prediction is judged against the environment itself rather than against another model's guess.

Twenty models were tested, a mix of general-purpose LLMs and specialized text world models. The best was GPT-6 Astra, with a task success rate of forty-nine point five seven percent. Accuracy falls as an action affects a larger portion of the environment. When the agent's changes have to propagate to dependent objects, task success drops to an average of five point one percent across all twenty models. Model strengths differ from task to task, yet six hundred and thirty-one tasks remain unsolved by every model.

Hold on to that last figure. Even the leader solves fewer than half the tasks, and the hardest case is the one a simulator most needs: a change that ripples out to everything depending on it. The authors read the unsolved set as a significant gap in what general LLMs can do as world models, and GlobeBench is a way to measure it.