Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms

Verifiable Hidden Dynamics Play. Generating Agentic RL Environments from Solved Mechanisms. By Xinjie Shen and seven others, from the Georgia Institute of Technology and Alibaba Group. Published on the twenty-third of September. No code repository is linked to it.

Language-model agents increasingly face long-horizon tasks with evolving state, interdependent decisions, and delayed outcomes. Scaling their training needs three things: diverse agentic environments, dependable outcome signals, and low cost to extend. Existing generation pipelines usually build an environment first, and only afterwards define its outcome rule or annotate its trajectories. Dynamics and evaluation then have to be aligned after the fact. That ordering is where mismatch creeps in. An environment built without its scoring rule in mind can behave in ways the rule later fails to judge fairly.

VHD-Play reverses that dependency. It first samples a mathematical model and solves it. Only then does a corpus-grounded setter render the model's decision process as stateful tools. The executable dynamics and the reference for scoring trajectories are both inherited from the same solved model, so they come from one source instead of being reconciled later.

The pipeline produces three thousand three hundred diverse agentic environments, at a cost of a few cents each. Training Qwen3.6-35B-A3B on three families of them raises its mean agentic score from zero point two zero four to zero point eight one five, in a diagnostic spanning five families. That is roughly a fourfold rise, from training on only three of the families.

The gains appear on held-out instances from all three training families, and on eight mechanism families never seen in training. They also extend beyond the generated environments, to external benchmarks for general function calling, travel planning, and three hundred sixty-five day e-commerce. On E-Commerce Bench, the trained checkpoint completes every run without bankruptcy and exceeds Qwen3.7-Max.

A comparison between problems written out in full and stateful versions that reveal or hide their parameters shows that most of the learnable gap lies in stateful interaction, not in solving the underlying problem. A frozen setter of thirty-five billion parameters realizes larger environments, and scale-matched training keeps its gains as mechanism size and horizon grow. The first result says where the difficulty sits. Agents struggle less with the mathematics than with acting in a world whose state they must track, and that is exactly what stateful environments train.

A training substrate that keeps evolving is the potential the authors see in this. The underlying move is to begin from the answer. When an environment is rendered from a model that has already been solved, knowing what a good trajectory looks like is not a separate annotation job. It comes with the environment. And since each environment costs only a few cents, the supply can keep growing as the mechanisms grow.