RuleTrek: Benchmarking Agentic Rule Reasoning in Complex Scenarios

RuleTrek. Benchmarking Agentic Rule Reasoning in Complex Scenarios. By Chenhao Yuan and twelve others, from the Institute of Automation, Chinese Academy of Sciences, the University of Chinese Academy of Sciences, and one other institution. Published on the fourteenth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Large language models are increasingly asked to handle complex real-world tasks in specialised domains, and those domains run on rules. Finding the rule that applies, and then applying it correctly, stays hard when the facts of a scenario are scattered across multiple files. Existing rule-reasoning benchmarks do not test that. They typically represent a scenario with only a few factual statements, which makes it much easier to see which rules are relevant and how they apply. The difficulty of digging the situation out of the material has been removed before the model ever starts.

RuleTrek is a multi-domain benchmark for agentic rule reasoning in complex scenarios. It assesses three capabilities, each inside an interactive workspace. Rule Identification asks whether the agent can find the rules that matter. Rule Deduction asks whether it can reason correctly from them. And End-to-End Rule Reasoning asks for the whole job at once.

The scenarios come from a rule-grounded construction pipeline. It extracts structured rule graphs, then generates sub-scenarios that are each specific to a rule, and composes them incrementally into larger ones. Throughout, the pipeline ensures the result stays consistent and solvable, so a harder scenario is still one with a right answer.

Eleven frontier language models were evaluated. The best Rule Identification score is only sixty-two point four seven percent. The best Rule Deduction score is sixty point four three percent. End-to-End Rule Reasoning scores range from twenty-six point nine seven percent at the bottom to fifty-five point two nine percent at the top. The authors read these results as showing that current language model agents face substantial challenges in rule reasoning over complex scenarios.

Notice where the ceiling sits. Even the strongest model, on the narrower sub-skills, gets a little over three in five right, and putting the pieces together end to end drops further. The code and the dataset are publicly available, so the scenarios can be picked up and run by others.