HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memory

HABIT-Bench. Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memory. By Junming Zhang and nine others, from Shanghai Jiao Tong University, the Shanghai Artificial Intelligence Laboratory, and one other institution. Published on the twentieth of October.

Over long stretches of interaction with a large language model agent, users reveal their habits through the choices they make and the actions they take. Those habits are implicit information, and an agent has to infer them across many conversations. Any single observation, though, may offer only partial evidence: about the habit itself, or about when it applies. Keeping a record of past interactions is not the same as understanding them. Retention alone does not ensure an agent can infer a habit and apply it appropriately, and that is the gap this benchmark targets.

HABIT-Bench evaluates how well agent memory systems infer habits from weak evidence and then apply them to later requests. The emphasis is on both halves: working out the habit, and using it at the right moment. The object under test is the memory system, the part of an agent responsible for carrying what it has seen from one conversation into the next. A habit is a good probe for it, because no single exchange states the habit outright. It has to be pieced together from many scattered and partial observations.

The user histories are synthetic. They are constructed from task-oriented dialogue data combined with controlled habit graphs, and they span hundreds of sessions across four domains: food, finance, software and travel. The tasks are four-choice questions that test whether an agent can infer a habit and apply it appropriately. Some are deliberately tricky. They include temporary exceptions, changes in habits over time, and suggestions the user never accepted. Each of these is a case where applying a habit blindly would be the wrong answer, so the agent has to judge when a habit holds, not only what it is. A sample of the tasks goes through independent human review.

Eight memory methods are evaluated, using two models: Qwen3-235B-A22B and GLM-5.3-Flash. The best memory method reaches thirty-three point six one percent macro accuracy across the four domains. Performance stays limited even when the annotated evidence is handed to the system directly. Supplying the right past moments, in other words, does not by itself close the gap.

What the benchmark exposes is a persistent gap between having access to past interactions and being able to infer a user's habits and use them well. Remembering everything, on this evidence, is a long way from knowing someone.