The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks

The Tasteful Agent. Measuring and Improving Taste in Long-Horizon Tasks. By Wenbo Pan and eight others, from the City University of Hong Kong, Independent Researcher, and one other institution. Published on the twenty-second of September. Its code repository is public and real: seventeen source files.

LLM agents increasingly work on long-horizon tasks, and the decisions they make along the way decide how the whole run turns out. Which hypothesis to test. Which implementation to build on. Making these decisions well is becoming a key capability for both engineering agents and research agents. The authors call that ability the taste of an agent. Existing benchmarks measure end-to-end success on long-horizon tasks, but none measures taste directly. An agent can succeed despite poor choices on an easy task, or fail despite good ones on a hard task, so end-to-end success blurs the quality of the decisions that led to it.

Taste-Bench measures it through questions built around decision forks. A fork is a point in a trajectory where several directions are available and one of them leads to a better outcome. The evaluated model chooses among the directions without seeing what happens after the fork, which is exactly the position an agent is in when it has to decide. The question isolates one decision, with the outcome hidden, and asks whether the model would have picked the better branch.

The questions are constructed automatically from trajectories that agents produced on engineering and research tasks. Forks are mined from two sources: parallel attempts at the same task, where different runs went different ways, and detours inside a single trajectory. No human annotation is needed. Parallel attempts give natural forks, because the same starting point led to different results. Detours give forks within one run, where the agent went one way, turned back, and went another. Both are already present in agent trajectories, so the benchmark can be built by mining rather than by writing questions by hand.

Frontier models were evaluated, and the best answers only fifty-nine point seven percent of the questions correctly. Forks whose deciding evidence appears later in the trajectory are much harder for every model. And a larger reasoning budget does not improve accuracy, so thinking longer at the fork does not help.

The last result turns from measuring to training. Taste can be trained. The judgment of a teacher that has seen the outcome is distilled into a student model. The student then makes better decisions on unseen tasks, and improves end-to-end success on held-out SWE-bench Pro tasks. The teacher's advantage is hindsight, and distillation turns that hindsight into judgment the student can use before the outcome is known.

Two findings sit side by side. More reasoning at the moment of choice does not help, while learning from a teacher that knows how things turned out does. The authors frame taste as something distinct from end-to-end success, and show that it can be both measured and improved. The best frontier model getting only about three in five forks right leaves a lot of room, and the distillation result shows one way to start closing it.

The Tasteful Agent. Measuring and Improving Taste in Long-Horizon Tasks. By Wenbo Pan and eight others, from the City University of Hong Kong, Independent Researcher, and one other institution. Published on the twenty-second of September. This is the long version: the whole paper rather than the abstract.

An agent on a long task makes decisions whose effects outlast the current step: which hypothesis to test, which implementation to build on. A wrong decision often looks reasonable at the moment, and its cost appears only much later, after much of the budget is spent. Existing agent benchmarks report whether the task was finished, not the quality of the decisions along the way. The authors call that quality the taste of an agent, and ask whether it can be measured without expert annotation.

The key observation is that the later part of a trajectory is hindsight evidence for its earlier decisions. Agent systems routinely make several attempts at the same task, and those attempts often share a beginning and then diverge. That point is a decision fork. Because the attempts share an equivalent prefix, and sampling randomness decided which way each went, the difference in outcomes mainly reflects the choice at the fork. Each fork becomes a question: the task, the trajectory up to the fork, and two directions. Everything after is hidden, and the label is whichever branch produced the better outcome.

Forks are mined two ways. Parallel forks pair one passing and one failing attempt at the same task, and locate where they part. These capture wrong turns the agent never noticed. Detour forks come from a single run in which the agent took a direction, hit an observed failure such as a non-zero exit code or an error message, and recovered with a different direction that worked. The fork sits just before the abandoned direction, so the question asks whether a model can see the failure coming earlier than the acting agent did. A generator model writes both directions in neutral, parallel wording, and must quote the trajectory verbatim; those quotes and their order are checked mechanically.

Two filters follow, run by four judge models that exclude the generator. The trivial filter shows the judges only the two candidates; if every judge gets it right, the wording gives it away and the question is dropped. The undecidable filter shows the full record and outcomes; the question survives only if every judge agrees with the label. Of four thousand six hundred and fifty-seven candidate forks, ten point eight percent survive, leaving five hundred and two questions: three hundred and ninety from engineering and one hundred and twelve from research. Scoring is strict. Each question is asked twice with the candidates swapped, and counts only if both answers are right, so guessing scores twenty-five percent.

Fourteen models were evaluated. The best, GPT five point six Sol, averages fifty-nine point seven percent, on questions that are all binary choices. The more telling result is the time horizon. A judge annotated each fork by how far past it one must see before the right answer is justified, from evidence already in the prefix up to substantial later work. Averaged over the fourteen models, accuracy falls from sixty-two point three percent at the first level to twenty-one percent at the last, near the twenty-five percent of random guessing. Construction matters more than domain: detour engineering is the hardest cell, with a mean of thirty-five point nine percent against fifty-eight point one for parallel engineering.

Thinking longer does not help. Two models were rerun at three reasoning settings each, and moving from lowest to highest changed accuracy by less than three points, with no interaction with the horizon. The models did spend the most reasoning tokens on the long-horizon forks, precisely where they scored worst. Nor is taste a restatement of coding ability: the Pearson correlation with SWE-bench Verified is only zero point six three, and the four top models there, within four points of each other, spread ten point seven points apart here.

Then training. One Qwen three point six model, twenty-seven B, is both teacher and student. The teacher sees the question plus a description of the supported candidate; the student learns to match the teacher's reasoning tokens while seeing only the question. On held-out tasks, its both-orders accuracy rises from thirty percent to forty-seven point nine. Then its judgments were written as advice notes into the prompt of an executor agent, on forty-one held-out SWE-bench Pro tasks. Success went from fourteen point six percent with no advice to thirty-three point seven with student advice, against thirty-nine percent when every note was correct.

The paper has no limitations section, though its appendices supply caveats. The correlation with SWE-bench Verified rests on eleven models, with an interval running from nearly zero to zero point eight nine. The end-to-end experiment is thinnest. It covers forty-one tasks, and each task's advice comes from forks mined in earlier runs on that same task, so it is task-specific guidance rather than judgment exercised live. A significance test is reported for correct advice but not for student advice. The advice header also tells the executor to verify its patch before submitting, and no condition tests that header without notes. Distillation was tested only in engineering, on one base model.

The idea to carry away is that an agent's past runs already contain the answer key to its decisions. Where two attempts diverge, or one run backs out of a mistake, the recorded outcome says which way was better, with no expert needed. Measured that way, frontier models are weakest exactly where taste matters most: when the deciding evidence lies far past the fork. More thinking does not close that gap. Learning from hindsight, the authors show, begins to.