Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive Optimization

Ask Before You Optimize. Dynamic Pre-Formulation Clarification for Interactive Optimization. By Sihan Ge and five others, from Cardinal Operations and Shanghai Jiao Tong University. Published on the fourth of September. Its code repository is worked out: about forty source files, with a readme.

Large language models are increasingly used to turn a natural-language problem description into a formal optimization model. Realistic operations research requests are often incomplete. An objective may be missing, or a constraint, or a business rule, and any of those changes the mathematical program that comes out. Existing evaluations largely assume a complete specification, which means they never ask the question that comes first: does the agent know when clarification is needed, before it models anything.

OR-Clarify is a benchmark for pre-formulation clarification, and its design follows from what it wants to test. Each task presents a partial public problem description and withholds structured hidden slots. Agents are then evaluated through bounded interaction with a simulated user. Bounded is the operative word: asking is possible but not free, so an agent that asks about everything is not rewarded for it.

Four things are measured rather than one. Slot recovery, meaning how much of what was withheld the agent got back. Stopping behavior, meaning whether it knew when to stop asking. Silent assumptions, meaning what it filled in without checking. And interaction cost. The benchmark supports both open-ended and choice-based clarification, so an agent can be tested on generating questions and on selecting among them.

Alongside the benchmark the authors propose Interactive Optimization, or InterOPT, a two-stage framework that identifies unresolved formulation-critical gaps and uses them to guide whether to ask the next question or to stop. In the choice-based experiments InterOPT substantially outperforms all baselines on exact slot recovery. In the open-ended setting it remains competitive with strong prior methods rather than beating them.

Taken together the two reframe the problem. Assistance with operations research becomes a selective completeness decision: clarify when needed, stop when ready, and quantify what remains missing. That third item is the one existing evaluations had no way to report, because a benchmark assuming a complete specification has nothing left over to count.

Ask Before You Optimize. Dynamic Pre-Formulation Clarification for Interactive Optimization. By Sihan Ge and five others, from Cardinal Operations and Shanghai Jiao Tong University. Published on the fourth of September. This is the long version: the whole paper rather than the abstract.

A user describes a delivery problem: couriers, stops, capacities, a time budget. They do not say whether a courier has to return to the origin at the end of its route. An agent formulating this as an optimization model has to decide, and if it silently assumes a closed tour, it has changed the constraint structure of the problem without asking anyone. The answer it produces will be correct for a problem nobody posed. That is the failure the paper is about, and it names it premature formulation: declaring readiness while core business facts remain unclarified, or filling them in with unsupported defaults.

The reason this goes unmeasured is an assumption in how these systems are evaluated. Benchmarks for turning natural language into optimization models largely assume a sufficient specification and ask whether the agent can express or solve the model it was given. That skips the earlier question of whether the available information supports a meaningful formulation at all. And the omissions that matter are not superficial. A user may give capacities and demands without stating the objective, or state a routing rule without saying whether the time windows are hard or soft. Each of those changes the mathematical structure of what gets built.

The definitional move that makes this tractable deserves stating precisely. A fact is formulation-critical if its value can change the structure of the resulting optimization formulation. That is not a judgement about importance or about what a user might like to be asked; it is a testable property of the fact. From it the task falls out: decide whether the current public specification is model-ready, and if it is not, recover the missing formulation-critical facts with as little interaction as possible. Note how different this is from generic abstention, which asks whether a question can be answered. Here the question is whether the information suffices to build the correct model.

The proposed method separates two jobs that are usually tangled. A first stage searches for formulation-critical gaps and maintains a record of those still unresolved by the public transcript, carried across turns. A second stage takes the currently unresolved gaps and decides only whether to ask a targeted question or to stop. Keeping persistent gap tracking apart from action selection is the design claim. Four things are then measured rather than one: how much of what was withheld got recovered, whether the agent knew when to stop, what it filled in without checking, and what the interaction cost.

The construction pipeline is the main boundary. Instances are derived from fully specified tasks by removal, and the simulated user answers only from the facts that were removed, so the interaction is cleaner and more cooperative than a real stakeholder would be. Real requirements shift as a business changes, and nothing here models that. The measurements also depend on the slot-level rubrics that define what counts as recovering a fact, which are artifacts of the same pipeline. And the framing is deliberately narrow: this is everything that happens before an optimization model is built, and says nothing about whether the model that follows is any good.

The reframing in the title is the durable part. Assistance with operations research becomes a selective completeness decision: clarify when needed, stop when ready, and quantify what remains missing. That third clause is the one existing evaluations had no way to express, because a benchmark that assumes a complete specification has nothing left over to count. An agent that reports what it still does not know is doing something categorically different from one that quietly picks a default and proceeds.