Improving Proactive AI Assistance with Hierarchical Procedural Understanding

Improving Proactive AI Assistance with Hierarchical Procedural Understanding. By Jin-Seop Lee and seven others, from Sungkyunkwan University and Nanyang Technological University. Published on the sixth of October. No code repository is linked to it.

A proactive AI assistant watches what a user is doing and keeps deciding whether to offer new guidance or stay quiet. To do that well it needs three abilities: give guidance that fits the task, judge when the next piece is due based on progress, and adjust how detailed that guidance is to the user's expertise and needs. What now exists to support this is the ProactiveCoach suite: a training set, a benchmark, and fine-tuned vision language models with an adaptive guidance system.

The training set is ProactiveCoach-Instruct. Its defining feature is hierarchy. Guidance is structured at three levels, phase, step, and action, so a model can learn both fine-grained task progress and the broader procedural context around it. The benchmark is ProactiveCoachBench. It evaluates whether a model offers appropriate guidance at the right time across the different guidance levels, and whether it adapts when the requested level changes partway through.

The suite is a response to a gap in existing datasets. Those either focus on detection-based proactive understanding, or they supply procedural guidance at one fixed granularity. The authors argue that fixed granularity tells a model little about fine-grained progress or the wider procedure, which makes it hard to judge when something is complete and hard to change the level of guidance. The hierarchical structure is built to supply exactly that missing information.

The resource is aimed at people building assistants of this kind, and the authors test it themselves. Fine-tuning pretrained vision language models on ProactiveCoach-Instruct works across backbones. Compared with fixed-granularity supervision, hierarchical supervision improves overall performance by up to nine point six percentage points. They then combine a fine-tuned model with a lightweight guidance router to form the adaptive guidance system. With no additional fine-tuning, that system beats an in-context adaptation baseline by fifty-seven point one percentage points across four guidance-level transitions. A project page is available.

The suite treats the level of guidance as something to be learned rather than fixed in advance. Training data that marks phases, steps, and actions gives a model a sense of where in a procedure the user stands, and a benchmark that changes the requested level checks whether that sense can be put to use.