Ambient @ EgoProactive 2026 : Proactive Egocentric Assistance with Visually Grounded Supervision
Logesh Kumar Umapathi
TeamAmbient
Published September 10, 2026 · on the feed September 14, 2026 · ▲ 3 · arXiv 2609.07099
The abstract
We present our submission to the EgoProactive track of the ECCV 2026 Wearable AI Challenge, which ranked first in the large-model division and second in the <=2B division. The task requires a wearable assistant to decide after each eight-second segment of egocentric video whether to intervene or remain silent. Our approach has two main components. First, we reformulate intervention timing as single-token classification. Rather than generating either interrupt<utterance> or silent, the model predicts yes or no, and we derive the decision from the renormalised probabilities of these two tokens. This formulation improved macro-F1 by 0.249 and G-mean by 0.30 over free-form generation. Second, because labelled data were limited to the released validation set, we generated additional supervision using a tool-calling video agent that inspects each clip and assigns intervention timestamps. A narration-only alternative was four times larger and ten times cheaper, but transferred worse than supervision from an unrelated real corpus, suggesting that visual grounding is more important than annotation volume for this task.
The Brief
2:10A short podcast summary of the paper. Written and voiced by AI.
Transcript
Ambient @ EgoProactive 2026. Proactive Egocentric Assistance with Visually Grounded Supervision. By Logesh Kumar Umapathi, from Team Ambient. Published on the tenth of September. Its code repository is public and real: fourteen source files.
The EgoProactive track of the ECCV 2026 Wearable AI Challenge poses a timing problem. A wearable assistant watches egocentric video, and after each eight-second segment it has to decide: intervene, or stay silent. The obvious way to put that to a generative model is to have it write out its answer, either an interruption with an utterance or the word silent. A second constraint sits on the data side: labelled examples were limited to the released validation set.
This submission ranked first in the large-model division, and second in the division for models of two billion parameters or fewer. It has two main components. The first reformulates intervention timing as single-token classification. The second generates additional supervision with an agent that actually inspects the video.
For the first, the model does not generate a free-form answer at all. It predicts yes or no, and the decision is derived from the renormalised probabilities of those two tokens. The whole decision collapses into one comparison between two probabilities, rather than depending on how a longer generated answer happens to begin. For the second, a tool-calling video agent inspects each clip and assigns intervention timestamps. That produces training labels well beyond the small validation set, and they are grounded in what the agent saw in the footage rather than in a description of it.
The single-token formulation improved macro-F1 by zero point two four nine, and G-mean by zero point three zero, over free-form generation. The supervision result is a comparison. A narration-only alternative produced data four times larger and ten times cheaper, yet it transferred worse than supervision drawn from an unrelated real corpus. More data at lower cost lost to data that was not even about the same task.
That last comparison carries the lesson the authors draw. For deciding when to speak up, visual grounding appears to matter more than annotation volume: labels from an agent that watched each clip beat a larger, cheaper set that was built from narration alone.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A helper in your glasses has one hard job: knowing when to speak.
You are building a wearable assistant that watches a person cook or repair something through a head camera. After every eight-second chunk of video it must choose: speak up with a tip, or stay quiet.
The road ahead
- 1 · The problem ~6 min A head-camera assistant must choose to speak or stay quiet every eight seconds, but a model that writes full replies barely ever speaks.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Have the model output just yes or no, and read the speak-or-silent call from those two tokens' probabilities against a tunable threshold.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min An agent labels clips with intervention times, the model learns yes or no from them, and a threshold on renormalised probability makes each call.
- 6 · The experiment ~6 min You’ll guess: A narration-only label set was four times larger and ten times cheaper than agent labels. How did it transfer, compared with supervision from an unrelated real corpus?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.