Focused Omni-Modal GUI Agents
Fuyao Zhang, Wenwen He, Jiaming Zhang, Yurong Hao, Xuan Wang, Guowei Guan, Che Wang, Longtao Huang and 1 others
Nanyang Technological University · Alibaba Group
Published October 19, 2026 · on the feed October 21, 2026 · ▲ 46 · arXiv 2610.26203
The abstract
Graphical User Interface (GUI) agents increasingly act over streaming omni-modal observations spanning screenshots, audio, video, interaction history, and device states. However, exposing this complete context uniformly at every step ignores that relevance is transient and sparse: the signal determining one action can become clutter at the next. Because executed actions feed back into the context, these errors compound into Sequential Distraction, driving the agent into irreversible trajectory drift. We formulate the underlying requirement as Focused Perception: a representation that is step-adaptive, action-sufficient, and non-intrusive. We realize this in FocusGUI, a lightweight adapter that dynamically prioritizes multimodal channels against a step-specific decision reference. Instead of uniformly processing all inputs, it compresses the most relevant evidence into a compact latent and subtly injects it into the frozen decoder's embeddings, guiding the model's attention without altering its native parameter space. FocusGUI trains from action supervision alone, requiring neither modality-relevance labels nor backbone fine-tuning. Across four omni-modal backbones, it raises overall Type Match by +33.2 and Exact Match by 3.0 times over the frozen backbone on average, and Type Match by +30.1 over the strongest existing enhancement method.
The Brief
2:25A short podcast summary of the paper. Written and voiced by AI.
Transcript
Focused Omni-Modal GUI Agents. By Fuyao Zhang and eight others, from Nanyang Technological University and Alibaba Group. Published on the nineteenth of October. Its code repository is worked out: about eighty source files, with a readme.
Graphical user interface agents, or GUI agents, increasingly act on streaming observations that span many modalities: screenshots, audio, video, the history of interactions, and the state of the device. The common practice is to expose all of that context uniformly at every step. That ignores something about relevance. It is transient and it is sparse, and the signal that determines one action can become clutter at the next.
Because executed actions feed back into the context, those errors compound. A distraction at one step shapes the context of the next, which makes the next distraction more likely. The authors call this Sequential Distraction, and it drives an agent into trajectory drift that cannot be undone.
They state the underlying requirement as Focused Perception: a representation that is step-adaptive, that is sufficient for choosing the action, and that is non-intrusive. Each of the three words answers part of the problem. The representation must change from step to step, because relevance does. It must carry enough to choose the action. And it must not disturb the model it feeds. FocusGUI realizes it as a lightweight adapter that dynamically prioritizes multimodal channels against a decision reference specific to each step.
Instead of processing every input uniformly, FocusGUI compresses the most relevant evidence into a compact latent. It then subtly injects that latent into the embeddings of the frozen decoder. This guides the model's attention without altering its native parameter space. Training needs only action supervision, with neither modality-relevance labels nor fine-tuning of the backbone. No one has to annotate which channel mattered at which step; the actions themselves are the only signal.
Across four omni-modal backbones, FocusGUI raises overall Type Match by thirty-three point two, and Exact Match by three times, on average over the frozen backbone. Against the strongest existing enhancement method, it raises Type Match by thirty point one.
Seeing more is not the same as seeing what matters. FocusGUI leaves the backbone and its parameters untouched, and changes only which evidence is brought forward at each step, which is how a small adapter can move results on four different backbones.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
An agent that sees and hears everything at once. The trouble is that it hears everything at once.
You are building an agent that operates a phone or computer. At every step it receives screenshots, audio, video, its interaction history and the device's state, all handed to the model at once.
The road ahead
- 1 · The problem ~7 min A phone-operating agent gets every input at every step, but only a small, shifting part matters, and its mistakes feed back into what it sees next.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Keep every input and the model untouched, and add a small learned signal each step that tells the frozen model what to emphasise now.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Split the observation into channels, weight them against this step's reference, compress, add the result to the frozen decoder's embeddings, and train from actions.
- 6 · The experiment ~6 min You’ll guess: Averaged across four omni-modal backbones, how many times higher was Exact Match with FocusGUI than with the frozen backbone alone?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.