MintAct: A Unified Visual Agent for Digital Environments

MintAct. A Unified Visual Agent for Digital Environments. By Mingfei Gao and thirteen others, from Apple. Published on the eighteenth of September. No code repository is linked to it.

A visual agent for digital environments is asked to do several different things. It has to ground elements of a user interface. It has to navigate across multiple steps on mobile, desktop, and web. And it has to use visual tools. Each of those has its per-domain specialists, and the question is whether one family of models can match them all.

MintAct is a family of vision-language models that unifies those capabilities, trained at two, four, and eight billion parameters. Through careful design of the environments, the data, and the training recipes, MintAct models match the performance of per-domain specialists across every one of these capabilities. One model family covers what would otherwise take several separate models, and does so at modest sizes.

Getting there took a scalable environment and reinforcement learning infrastructure. On the environment side, hundreds of concurrent instances are hosted across heterogeneous per-domain backends, serving both trajectory data collection and online reinforcement learning. For training, an asynchronous framework keeps explicit control over the cross-domain training distribution, and remains stable under noisy environment feedback and off-policy drift. Keeping that distribution under explicit control matters when a single model is being trained on several domains at once.

MintAct achieves state-of-the-art performance across a wide range of benchmarks at comparable model sizes, including a score of forty-eight point nine on OSWorld-Verified.

Unification here comes without a penalty against the specialists. The enabling work is as much infrastructure as modeling: many environments running at once, and a training loop that keeps the mix of domains under control.