Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching

Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control. Commanding, Approaching, Tracking and Searching. By Jaewoo Park and thirteen others, from Naver Cloud. Published on the first of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Multimodal large language models are strong perceivers of images and video. The question here is how far that reach extends into acting. Picture dropping one of these models directly into a drone's control loop, with its entire action space declared in nothing but the prompt. Recent systems approach that setting, but they increasingly narrow what the model is allowed to decide. The authors want to widen it back, and to do that they need a way to see what happens when the model is given more of the decisions and then held responsible for them.

That is what DroneCATS is for. It is a benchmark that treats the model itself as the independent variable. Everything else stays fixed, and the model is swapped in and out. It measures four core capabilities. Approaching a target that is already visible. Tracking a target that moves. Searching for a target outside the initial view. And commanding a fleet of several drones at once. The point is to measure the distance between where models are now and a model that plans persistently and knows exactly when it is done.

Alongside the benchmark comes DroneCATS-Agent, an architecture in which the model is a swappable component. The agent asks for more than flying toward a pixel. It entrusts the model to yaw and search, to deliberate when it is unsure, and to declare for itself that it has arrived. None of this uses fine-tuning, and none of it relies on function-calling schemas. The action protocol lives in the prompt and the model has to keep to it.

The roster of models covers both frontier and open systems. Crucially, it scales down to two billion parameters. The reason given is to find out what breaks first at the edge: when you shrink the model toward something that could run onboard, which ability goes before the others?

The headline result is that even the simplest embodied settings are far from solved. Underneath it sits what the authors call a stark paradox. It is not the flying that fails. Small open models often navigate into the success radius more reliably than frontier models do. They then lose the episode anyway, either by declaring arrival too early or by never declaring it at all.

Commanding several drones amplifies the divide. There, small models fail by blindly copying a single coordinate across views that are actually distinct. Read as vision-language-action agents, the models' spatial perception holds up. Their action protocol does not. What separates a deployable edge model from a frontier model, on this evidence, is not navigation. It is the discipline to sustain a declared protocol and to emit the correct terminating action.

The failure being described is a quiet one. The drone gets where it should be, and the episode is still lost because the model does not say so, or says so too soon. The open problem the authors name is closing that gap at onboard compute costs: a fast model that keeps to its plan and recognises its own finish line. DroneCATS is built to measure how far away that model still is.