OpenFlyV2: An onboard vision-language-action model for aerial perception and navigation

OpenFlyV2. An onboard vision-language-action model for aerial perception and navigation. By Junli Liu and eleven others, from the Shanghai Artificial Intelligence Laboratory, Northwestern Polytechnical University, and five other institutions. Published on the twenty-ninth of October.

Human cognition draws flexibly on one shared perceptual foundation to understand an environment, attend to the things being referred to, reason about spatial relationships, and act. Giving aerial robots a comparable functional organization is still hard. Existing systems are often specialized for single capabilities, or depend on offboard computation, or cannot sustain closed-loop operation on platforms with limited resources. Each of those is a different way of falling short. A specialist does one thing well but not the others. A system that leans on offboard computation is not self-contained. And one that cannot keep the loop closed on small hardware cannot keep flying on its own.

OpenFlyV2 is a unified onboard vision-language-action architecture. A single shared vision-language representation supports semantic understanding, language-grounded perception, spatial reasoning, and continuous aerial control, all within one model. The same internal picture of the scene is used whether the model is answering a question about it, locating something named in an instruction, judging where things sit relative to one another, or steering the aircraft through it.

Control comes from a compact action expert, which converts the shared multimodal context into short-horizon flight waypoints. Two further components, motion-aware temporal memory and action priors, reduce the computational cost of running the loop closed. For training data, the authors construct a simulation-based dataset of approximately seventy thousand navigation trajectories. Alongside those trajectories it holds sixty thousand vision-language instruction and response pairs.

OpenFlyV2 generates action chunks of twelve waypoints each, entirely onboard, at ten hertz on an NVIDIA Jetson Orin NX. It transfers zero-shot from simulation to a physical quadrotor in flight. Across four real-world environments it achieves a forty-eight percent navigation success rate, without any real-world aerial training data. Every flight it makes in the real world is therefore a test of what it learned in simulation alone. The authors present extensive results as showing that broad vision-language competence and continuous closed-loop flight can coexist in a single model on a resource-constrained aerial platform.

The bet is on sharing. Rather than separate systems for seeing, answering, and flying, one representation serves all of them, and a small action expert reads flight commands off it. The constraint that shapes everything is the onboard computer: the memory and the action priors are there to keep the loop cheap enough to close on the drone itself.

OpenFlyV2. An onboard vision-language-action model for aerial perception and navigation. By Junli Liu and eleven others, from the Shanghai Artificial Intelligence Laboratory, Northwestern Polytechnical University, and five other institutions. Published on the twenty-ninth of October. This is the long version: the whole paper rather than the abstract.

A drone that takes spoken directions has two places to put its brain. It can send camera frames to a ground computer and wait, which makes every decision depend on the radio link and its latency. Or it can think on board, on a small computer the aircraft can carry. The authors point to a quieter cost as well. A drone can hover while it waits for the next prediction, but frequent interruptions cut average flight speed and waste the agility that makes an aerial robot useful. Existing aerial systems also tend to be specialists, or depend on offboard computation. OpenFlyV2 puts scene understanding and flight into one model of about one billion parameters, running entirely on an NVIDIA Jetson Orin NX carried by a DJI M30T quadrotor.

Two ideas come first. A vision-language backbone, here a Qwen three point five VL model of zero point eight billion parameters, reads the camera image and the instruction. A separate action expert of zero point two billion parameters turns that shared context into motion. The same backbone can instead decode a text answer, so a question about the scene and an order to fly pass through one representation. Each navigation output is an action chunk: twelve waypoints at once, each holding three body-frame motion increments and a stop signal, which a low-level flight controller tracks.

The second idea is flow matching. The action expert starts from a random sample and learns a velocity field that moves it toward a plausible action chunk. That velocity describes changes to the candidate chunk, not the aircraft's speed. Each integration step costs time on the onboard computer, so fewer steps means faster commands.

The first component, Motion-Adaptive Memory, handles the past. Rather than feeding the backbone a stack of old frames, it cuts the flight history into events. A new event begins when the action changes by more than a threshold or a segment reaches a maximum length. Each event keeps its action segment and two anchor images, at its start and its end. A temporal convolution encodes the actions into a fixed number of queries, which use cross-attention to pull information from the anchor images. The event's age is added afterwards, so a finished event is encoded once and reused across prediction cycles. Caps on events and on tokens per event keep the memory bounded.

The second component, the Motion-Structured Prior, handles the future. Its premise is that short drone trajectories cluster into a few recurring shapes. Offline, training chunks are normalised by path length and clustered into four motion prototypes. At run time, a prediction head reads the backbone's hidden state at a designated motion token, picks one prototype, and predicts how far to stretch it and how much yaw to accumulate. That adjusted prototype becomes the centre of the starting distribution, with noise shaped by its class's correlations across steps and coordinates. The expert then refines the sample, so the prototype sets where generation begins without confining where it ends. After two supervised stages, reinforcement post-training updated only the action expert.

The training collection spans thirteen environments, including seven Gaussian splatting reconstructions of real campuses and post-disaster ruins, with four held out of training entirely. GPT five generated the instructions and questions, and human annotators checked them. Only sixty percent of samples passed. On perception, the small backbone beats GPT five point five at visual grounding, sixty-seven point one percent against sixty-one point six. In simulated navigation, OpenFlyV2 succeeded on sixty-two point eight eight percent of seen episodes and fifty-two point six seven percent of unseen ones, against thirty-eight point eight eight and thirty-six point six seven for DualVLN. None of the baselines could run on the Jetson, so that comparison is simulation only.

The ablation, run before reinforcement learning, tests the design most directly. Replacing four historical frames with memory tokens cut the input from four hundred and forty-six tokens to at most two hundred and eighteen, cut denoising time fourfold, and raised one-step success from about fifty to fifty-three point four five percent. Adding the motion prior lifted one-step success to fifty-seven point zero nine, above the plain model at two steps. But at two steps, memory alone was strongest, and the authors state that the prior's advantage is concentrated at the single-step budget. That is the budget the drone flies with. On the real aircraft, across one hundred trials at four sites, the success rate was forty-eight percent, with no real-world training data.

The paper has no limitations section. Its discussion concedes that reliable real-world aerial navigation remains a substantial challenge, and names detecting deviations, recovering from errors, and resuming under changing observations as open problems.

The evidence has gaps of its own. The forty-eight percent is pooled over all four sites and both instruction types, with no breakdown, and two of those sites appeared in training as reconstructions. Real-world success was judged by hand. The ablations compare single checkpoints with no error bars. Description quality was scored by another model. And because no baseline fits on the Jetson, the onboard result has no competitor beside it.

The idea to carry away is that on a drone, the compute budget is part of the problem, not a detail of deployment. The authors evaluate each component at the budget the aircraft actually has: the motion prior helps at one step and not at two, and one step is what flies. Compress the past into events, seed the future with a likely shape, and one small model can both answer questions about a scene and fly through it.