DRG-MAPPO: Hierarchical Dynamic Role-Graph Multi-Agent Reinforcement Learning for Cooperative Air Combat

DRG-MAPPO. Hierarchical Dynamic Role-Graph Multi-Agent Reinforcement Learning for Cooperative Air Combat. By Junlin Liu and six others, from the Institute of Automation of the Chinese Academy of Sciences and the University of Chinese Academy of Sciences. Published on the tenth of September. No code repository is linked to it.

Multi-agent reinforcement learning, MARL, has become a pivotal approach to complex decision-making in autonomous systems, air combat among them. It has shown significant potential there, but sophisticated tactical coordination remains hard to achieve. The authors trace the difficulty to two limitations.

The first is the absence of structured relational modelling. Without it, agents cannot capture the complex interactions among battlefield entities, interactions that change over time. The second is that conventional flat architectures often cannot model tactical roles explicitly. In highly dynamic environments, that leaves task allocation ambiguous: it is not clear which agent should be doing what. Both limits are about structure. One is missing structure in how agents see each other and their opponents. The other is missing structure in how the team divides its work.

DRG-MAPPO, short for Hierarchical Dynamic Role-Graph Multi-Agent Proximal Policy Optimization, answers both limitations together. It integrates graph-based relational modelling, for the first, with dynamic role assignment, for the second, inside one hierarchical framework. It builds on multi-agent proximal policy optimization, the MAPPO in its name, and adds the two missing kinds of structure on top. The graph handles who relates to whom. The roles handle who does what.

The framework first builds a graph representation of the interactions on the battlefield, and uses graph attention mechanisms to extract the critical relational features among allies, enemies, and threats. A high-level policy then uses a dynamic role assignment mechanism to decide tactical responsibilities, such as leader and supporter. Attention over the graph lets each agent weigh which of those relationships matter most at a given moment, rather than treating every entity on the battlefield alike.

Below it, a low-level policy executes discrete manoeuvre actions, conditioned both on the assigned roles and on the encoded graph features. Splitting the work this way lets tactical strategy and collaborative execution be optimized jointly. On top of this, the authors design an auxiliary task about target priority, meant to encourage behaviours such as focus-fire to emerge. The behaviour is meant to emerge from training rather than being scripted in.

In experiments, DRG-MAPPO reaches a win rate of eighty-seven percent, which the authors report as state of the art. They read this as a sign that the framework balances relational modelling, interpretability and optimization stability for cooperative air combat. That is a claim about three properties at once, not about win rate alone.

The architecture mirrors the two diagnosed limits one for one. A graph gives agents a structured view of who is interacting with whom. Explicit roles, reassigned as the situation changes, give each agent a clear job. And the split between a policy that assigns roles and a policy that flies the manoeuvres is what lets tactics and execution be learned together rather than in isolation.