Intrinsic Goals for Autonomous Agents: Hierarchical Exploration Captures Future Volitional Action in Drosophila

Intrinsic Goals for Autonomous Agents. Hierarchical Exploration Captures Future Volitional Action in Drosophila. By Declan Kutscher and four others, from the University of Maryland, College Park and Carnegie Mellon University. Published on the twenty-fifth of October. Its code repository is worked out: about fifty-five source files, with a readme and a licence.

Animals act autonomously. Driven by intrinsic motivation, they behave without external sensory cues and without objectives set from outside, and the computational mechanisms behind that remain largely unknown. This work asks whether optimising intrinsic goals produces embodied artificial agents that quantitatively match a real animal, both in behaviour and in the internal representations of action it is about to take.

The answer it reports is yes, for one algorithm in particular. DIAYN, a hierarchical competency method, gives the strongest joint match to fruit flies in behaviour and in neural prospective coding: the ability to predict a future action from activity before it happens.

The comparison rests on two pieces. One is whole-brain recordings in the fruit fly, Drosophila melanogaster, which reveal neural representations of spontaneous exploratory decisions more than ten seconds before they happen. The other is a neuromechanical digital twin of the fly. The agent's hierarchical policy selects from a discrete library of low-level motor primitives, in keeping with ethological theories of behavioural syllables.

Agents are trained without external reward, using twenty intrinsic motivation algorithms that span curiosity, entropy, and competency methods. They are then tested in a virtual tethered-ball assay modelled on the fly experiment. The comparison covers spontaneous locomotor statistics, internal representations, and decoding of future turn direction from kinematics and neural activity, on the same timescales seen in flies.

DIAYN learns discrete policies that maximise behavioural entropy and the separability of the sensory observations they induce. Beyond the joint score, its learned policies closely capture individual variability in how flies explore. Its latent activity also closely captures the sum-difference rule by which biological flies encode kinematics across the two hemispheres.

The match is to one animal, in one assay: a fly on a tethered ball, exploring spontaneously, and an agent in a virtual version of that same setup. The algorithms compared are the twenty intrinsic motivation methods trained here, and DIAYN is strongest relative to flies among them, not against every possible model. The explanation offered for individual variability is framed as a suggestion. The diversity of strategies may serve to maximise coverage of the state space across the population as a whole. It is offered as a reading of the results, not stated as an established mechanism.

The approach treats autonomy as something that can be reverse-engineered: build an embodied agent, give it an objective, and ask whether both its behaviour and its internal activity resemble the animal's. Here the objective comes from within the agent, and the resemblance extends to signals that anticipate a turn well before it is made. The authors present hierarchical, intrinsically motivated agents as a framework for reverse-engineering animal autonomy.