EarthRecord: From Pixels to Land Change with Multimodal Agents over Four Decades of Satellite Imagery

EarthRecord. From Pixels to Land Change with Multimodal Agents over Four Decades of Satellite Imagery. By Haoran Zhang and five others, from AIEyes Technologies, the Hong Kong University of Science and Technology, Guangzhou, and three other institutions. Published on the sixth of November.

Multimodal agents can now carry out complex data analysis with tools. Whether they can turn decades of satellite imagery into reliable analysis of land change, work that matters for Earth observation science, has not been measured. Existing benchmarks miss the starting point. They ask questions about curated images, or give agents inputs that have already been preprocessed, and so they leave out the raw multi-sensor archives where this kind of analysis actually begins.

EarthRecord measures the whole path, from raw pixels to an answer about what changed on the ground. It also probes what those answers depend on: knowledge the agent brought with it, the imagery in front of it, or the real change the imagery records. Those are three different things an answer can rest on, and a correct answer alone does not say which one it came from. The benchmark is built to tell them apart, and to do it on the messy inputs that real analysis starts from rather than on images someone has already cleaned up.

The benchmark is built on the raw record of fifty-six sites: four decades of Landsat, MODIS, VIIRS and Sentinel-2 data. Cloud masking and harmonization across sensors are left for the agent to do. Over this record it poses two hundred and thirty-eight land-change questions spanning ten analysis tasks. Answers are scored against withheld references drawn from three sources: global products, agreement among methods, and visual interpretation.

Two controls test what an answer rests on. The closed book takes the record away entirely. Instrument pairs alter one satellite's data while nothing on the ground changes, so an agent reading the real landscape should be unaffected. If answers change under the closed book, the record mattered. If they change under an instrument pair, the agent was tracking the sensor rather than the land.

Twelve multimodal agents were evaluated. The leading ones complete most analyses end to end, but falter when an answer requires mapping from the imagery. The controls are more revealing. Giving agents the record does move their answers away from prior knowledge and onto the pixels. It does not yet move them onto the real change on the ground. When one satellite's data are altered, half of the right answers follow the alteration.

A right answer, in other words, is not yet evidence that the agent saw the land. Half the time it was following one sensor's numbers, and the instrument pair control is what made that visible. The authors offer EarthRecord as a principled platform for measuring and advancing agents along that remaining distance, from pixels to land change.