Sensor-Language-Action Models

Sensor-Language-Action Models. By Yuekai Xu and two others, from the University of California, Los Angeles. Published on the sixth of October. Its code repository is public but slight: nine source files.

Sensors help us understand the world, and they could also help decide what to do next. Existing sensor models, however, mostly stop at perception. They recognise states or predict outcomes. Actions are modelled separately, through label spaces that are specific to a task and often closed, so a model can only ever choose from the list it was given.

Sensor-Language-Action modelling, or SLA, is a framework that connects multimodal sensor observations, natural language, and actions inside a single model. Language serves as the semantic interface between sensing and acting. Because actions are expressed in language, actions of very different kinds can be represented, predicted, and explained, while staying grounded in the sensor evidence underneath.

To support this, the authors build a large-scale SLA benchmark. Its datasets span more than one hundred and sixteen thousand individuals, seventy-nine sensor modalities, and sixty action groups. A captioning pipeline with several facets aligns user context, sensor dynamics, and evidence of action. That alignment is what ties the language to the signals: a description of what a person did is linked to what the sensors recorded and to the context the person was in.

On top of the framework sits OpenSLA, a unified model that handles three jobs. It predicts actions hierarchically, it understands states, and it explains actions. Where existing sensor models stop at perception and leave actions to separate label spaces, here all three share one model and one language interface.

Extensive experiments on real-world tasks in clinical prediction, in operating rooms, and in metabolic health show OpenSLA outperforming the state of the art. It also shows capabilities beyond those tasks: grounding its evidence under guidance from language, and generalising zero-shot to actions and cohorts it has not seen, and to cohorts of people outside the ones it was trained on.

The shift here is from sensors that describe to sensors that inform decisions. Routing actions through language opens the closed label space, which is what lets a model explain a choice and reach actions it was never explicitly trained on.