Measuring Language Transfer in Robot Policies: Adding Greek to a Cosmos3 Vision-Language-Action Policy

Measuring Language Transfer in Robot Policies. Adding Greek to a Cosmos3 Vision-Language-Action Policy. By Ayoub Kirouane and two others, from Sophea AI and Kiefer SA. Published on the seventh of September. No code repository is linked to it.

Robot foundation models are trained and evaluated mostly in English, and for most languages no robot demonstration corpus exists at all. This study adds Greek to an open vision-language-action stack, a Cosmos3 policy, using only machine-rephrased instructions and no changes to the architecture. The central finding is that the hard part is measurement, not translation. Several plausible instruments lead to false conclusions, and only with a guaranteed null and replicated seeds does a real, modest effect show.

Consider the instruments that fail. A colour-histogram metric rewards noise. A single-goal benchmark scores eighty-four point six percent under correct Greek instructions, and eighty-two point six percent under instructions that are deliberately wrong, so it barely distinguishes understanding from none. Training loss fails to predict success in Greek. And comparisons from a single run are dominated by variation between seeds.

On a discriminative suite of ninety tasks, with three seeds per arm, the picture becomes clear. A multilingual text tower with no Greek demonstrations stays at its wrong-instruction floor. Greek-only training beats its control by at most two point seven points. Bilingual training gives a consistent margin of six point seven to seven point one points over its control, and reaches about two fifths of English performance.

The policy also overfits to the translator's phrasing. Training on seven phrasings per task roughly halves that penalty. Two other interventions hurt: warm-starting from a world model adapted to the language, and unfreezing the text tower, both degrade performance.

The setting is one stack, one added language, and instructions produced by machine rephrasing rather than native demonstrations. The gains are measured on the ninety-task suite with three seeds per arm, and bilingual training reaches roughly two fifths of English, not parity. The recommendations are framed as practical requirements for localising robot policies into low- resource languages.

Two requirements come out of it. Build a guaranteed null before trusting a metric, since a benchmark that scores wrong instructions almost as well as right ones cannot tell you anything. And replicate results in a low-resource language across seeds, since a single run can show an effect that is really just noise.