Privacy Failure in Split-LLM Training, The Returned Gradient Nullifies the Decoys

Privacy Failure in Split-LLM Training, The Returned Gradient Nullifies the Decoys. By Georgios Politis and one other, from Setloop dot i o. Published on the third of September. Its code repository is worked out: about one hundred and seventy source files, with a readme.

This is a systems-security case study, and its claim is uncomfortable in a specific way. A two-node system for training a split LLM passed its privacy evaluation, yet an observable channel had been left untested, and that channel leaks.

The setup runs like this. A Trusted Local Node, the TLN, sends protected activations to an Untrusted Cloud Node, the UCN. The UCN returns its output. The TLN, which holds the private loss, sends back the gradient of that output. To hide the real data, the frame the UCN receives mixes real rows with decoys, and the loss ignores the decoys. That is the flaw. Because the loss ignores them, the decoys' gradients are exactly zero. The pattern of zeros in the returned gradient tells the cloud node precisely which rows were real. The decoys are nullified by the very signal the protocol sends back.

The measurement protocol was fixed in advance, with three safeguards. A leak was injected at a known strength, to prove the instrument can see one. A control with shuffled labels proved it does not report leaks that are absent. And the threshold was set before any runs.

Across nine seeds, the zeros identified the real rows on every frame: four thousand and ninety-six out of four thousand and ninety-six, per run. An attack on the frame contents then recovered about one extra token per hundred over a baseline that simply guesses a constant, an advantage of between zero point six five and one point five percentage points. The shuffled controls recovered nothing.

The scope is where the paper is most careful, in both directions. First, the failure is not confined to some setting nobody would deploy. A second set of runs repeated the test on a configuration that keeps model quality within budget. On both datasets, every one of those runs passed the forward-channel privacy check and passed the quality check, and then failed that same privacy check once the returned gradient was included.

A fix exists for this particular channel. Clipping and adding noise to each row of the gradient closed the leak, at a cost of about zero point zero one nats of held-out cross-entropy.

Second, and just as firmly, closing the leak does not make the system safe. Five classes of attack were never measured, among them attacks that accumulate observations across training steps. The study establishes one failure and one repair. It does not certify anything beyond them.

An evaluation that looks only at what flows forward can pass a system whose backward flow gives the secret away. Here the defence was decoy rows, and the loss, by ignoring them, marked every one with a zero. The gradient that returns to the cloud carried exactly the information the decoys were meant to conceal. A privacy check is only as complete as the list of channels it looks at, and in this case that list was one channel short.