Halley: A Predictive Control Plane for Optical AI Data Center Networks

Halley. A Predictive Control Plane for Optical AI Data Center Networks. By Manav Mishra and one other, from Prompeteer AI and the Indian Institute of Technology Madras. Published on the third of November.

A frontier AI model is trained in an AI data center, on tens of thousands of accelerators joined by a dedicated network, and every training step ends with all of them exchanging data at once. The network sits on the path of every step. Between racks, that network is optical. Each of its four layers, from lasers up to switches, runs a policy that is either fixed or applied after the fact. And failures are not independent. Lasers that share an enclosure, cooling and power can fail together. At a fixed spare ratio, correlated failure provably caps a pool's availability, however large the pool grows. A bigger pool, in other words, does not buy its way out of failures that arrive together.

Halley is a predictive control plane for that network. The shift is from reacting to anticipating. It reads each resource's next state and acts before the failure occurs, rather than responding once something has already gone wrong. Every layer gets a predictive controller, and that controller defers to the deployed policy whenever it is unsure, so prediction sits on top of what already runs instead of displacing it.

There are four components, each carrying a formal guarantee. LightReserve sizes the laser pool to provide a coverage bound. FoldFit schedules the switch to within eleven ninths of the optimal. ThermaLead holds a microring with one-step-ahead control. TriageGuard protects the highest-risk modules first. Below them sit the fallback circuits, the part that takes over when prediction steps aside. They are formally verified, are shown to achieve close timing on a commercial FPGA, and pass a one hundred and thirty nanometre signoff.

The system-level results are composed from simulations, and they are given under two ways of counting a failure. In the harsher one, a window that exhausts its spares is lost entirely. There, effective goodput, the share of ideal throughput that is kept, rises from sixty-five point two percent to seventy-six point five percent. In the milder one, only the bandwidth of the dark channels is lost, and goodput reaches sixty-seven point four percent.

Expressed as capacity, that is up to one point zero three to one point one seven times the training per megawatt. Put the other way round, for the same training, it is up to three point three to fourteen point seven percent less data center capacity. Both framings describe one improvement: more training from the same power, or the same training from a smaller facility.

Running through all four layers is a kind of caution about its own predictions. Halley acts early where it can, yields to the deployed policy where it cannot be sure, and backs the components with formal guarantees and the fallback with verified circuits.

Halley. A Predictive Control Plane for Optical AI Data Center Networks. By Manav Mishra and one other, from Prompeteer AI and the Indian Institute of Technology Madras. Published on the third of November. This is the long version: the whole paper rather than the abstract.

A frontier model trains on tens of thousands of accelerators, and every training step ends in a collective: all of them exchange data at once, and the step finishes only when its slowest link does. Between racks that network is optical, and the optics are moving onto the package itself, which is called co-packaged optics. A failed laser, a drifted ring or a late switch reconfiguration leaves every accelerator idle at that barrier. Today each optical layer runs a policy fixed in advance or applied after the event. The authors, Manav Mishra and Krishna Sivalingam, observe that no deployed control plane forecasts this hardware, and they find a trap in one layer: lasers that fail together.

The data plane has four layers. A shared pool of external lasers feeds light to the photonic engines. An optical circuit switch joins engines by whole light paths. Microring modulators encode data onto each wavelength, and their resonance moves with temperature. And the lasers arrive in field-replaceable modules. NVIDIA's module holds eight lasers in one temperature-controlled enclosure, sharing heat and often one power supply.

The paper explains the trap with insurance. Ten thousand houses in different towns burn independently, so a small reserve covers the year. Ten thousand houses on one floodplain produce every claim in a single flood. The first theorem makes this exact. A pool where no laser is special behaves as if a hidden severity level were drawn for each restock window, after which each laser fails independently at that level. Without correlation, the variance of the failed fraction shrinks as the pool grows. With it, the variance flattens onto a floor, so at a fixed spare ratio availability converges to a ceiling below one, whatever the pool size. At the paper's stress point, a five percent failure chance per laser per window and a correlation of zero point one five, a pool with one spare per active laser cannot pass ninety-nine point six seven percent.

Halley puts one controller over each layer, under one rule: act on a prediction of the layer's next state, and when the forecast is stale or unsure, step aside to the deployed policy. LightReserve sizes the laser pool. It counts the spares as part of the pool, because spares fail too, and finds the smallest spare count whose coverage meets the target under the correlated model. If that count breaks the budget cap, it reports the target unreachable and returns the correlation that would make it reachable. On a thirty-two channel pool, the usual four-laser reserve covers eighty-six point three percent of windows; ninety-nine percent needs twenty-three spares, where independence needed five.

FoldFit schedules the switch. The classical method splits demands across many configurations, each paying a reconfiguration delay. FoldFit treats each demand as one indivisible job, binary-searches a deadline, and at each candidate places jobs longest first on the first channel that stays within it. That is the MULTIFIT algorithm, so total dwell time stays within eleven ninths of optimal, exactly only when demands may be split at configuration boundaries, and leaving out switching delay. ThermaLead forecasts a ring's next drift from an eight-sample moving average of the temperature slope and pre-positions the heater a step early, so the receiver sees only the forecast's error. TriageGuard gives its few hot spares to the modules with the highest risk scores, which is provably optimal for a given risk estimate. The fallback rule is built in hardware: a watchdog parks the actuator on the reactive path within about twenty microseconds of the predictor going quiet, a property machine-proven on every cycle.

The figure of merit is effective cluster goodput, the share of ideal throughput that survives the fabric's losses. Composed arithmetically from the component simulations, it rises from sixty-five point two percent to seventy-six point five at the stress point. That counts any window whose failures exhaust the spares as entirely lost. If only the dark channels' bandwidth is lost, goodput reaches just sixty-seven point four. The ablation shows the source: availability alone supplies about ten of the eleven points, through a larger spare pool, past the budget cap.

Prediction itself is more modest. On the stress thermal trace, ThermaLead's raw forecast loses lock two percent more often than the deployed dither lock, because the drift barely correlates from step to step. Gated to fall back when its recent error is larger, it does two point one percent better. Fresh telemetry roughly doubles TriageGuard's cut in recovery time. On cost, decorrelating first reaches ninety-nine point nine nine nine percent for forty-four thousand one hundred dollars, against two hundred and thirty-four thousand for spares alone.

There is a limitations section listing six limits. The biggest: the correlation and the per-laser failure rate are assumed, with no field data for these lasers. No number is a hardware measurement. FoldFit's bound for unsplittable demands remains open. ThermaLead's deficit grows sharply under abrupt thermal transients. A confident but wrong forecast has no fallback floor. The goodput model's loss shares and the cost model's prices are assumptions. And the gain assumes fabric stall is the only recoverable loss: scaled by the share of interruptions Llama 3 traced to the network, it would be near one point seven percent.

The idea to carry away: in a shared laser pool, correlation, not count, sets the ceiling. Past a point more spares buy almost nothing, while separating lasers across modules raises the ceiling itself, since the correlation between modules sets the cap. Decorrelate first, size spares against what remains, and plan for a bad window rather than the average. And the measurement that would turn the bound into a forecast is simple: log every laser failure across many pools, and count how many fail together.