RayOrch: Programming and Executing Lineage-Controlled Multi-Grain Dataflows for Foundation-Model Data Preparation

RayOrch. Programming and Executing Lineage-Controlled Multi-Grain Dataflows for Foundation-Model Data Preparation. By Xiaochen Ma and fifteen others, from Peking University, the Hong Kong University of Science and Technology, and three other institutions. Published on the sixteenth of September. Its code repository is worked out: about one hundred and twenty source files, with a readme and a licence.

Training data for foundation models is prepared by pipelines that turn heterogeneous documents and videos into structured records. These pipelines have a characteristic shape. Each parent item expands into an ordered sequence of children, and how many children depends on the input, with counts that can be long tailed. For the GPUs to be used well, children from many different parents should be batched together. Yet the system still has to preserve which parent each child belongs to, the order of the children, whether each one has finished, and where each result should be routed.

Existing systems take one of two unsatisfying positions. Some hide the parallelism behind coarse grained jobs, so the pieces inside a document or video never run side by side. Others expose flat records, which forces the application itself to manage lineage and to regroup results afterwards.

RayOrch is a programming model together with a distributed execution engine, and its organising commitment is that parent child relations are preserved throughout execution. The structure of the data is declared once and then honoured by the system everywhere, rather than rebuilt by the application.

A program declares ordered expansions of variable cardinality, and gathers that match them. The compiler validates each expansion and gather pair. At runtime, RayOrch records child membership, each child's immediate parent, an immutable ordinal, and terminal states. Per Call FIFO Ready Queues batch ready children across parents. When results come back, gathers reconstruct them from the declared membership and the ordinals, not from batch boundaries and not from the order in which things happened to finish.

Two consequences follow. A parent can advance as soon as all of its required children have become terminal, without waiting for anyone else. And failure is scoped. Typed failures belong to a parent, so when one parent fails, its undispatched siblings are suppressed while unrelated parents carry on.

The measurements are on NVIDIA H20 GPUs. Scaling MinerU from four to sixty-four GPUs, RayOrch achieves a fifteen point one four times speedup. Scaling a video pipeline from eight to sixty-four GPUs, it achieves seven point eight two times. Against other engines, it cuts end to end time on MinerU by thirteen point one percent versus Ray Data, and by twenty-nine percent versus Daft. On Docling it cuts end to end time by sixteen percent versus Ray Data.

The design choice underneath all of this is to make lineage something the engine knows rather than something the application reconstructs. Batching across parents is what keeps the GPUs busy, and the recorded membership and ordinals are what let those results find their way home in the right order.