Institutional Newspapers Pipeline: Deriving billions of high quality tokens from historical newspapers

Institutional Newspapers Pipeline. Deriving billions of high quality tokens from historical newspapers. By Matteo Cargnelutti and eight others, from the Institutional Data Initiative at Harvard Law School Library, the Boston Public Library, and four other institutions. Published on the nineteenth of August. Its code repository is worked out: about seventy-five source files, with a readme and a licence.

Three things are released together: a pipeline, the small models trained for it, and an open dataset produced by running it. The dataset is the headline. Its optical character recognition output comes to sixteen point three billion tokens, counted with the o200k_base tokenizer. Those tokens sit across eighty-three point one million individual crops, cut from roughly one point four seven million public domain newspaper scans held by Boston Public Library.

Every crop carries more than text. After segmentation and recognition, the pipeline runs text analysis, type classification, reading order detection, named entity recognition, subject classification and language detection on each one, and generates embeddings for it in advance. So what comes out is a structured dataset rather than a pile of page text. The scans span well over a century of printing, the most recent of them from the early twentieth century.

Historical newspapers are an abundant record of public life, but their layouts are dense, irregular and sometimes noisy, which has kept computational access to them hard and limited. The Institutional Newspapers Pipeline was designed jointly with Boston Public Library to deal with that. Its first move is to segment each scan into individual crops that are type agnostic, then run recognition on each segment before any of the analysis steps. Two design commitments shaped it. Each step stays interpretable and customizable, and the whole pipeline stays computationally frugal enough to run on workstation-level hardware. The report describes the method for every processing step, along with the evaluation results and the dataset-scale measurements collected along the way.

One limit on that provenance is stated plainly. The pipeline was run against a portion of the library's holdings, not all of them, so the release reflects that portion.

The obvious audience is anyone who needs large amounts of high-quality text with known origins, and anyone studying the historical record those papers hold. Because the pipeline itself is released and is built to run on modest hardware, it is also for institutions holding scans of their own, since each step can be inspected and adjusted rather than taken whole.

The authors position this as a substantial step toward unlocking high-quality data from tens of millions of newspaper scans. The scale already reached is in the billions of tokens. The ambition in that framing is that the same frugal, inspectable process can keep going, archive by archive.