OmniSpectra: A Unified Foundation Model for Universal Spectra Representation Learning

OmniSpectra. A Unified Foundation Model for Universal Spectra Representation Learning. By Md Khairul Islam and three others, from the University of Virginia. Published on the twenty-third of October.

Standard sequence models know where a token sits by its rank in the line, not by any physical coordinate. For text that is harmless. For a continuous physical signal it is not. Spectra are observed across many different instruments, each sampling on its own grid, and to feed them into a sequence model practitioners are usually forced to resample them, which destroys information. The authors name this coordinate mismatch as a structural limitation for learning representations that work across instruments.

Their answer is OmniSpectra, a foundation model that is wavelength-native. It is a masked autoencoder that works in continuous coordinates and drops the integer pixel indexing that sequence modeling normally relies on. Position means wavelength, not slot number.

One move makes the physics line up with the architecture. Physical coordinates are mapped to a logarithmic basis. Cosmological redshift stretches a spectrum multiplicatively, and on a log scale that stretch becomes a uniform additive shift. Inside continuous rotary position embeddings, a shift shared by every token cancels out of the relative distances between tokens, instead of producing positional distortion that depends on the grid.

A second piece handles the output side. Implicit neural decoders carry a spectral bias, and to counter it the authors use a Fourier decoder that is physics-grounded and resolution-gated, providing anti-aliasing.

The model is pretrained on eight surveys and benchmarked against seven baselines. It reaches state-of-the-art performance with only twelve point two million parameters, and it outperforms models that are substantially larger.

That size gap carries the paper's argument. A small model that builds the structure of the measurement into its coordinates beats bigger ones that do not, and the authors read this as a sign that encoding physical structure can stand in for parameter scale.

OmniSpectra. A Unified Foundation Model for Universal Spectra Representation Learning. By Md Khairul Islam and three others, from the University of Virginia. Published on the twenty-third of October. This is the long version: the whole paper rather than the abstract.

Astronomy has more spectra than it can label. The Multimodal Universe alone curated over a hundred terabytes of machine-learning-ready data, most of it unlabeled, and spectroscopy is the richest modality in it: redshift, stellar composition, galactic kinematics and chemical abundances are all written into the light. Yet most applications still rely on small models built for one task or one instrument. The recent foundation models for spectra are pretrained on a single survey or at a fixed resolution. SpectraFM reads spectra in fixed windows of five hundred and twelve tokens and averages its predictions across chunks. Specformer, the spectrum encoder inside AstroCLIP, takes a fixed sequence length because it only ever saw DESI spectra. Pointing either model at a new instrument means substantial retraining or a new encoder.

The obstacle is that surveys disagree about almost everything. They differ in resolution, in wavelength coverage and in instrument design. The eight surveys used here run from VIPERS, whose longest spectrum is five hundred and fifty-seven points, to DESI, at seven thousand seven hundred and eighty-one. APOGEE sits in the near infrared, at wavelengths no optical survey in the set reaches. The paper's answer is what it calls native resolution: a model that takes a spectrum of any length, from any instrument, at its original size, without truncating it, resampling it or projecting it onto a predefined grid. OmniSpectra is a Transformer pretrained with a masked modeling objective, and the claim is that it is the first such model to learn from several real surveys with different configurations at once.

Each spectrum is a list of flux values paired with wavelengths. After normalization it is cut into partially overlapping patches: twenty flux points each, overlapping by ten. Every patch token also carries the mean and standard deviation of its flux. For pretraining, the model hides contiguous chunks rather than scattered patches. Each chunk is two and a half patch widths wide, three quarters of the sequence is targeted, and no two masked chunks may touch. Because each chunk is a continuous stretch of wavelength, the model has to infer a missing region from its surroundings.

Three components make variable length work. The first replaces position with physics. Instead of telling the Transformer a token's index, a sinusoidal encoding is computed from the patch's mean wavelength, so a token at a given wavelength carries the same position code whichever instrument produced it. The second is validity-aware attention: any pair of tokens involving a padded patch gets an attention score of minus infinity, so padding is never attended to. The third is a local positional embedding. A depthwise convolution with kernel size three runs over the value tensor and is added to the attention output, giving each token an explicit sense of its immediate neighbours, which is where line shapes live. Training minimizes squared reconstruction error only at masked, valid positions. The model has six layers, six heads and a seven hundred and sixty- eight dimensional embedding, about forty-two point six million parameters, trained on all eight surveys together for forty-eight hours on four A100 GPUs.

Downstream, the encoder is wrapped with a linear head on mean-pooled tokens. The labels matter here. What the paper calls zero-shot is mostly a hundred steps of tuning, and few-shot is five hundred; galaxy properties instead use a nearest-neighbour regressor for zero-shot. The supervised baselines, GalSpecNet and Spender, are trained from scratch for ten epochs on each dataset. On stellar properties across SDSS and its subsets, few-shot OmniSpectra has the lowest error everywhere, though GalSpecNet ties it on SEGUE1. SpectraFM, pretrained on synthetic stellar spectra, does worst throughout. Source classification is the most striking: on eBOSS, the zero-shot star score is nought point nine five eight against AstroCLIP's nought point five seven three. Galaxy properties are close to a tie with AstroCLIP. The test that most directly checks the thesis is the ablation. Removing the wavelength encoding raises reconstruction error from nought point five zero five to nought point nine zero three, while dropping the local embedding or the validity mask costs only a few hundredths.

The paper has no limitations section, so the limits come from what it reports. Downstream results are averages over five runs, but no spread is given, and some margins are a few thousandths. The tuning budgets, a hundred steps for zero-shot and five hundred steps for few-shot, are, in the authors' own words, arbitrarily chosen. Zero-shot here does not mean unseen data: the SDSS surveys used for evaluation are also in the pretraining corpus, and most zero-shot results involve some training. The model does not win everywhere either. On BOSS classification its few-shot scores fall below its zero-shot scores, and in several columns the supervised models are within a hair. Two pretraining surveys, APOGEE and VIPERS, have no downstream task of their own.

The idea to carry away is that, for spectra, position should be physical. A token's index means nothing across instruments with different sampling, but its wavelength means the same thing everywhere. Encode that, keep spectra at their own length, and mask out the padding, and one model can train on eight surveys at once. The paper's own ablation says the first of those choices did most of the work.