RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding

RubatoCodec. Budgeted Adaptive Speech Tokenization with Streaming Decoding. By Chihiro Arata and three others, from Third Intelligence. Published on the twenty-first of October. Its code repository is worked out: about fifty source files, with a readme and a licence.

Neural speech codecs turn speech into discrete tokens that a generator can produce. There is a basic trade-off in how many tokens to use. A lower frame rate gives shorter token sequences, but then each token has to carry more of the linguistic and acoustic information. Variable-frame-rate codecs respond by adapting token allocation to the content of the speech.

The catch is in how that allocation is usually done. Threshold-based allocation lets the content decide the token count. So at one fixed setting, the length of the sequence a speech generator must produce varies from utterance to utterance and from dataset to dataset. Adaptivity, done that way, costs a predictable sequence length.

RubatoCodec is a variable-frame-rate speech codec, built on X-Codec-2, that keeps the adaptivity and removes the unpredictability. It has an exact event budget at the level of the utterance, and it supports streaming decoding. Tokens still go where the speech needs them, but the total is fixed.

Its Adaptive Firing mechanism divides speech into events of variable duration. It does this from changes in codec features and in waveform spectra, with no learned boundary predictor. Each event is then represented by discrete codes together with an explicit duration. That pairing is what enables streaming: a speech generator can predict one event and decode it, then move to the next.

Put the pieces side by side. A fixed-rate codec spends the same number of tokens on every stretch of speech. A threshold-based variable-rate codec spends them where the content calls for them, but the total floats. RubatoCodec spends them where the content calls for them and also holds the total to an exact budget for each utterance. The event, a span of variable length with its codes and its duration, is the unit everything else is built on, from reconstruction to generation.

In controlled experiments, adaptive allocation beats fixed-rate coding on reconstruction fidelity and on intelligibility, at low event rates. A model trained on large-scale multilingual speech reaches competitive reconstruction quality across English, Japanese, and Mandarin. On the evaluated English and Japanese benchmarks, at comparable bitrates, it shows the best intelligibility preservation among the codecs compared. Mandarin is included in the reconstruction claim but not in that intelligibility claim.

Then the generation test. In a matched text-to-speech comparison, the adaptive representation gives lower WER, and higher UTMOS and UTMOSv2, with comparable speaker similarity. One more observation stands out. The speech generator learns the codec's utterance-level event budget on its own, without any explicit rate constraint imposed at generation time. Code and weights are public.

The design choice at the centre is to fix the budget while letting the boundaries move. Content still decides where events fall, through Adaptive Firing, but no longer how many there are. The authors conclude that budgeted adaptive speech representations can support both high-quality reconstruction and streaming text-to-speech generation.