Speech event detector: Timestamped detection of non-verbal vocalizations and background sounds for rich speech transcription

Speech event detector. Timestamped detection of non-verbal vocalizations and background sounds for rich speech transcription. By Ayush Kumar Bar and one other. Published on the ninth of October. Its code repository is public and real: twelve source files.

Automatic speech recognition gives you the words and drops everything else in the recording. The breaths, coughs, sighs and laughter of the speaker vanish, and so do the horns, the music or the barking around them. A transcript that wanted to carry that non-verbal content would need something to find those sounds and say when they happened. And it would have to deal with two very different families of sound at once: the ones the speaker makes, and the ones the world makes behind them. The goal here is rich speech transcription, a transcript where those events appear in place, each with its own timestamp.

Speech Event Detector is that something. It is a small convolutional Transformer model, whose parameter count, in millions, is five point eight. Every forty milliseconds it outputs probabilities for fifty-four event classes: twenty kinds of speaker vocalization and thirty-four kinds of background sound. Around the model sit two more pieces. Post-processing converts the probabilities into timestamped events, and a procedure merges those events into a Whisper transcript.

The hard part is the training data. It is drawn from seven public datasets whose labels arrive in incompatible forms: single detected regions, free-text tags with timestamps, temporally strong ontology labels, symbols embedded in transcripts, isolated recordings and environmental recordings. Some of those forms say exactly when a sound occurs, and some say only that it occurs somewhere. The authors unify them with rule-based label mapping, so that every source speaks the same set of classes.

For labels embedded in transcripts, timestamps are recovered through CTC forced alignment, which locates six thousand three hundred and sixty of eight thousand six hundred and twenty sounds. About one hundred and sixty-eight hours of real annotated recordings are then combined with synthetic mixtures, and with augmentation that uses real background noise. Per-class thresholds for two operating profiles are tuned on real held-out recordings, and performance is estimated with two-fold cross-validation.

Testing uses whole held-out recordings from four datasets, restricted to the forty-eight classes that have at least ten held-out events. The sensitive profile detects on average forty-nine percent of a class's events. Counted over all nine thousand three hundred and sixty-seven events, it covers sixty-seven percent, at a macro precision of zero point three zero.

The balanced profile trades the other way, with a macro recall of zero point three seven at a precision of zero point three six. Macro event F1 scores are zero point three two for the sensitive profile and zero point three three for the balanced one. On eight CPU threads, the detector runs about thirty-nine times faster than real time.

The contribution is as much plumbing as model. A compact network does the listening, but what makes it trainable is the work of reconciling seven datasets that never agreed on what a label is, and what makes it usable is the step that turns frame-level probabilities into events placed inside an ordinary transcript. The two profiles then let a user choose between catching more events and raising fewer false ones, and the speed figure means all of it runs on an ordinary processor.