The Functionalizer: Lossless Functional Decomposition for Subword Tokenization
Connor Makowski, Willem Guter
Massachusetts Institute of Technology
Published September 18, 2026 · on the feed September 22, 2026 · ▲ 13 · arXiv 2609.15991
The abstract
Standard subword tokenizers either treat every orthographic variation of a word (such as hello, Hello, HELLO, and Héllo) as unrelated vocabulary entries, which fragments the embedding space, or discard this variation through lossy normalization. We present the Functionalizer, a lossless pre-tokenizer framework that factors orthographic and structural variations into a compositional opcode/operand prefix stream before tokenization: a canonical base token (operand) prefixed by parametric transformation operators (opcodes) encoded in the Unicode Private Use Area. We introduce operators covering casing (CAPITALIZE), diacritics (13 dedicated opcodes), and character repetition (REPEAT, MULTIREPEAT), which are fully reversible. Across natural language and code corpora, the Functionalizer enables complete corpus coverage with significantly smaller vocabularies under unconstrained exhaustion conditions, reducing actual vocabulary slot requirements by up to 19.7%. Downstream evaluations on 98M-parameter GPT-2 models show that the Functionalizer improves Python code syntax validity (9.12% vs. 7.70%) while reducing duplicate n-gram repetition in natural language prose. These findings demonstrate that functional decomposition can be an effective mechanism for vocabulary-efficient, structurally aware language modeling, and motivate further validation at production scale.
The Brief
2:24A short podcast summary of the paper. Written and voiced by AI.
Transcript
The Functionalizer. Lossless Functional Decomposition for Subword Tokenization. By Connor Makowski and one other, from the Massachusetts Institute of Technology. Published on the eighteenth of September. Its code repository is public but slight: four source files.
Take one word written four ways: hello in lower case, capitalised, in all capitals, and with an accent on the e. A standard subword tokenizer has two options, and both cost something. It can treat every orthographic variation as an unrelated vocabulary entry, which fragments the embedding space across near-duplicates. Or it can normalize the variation away, which is lossy: the casing and the accent are gone and cannot be recovered.
The Functionalizer is a pre-tokenizer that refuses that trade. It is lossless. Before tokenization, it factors orthographic and structural variation into a compositional prefix stream built from opcodes and operands. The operand is a canonical base token. The opcodes in front of it are parametric transformation operators, which say how to turn that base into the form that actually appeared. Those operators are encoded in the Unicode Private Use Area.
The operators introduced cover three kinds of variation. Casing is handled by CAPITALIZE. Diacritics get thirteen dedicated opcodes. Character repetition is handled by REPEAT and MULTIREPEAT. All of them are fully reversible, so the original text can always be rebuilt. The word keeps one identity in the vocabulary, and its surface form travels alongside it as a few instructions.
On natural language and code corpora, the Functionalizer reaches complete corpus coverage with significantly smaller vocabularies, under unconstrained exhaustion conditions. The actual number of vocabulary slots required drops by up to nineteen point seven percent.
Downstream, the tests use GPT-2 models of ninety-eight million parameters. With the Functionalizer, Python code syntax validity improves to nine point one two percent, against seven point seven zero percent without it. In natural language prose, duplicate n-gram repetition goes down.
The authors present these as evidence that functional decomposition can be an effective mechanism for language modeling that is both vocabulary-efficient and structurally aware. They are also explicit about the scale: the models here have ninety-eight million parameters, and the results are framed as motivation for further validation at production scale rather than as a settled verdict.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
One word, four spellings, four vocabulary entries. Start there.
You are building a tokenizer. The same word turns up as hello, Hello, HELLO and Héllo, and each spelling takes its own vocabulary entry with its own embedding.
The road ahead
- 1 · The problem ~6 min A tokenizer must either give every spelling of a word its own entry, wasting space, or flatten spellings and lose information for good.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Keep one base token per word and write its casing, accents and repetition as reversible operators placed in front of it.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Reduce each piece to a base token, prefix operators for its variation, encode them as private-use characters, then run subword tokenization.
- 6 · The experiment ~6 min You’ll guess: Under unconstrained exhaustion, by up to how much did the Functionalizer cut the vocabulary slots needed to fully cover a corpus?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.