· 1 min read
One full stop cut my word error fivefold
I fed all-caps scoring transcripts to a speech synthesiser as though they were input. The model was fine; my input was the bug.
LibriSpeech and EdAcc ship their transcripts as unpunctuated ALL CAPS. That is a convention for scoring, not for generation, and I passed it straight to a generative model as though it were input.
On LibriSpeech test-clean at n=60, Chatterbox scored 0.112 word error on the raw capitals and 0.022 in sentence case, on identical utterances. The real recordings scored 0.021. Changing the case and adding a full stop cut the error fivefold, and it refuted a claim in my own README that the gap was real.
What made it findable
Every table in the harness carries a ground-truth row: the real recording, put through the same scorer. That row reproduced the published 2.1%, so the scorer had been measuring correctly all along. The anomaly had to be my input.
It looks specific to one kind of model
In a paired study, same utterance and seed with only the case differing, Chatterbox went from 0.109 to 0.025 (sign test p = 1e-7). Piper, which emits in one pass, went from 0.051 to 0.036 (p = 0.36). Runaway generation appeared only in autoregressive decoders, and only on capitals.
One caveat I would rather state than bury: one non-autoregressive model is not a family. The architecture claim is a hypothesis, not a result.