SpeechLens

Problem

I wanted to understand how a pressure waveform becomes words, and then how those waveforms differ between languages — the accent patterns a learner would have to hear to sound more like a native speaker.

Getting there meant first solving a less interesting problem. Whisper is easy to call and hard to trust. At library defaults it writes fluent text over silence and music, loops on itself, and reports none of it. A confident wrong transcript is indistinguishable from a right one, and any analysis layer built on top inherits that.

Constraints

• Entirely local: no cloud calls, no API keys, no audio leaving the machine.
• No torch. The ceiling is faster-whisper on CTranslate2 plus NumPy, which keeps the thing installable on an edge device.
• No NVIDIA desktop GPU to develop against, so CPU is the default path rather than a degraded fallback.

Approach

Voice activity detection runs first: Silero gates the audio before the decoder ever sees it, because the most reliable way to stop a model inventing words over noise is not to hand it the noise. Language ID then takes several chunks and votes, fusing the distributions as a log-space geometric mean so two clean windows can veto one corrupted window. Every decode knob — beam width, the temperature fallback ladder, the repetition and no-speech gates — lives in a single config object rather than scattered call sites, so the decode behavior is diffable and testable. Confidence is surfaced rather than swallowed — which turned out to be where the interesting problem was.

Why it's technically hard

The failure mode is silence, not a crash. A hallucinating model returns well-formed output, so nothing throws and no test fails — you only find out by deliberately degrading the input and watching what the pipeline claims.

That forced a split: the pipeline takes its transcriber and detector by injection, so 30 tests cover the orchestration with no weights and no network in about two seconds, while everything needing three gigabytes of parameters lives in a separate validation harness driven from a notebook.

Result

On a free-tier T4, large-v3 at float16 decodes at 12.2× realtime. The number that matters more for a tool premised on running locally: with no GPU at all, large-v3 at int8 still runs at 3.9× realtime on a Ryzen 9 9950X, and on base the CPU actually beats the T4 — a 23-second clip never saturates the GPU, so most of its advantage is idle. Chunk voting identified all five languages in the multilingual check, including Mandarin, and held the correct language across a noise sweep from clean down to −5 dB SNR, where word error reached 33% without collapsing into invented text.

Where it got interesting: the safety net didn't work

The confidence gate never fired. Zero flagged segments at every noise level, including the one where a third of the words are wrong. The obvious reading is that log-probability is a useless signal here. It isn't: confidence falls monotonically with noise, 0.957 to 0.669, tracking the error rate closely. The threshold — 0.55 — simply sat below the lowest confidence the model ever produces, so nothing could cross it. An inert safety net is worse than none, because it implies a guarantee that isn't there.

Raising it to 0.85 made the flag fire. It was still wrong, and the reason only showed up in faster-whisper's source: avg_logprob is computed once per 30-second decode window and copied onto every segment from that window. A 130-second clip produces 22 segments and exactly 5 distinct confidence values. The number the gate flags on is a per-window judgement wearing a per-segment label — inside a window it is constant, so it cannot rank anything.

That reframed the question from "what threshold?" to "can this score tell me which word is wrong?" Answering it needs per-word correctness labels, so I aligned each hypothesis against its reference with a Levenshtein backtrace and measured how well each score ranks errors within a single noise condition — a between-condition trend proves nothing, since a score can track noise perfectly and still be useless for flagging.

Over 73 utterances, roughly 1,160 words per condition: per-word probability reaches AUROC 0.89 on clean audio, against 0.58 for the segment signal the gate uses — near chance, precisely where errors are rarest and a flag is most valuable. The signal that works was already in the output, computed by the decoder and serialized into the JSON, read by nothing.

Two more assumptions didn't survive. The spectral-gating denoise stage made word error worse at every level, by up to 15 points, widening exactly where it was supposed to help. And entropy-based confidence — reported in the literature to detect errors 1.5–4× better than probability, for CTC and transducer models — showed no advantage at all on Whisper: five estimators within 0.008 AUROC of each other. What helped was the aggregation instead of the estimator, taking the minimum over a word's tokens rather than the mean, since a word is wrong if any one of its tokens is.

What shipped

A reliability layer that answers the question a deployment actually asks — at a tolerated error rate, how much of this transcript can I auto-accept? — rather than "what confidence value feels safe?".

Per-word probability is calibrated against measured correctness with isotonic regression, and the accept threshold is derived from a target error rate. On clean audio, 90% of words auto-accept at 2% error; at 10 dB SNR, 78%; at 0 dB, nothing does, and the fitted policy encodes that refusal instead of quietly lowering the bar. Calibration is near-neutral where the raw score was already sound and decisive where it wasn't: at −5 dB it takes expected calibration error from 0.222 to 0.033 and flips the normalized cross-entropy from −0.164 to +0.149 — from worse-than-useless to informative — without touching the model.

The policy is chosen automatically. The voice-activity detector already splits the signal into speech and non-speech, and non-speech is a clean sample of the noise that speech competes with, so an energy ratio gives the SNR for free. It recovers the true value within 0.6 dB and picks the right policy at all six noise levels. When it can't estimate — no speech, or no silence to measure noise from — it applies no policy and says so, because a missing annotation is recoverable and a fabricated one is not.

In practice, on a clip where base misheard "similes drawn from eating" as "similarly he's drawn": both wrong words land in the review list, and similarly scores lowest of all sixty. The old segment gate flagged all four segments at an identical 0.83 — the per-window constant — calling a 95%-correct transcript entirely suspect.

What I'd do next

A second corpus. All of this is read English with additive white noise, so the positive result, the negative one, and the SNR estimator all need different data — spontaneous speech, real recorded noise, other accents — before I'd describe any of them as general rather than as findings about one setup. The per-word signal is also structurally blind to deleted words: a dropped word leaves no token to score. Only after that does the accent analysis that motivated the project get built on a foundation worth trusting.