· 2 min read
A safety net that could not fire
My transcription confidence gate flagged nothing at any noise level. The threshold sat below the lowest score the model ever produced.
The confidence gate never fired. Zero flagged segments at every noise level, including the one where a third of the words were wrong.
The obvious reading was that log-probability is a useless signal here. It is not: confidence falls monotonically with noise, from 0.957 to 0.669, tracking the error rate closely. The threshold, 0.55, simply sat below the lowest confidence the model ever produces, so nothing could cross it. An inert safety net is worse than none, because it implies a guarantee that is not there.
The fix that was still wrong
Raising the threshold to 0.85 made the flag fire. It was still wrong, and the reason only showed up in faster-whisper's source: avg_logprob is computed once per 30-second decode window and copied onto every segment from that window. A 130-second clip produces 22 segments and exactly 5 distinct confidence values. The number the gate flags on is a per-window judgement wearing a per-segment label.
Asking the right question
That changed the question from "what threshold?" to "can this score tell me which word is wrong?". Over 73 utterances, roughly 1,160 words per condition, per-word probability reaches AUROC 0.89 on clean audio, against 0.58 for the segment signal the gate used: near chance, exactly where errors are rarest and a flag is most valuable.
The signal that works was already in the output, computed by the decoder and serialized into the JSON, read by nothing.