Why Speech Recognition Fails on the Alphabet

Everything goes well until you ask for an account number.

The conversational parts of a voice agent are usually the parts that work. Intent is recovered from context, and being approximately right about "I want to change my address" is enough to proceed. Then the agent asks for a reference code, the caller says "B as in… no, just B, D, four, seven, V," and the whole interaction falls apart.

This is the most common place voice agents fail visibly, and it is not really a model quality problem. It is a property of the alphabet meeting a property of the phone network, and the fix is mostly structural rather than acoustic.

The letters that rhyme

English has an unusually bad alphabet for this.

Say B, C, D, E, G, P, T, V and Z out loud. In American English they all rhyme: each is a brief consonant followed by the same long ee vowel. That vowel is most of the sound and nearly all of the energy. The part that differs is a few tens of milliseconds at the very start.

This cluster has a name in the speech recognition literature: the E-set. It is the classic hard case, and it has been the classic hard case since long before neural models.

It is also dialect-dependent in a way worth knowing if you operate in more than one market. American "zee" sits squarely in the E-set. British "zed" does not; it moves to a different, less crowded neighbourhood. The same system can have materially different confusion patterns on either side of the Atlantic without anything changing but the callers.

The E-set is the worst group but not the only one:

| Group | Members | Why | |---|---|---| | E-set | B, C, D, E, G, P, T, V, Z | Identical vowel; differ only in a brief onset | | A-set | A, J, K | Shared vowel; H joins in some accents | | Sibilants | F, S, X | Noise-like, and the noise lives high | | Nasals | M, N | Differ mainly in a subtle formant cue | | Others | I / Y, Q / U | Vowel-heavy, easily confused under noise |

Between them, these groups cover more than half the alphabet.

Why the phone makes it worse

Part one covered the passband. This is where it does its most concrete damage, because the cues that separate these letters fail in two different ways and a phone call attacks both.

For the stops (B, D, G, P, T), the cue is brief. What distinguishes them is the shape of the formant transition in roughly the first 40 milliseconds after the consonant releases, plus voice onset time: the gap between the release and the vocal cords starting. These are short, low-energy events. Two 20 ms packets cover the entire discriminative window, so a single lost packet, or a concealment algorithm smoothing across the boundary, can remove the only evidence there was.

For the fricatives (F, S, TH, V, Z), the cue is high. Their identity lives in the distribution of noise energy, much of it above 4 kHz. A narrowband call does not carry it. Not attenuated: absent.

So the channel removes one discriminative channel outright and degrades the other. What remains is the vowel, which is the part all these letters share.

This is the honest reason a model that scores well in evaluation confuses letters constantly in production. It is not overfitting. The evaluation audio contained information that your calls do not.

It is also why humans invented phonetic alphabets. "V for Victor" exists because people hit this limit on radio and telephone links a century ago and worked around it by replacing a one-syllable token that rhymes with eight others with a multi-syllable word that rhymes with nothing. That is a good idea and it is still a good idea.

Digits are safer, and not safe

English digits are luckier. Zero through nine were not designed for machine recognition, but they came out reasonably distinct: different vowels, different lengths, different shapes.

The remaining problems are mostly not confusion between digits. They are structural.

Segmentation. Where does one number stop and the next begin? A caller saying "fifteen" and a caller saying "one five" are conveying the same thing in ways that must be normalised differently, and "fifty" versus "fifteen" is a genuine confusion that survives good audio, because the difference is stress placement more than phonetics.

Conventions. "Double four", "triple seven", "oh" for zero, "nought" in some markets. Callers reading a number aloud use compression that a naive digit grammar does not expect.

Homophones nearby. "Two" and "to", "eight" and "ate", "for" and "four": usually resolvable by context, occasionally not, particularly when the caller is reading a mixed string and the language model has little context to work with.

Pauses that are not endings. People group digits when reading them: "555… 0142". Those gaps regularly exceed a conversational turn gap, which is exactly the endpointing trap from part two. Cutting a caller off halfway through their account number is a worse failure than any transcription error, because it costs the whole attempt.

The arithmetic nobody runs

Here is the number that should drive the design, and it is rarely calculated.

Character errors compound multiplicatively across a string. A per-character accuracy that sounds excellent produces a field-level success rate that is not:

| Per-character accuracy | 6-character string | 10-character string | |---|---|---| | 95% | 74% | 60% | | 97% | 83% | 74% | | 99% | 94% | 90% | | 99.5% | 97% | 95% |

At 97% per character, which would be a respectable figure to quote, a ten-character identifier comes through correctly about three times in four. One caller in four has to do it again, on a channel where doing it again is slow and irritating.

To get a ten-character field right 95% of the time you need 99.5% per character, on the confusable subset of the alphabet, over a narrowband channel, from callers with varied accents in varied acoustic environments.

That is not a realistic target to model your way to. Which is the point: at these lengths the answer is almost never a better recogniser. It is to change what you are asking for.

Design around it

Roughly in order of how much they help.

Do not ask at all. The best-performing field is the one you deleted. You already have the caller's number from the signalling in most cases; use it to look up the account. Ask for something you can verify against a small set instead: a postcode, a surname, the last four digits. Identify first, confirm second, and never make the caller read out a primary key.

Match against a closed set. Open transcription is a much harder problem than choosing among known valid values. If you hold the list of account numbers, you are not transcribing a string, you are ranking candidates, and "BD47V" against a list of real identifiers will often have exactly one plausible match even when several characters are uncertain. Give the recogniser the format too: three letters then four digits is a constraint worth enforcing during recognition, not after.

Use the checksum. Many real-world identifiers have one. It lets you reject a bad read silently and ask again, instead of confidently confirming a wrong value back to a caller who says yes because it sounded close enough.

Offer the keypad. For anything alphanumeric and high-stakes, this is the single highest-leverage change available. Keypad entry does not go through recognition at all: a keypress travels as a named event, exact by construction. "You can say it or type it" removes the entire failure class for the callers who take it, costs nothing for the ones who do not, and is faster for most people anyway. Worth reading how DTMF actually travels before relying on it, because whether keypresses reach you depends on transport agreement rather than on the caller's phone.

Accept the phonetic alphabet; do not demand it. Callers who volunteer "V as in Victor" should be understood; that is free accuracy they are handing you. Requiring it up front makes an agent feel bureaucratic. The middle path: ask plainly the first time, and if the read fails, prompt for it on the retry. "Could you give me that again using words, like V for Victor?"

Confirm asymmetrically. If a caller says "B", do not read back "B". Read back "B as in Bravo". Confirming in the same ambiguous form you received invites the caller to agree with an error, because they hear what they expect. Expanding on readback surfaces the mistake precisely where it is cheap to fix.

Chunk the confirmation. Reading back a twelve-character string in one breath gives the caller nothing to grab. Character-by-character is exhausting. Groups of three or four match how people hold numbers in memory and how they read them out.

Fail forward, not in a loop. Three attempts at the same field is where callers hang up. After the second failure, change something: switch to keypad, hand to a human, or send a link by text. Repeating the same prompt more loudly is the behaviour that gives voice agents their reputation.

Measure the field, not the model

Overall word error rate will not tell you any of this. It is dominated by conversational speech, which is the part already working.

Track first-attempt success per field. That single number tells you where the interaction is actually breaking, and it is usually concentrated in one or two fields rather than spread evenly. Watch the retry rate on those fields, and the abandonment rate after a failed attempt; that last one is the true cost, and it does not appear in any accuracy metric.

Then split by call path. If accuracy on the same field varies by carrier, by region, or by time of day, that is not the model varying. That is the channel, and it is measurable: a path scoring poorly on loss and jitter is feeding your recogniser damaged audio, and every strategy above will underperform on it no matter how well it is implemented.

Next in this series: the retrieval turn. What happens when the agent has to look something up before it can say anything: the one turn where streaming cannot save you, and where what you have generated, spoken and been heard saying all start to diverge.