Transcription and ASR

Two Words That Are Not Synonyms

Transcription is the task and its artifact: speech written down. ASR is one way of performing it — a specific technology with specific failure modes. Keeping them apart lets you ask the right question. "Is the transcript good enough?" is about the artifact and your use for it. "What is the WER?" is about a machine on one particular recording, and the two answers routinely disagree.

There is a second distinction that catches editors specifically. Verbatim transcription records every sound as uttered — the stutters, the false starts, the "um". Clean-read (or intelligent verbatim) removes them for readability. For reading, clean-read is better. For editing, it is unusable, because you cannot cut a word the transcript does not contain, and a transcript that skipped a 210 ms filler has a 210 ms hole in its own timeline.

How ASR Actually Works

The front end: audio becomes a picture of itself

No speech model consumes a waveform directly. Audio is resampled to 16 kHz mono — telephony systems use 8 kHz, which is one reason telephone speech is measurably harder — and converted to a log-mel spectrogram: energy per frequency band per short slice of time, on a mel scale that spaces the bands the way human hearing does. Whisper computes this with a 25 ms analysis window stepped every 10 ms, giving 100 frames per second, over 80 mel bands (128 in large-v3). A stride-2 convolution in the encoder then halves the rate to 50 positions per second, so one encoder position covers 20 ms of audio. That 20 ms is the resolution floor of everything downstream, including word-level timestamps.

Three architectures, and why all three still exist

The hard part is that audio frames and output tokens do not line up. A 4-letter word can occupy 60 frames; a pause occupies frames and produces nothing. Three families solve that alignment problem differently, and the differences are not academic — they determine whether a system can stream, and how it fails.

• CTC predicts one symbol per frame from an alphabet extended with a blank, then collapses repeats and blanks. It is fast and monotonic, but it assumes each frame's prediction is independent of the others given the audio, so it has no built-in sense of what word is likely to come next — accuracy usually depends on fusing an external language model at decode time.
• RNN-transducer (RNN-T) adds a prediction network conditioned on the tokens emitted so far, joined to the acoustic encoder. It keeps CTC's frame-by-frame, left-to-right progress — which is what makes it the standard choice for streaming recognition on a phone — while removing the independence assumption.
• Attention encoder-decoder (AED), the Whisper family. The encoder reads a fixed window of audio; the decoder generates text autoregressively, attending anywhere in that window. It gives the best accuracy on offline audio and, because it is a text generator conditioned on audio, it is the only one of the three that can hallucinate — produce fluent text with no acoustic support at all.

The language model is half the system, and it has opinions

Every ASR system is doing two things: judging what the audio sounds like, and judging what a plausible sentence looks like. In classical pipelines those were separate acoustic and language models combined at decode time; in an end-to-end model they are entangled in one network, but both jobs are still being done. This is why ASR fixes obvious things for free — it will resolve "wreck a nice beach" into "recognize speech" when the context supports it. It is also the source of the most dangerous class of error: when the audio is ambiguous, the model resolves toward what is statistically likely, and returns it with high confidence and no indication that it guessed. Proper nouns, product names and technical jargon are exactly the words a general language model finds unlikely, which is why they are exactly the words that come back wrong.

Windows, chunking and long files

Whisper's encoder takes a fixed 30-second input — 3,000 mel frames in, 1,500 encoder positions out. Anything longer is chunked, and the chunk boundaries are a real source of error: a word split across a boundary can be transcribed twice or not at all, and each chunk restarts the decoder's context. Scale is what makes it work at all: Whisper was trained on 680,000 hours of multilingual audio scraped from the web, and large-v3 on roughly 1 million hours of weakly labeled plus 4 million hours of pseudo-labeled audio, which OpenAI reports as a 10–20% error reduction over large-v2across many languages.

Word Error Rate: What the Number Means

Align the output to the reference with an edit-distance alignment, count three error types, and divide:

WER = (S + D + I) / N

  S  substitutions — a wrong word          ("their" for "there")
  D  deletions     — a word that vanished  (a dropped "not")
  I  insertions    — a word never spoken   (a hallucinated clause)
  N  words in the reference transcript

reference:  we shipped it on friday
hypothesis: we shipped on a friday
            -------  D  --  I  ------      →  (0 + 1 + 1) / 5 = 40%

• WER can exceed 100%. Insertions are not bounded by N. A model that hallucinates a sentence over two seconds of silence can score several hundred percent, which is the correct behaviour of the metric and a good reason not to treat it as a percentage of anything.
• Normalization moves it by whole points. Casing, punctuation, contractions, digits versus spelled-out numbers, hyphenation. Whisper ships its own text normalizer precisely because unnormalized comparisons are meaningless; two published WERs are comparable only if they normalized identically.
• WER weights every word equally. Losing "the" and losing "not" cost the same. For anything that turns on meaning, WER understates the damage of the errors you care about.
• Some languages need CER instead. Chinese and Japanese are written without spaces between words, so the unit WER counts does not exist; character error rate is used.

The reference points worth memorising come from the human-parity work on the NIST 2000 conversational telephone set. Microsoft's 2016 system scored 5.8% WER on the Switchboard portion against 5.9% for professional human transcribers, and 11.0% on CallHome against 11.3% for humans. Same system, same week, same language — and roughly double the error rate, because CallHome is open-ended conversation between friends rather than strangers on an assigned topic. That single comparison is the whole lesson: WER measures a system and a recording together. A vendor number quoted without the corpus and the conditions is not a measurement.

Diarization: "Who Spoke When"

Transcription says what was said. Diarization partitions the audio into speaker-homogeneous regions and labels them relatively — Speaker 1, Speaker 2 — without knowing who anyone is. Attaching a name is speaker recognition, a different task with different requirements. The two are usually run together and then reconciled, which is its own source of error: a word whose timing straddles a speaker-change boundary has to be assigned to one of them.

Two designs dominate. The cascade runs voice-activity detection, cuts the speech into short segments, embeds each as a speaker vector, and clusters the vectors — robust, interpretable, and structurally blind to overlap, because one segment gets one label. End-to-end neural diarization instead predicts per-speaker activity for every frame, trained with a permutation-invariant loss so the arbitrary ordering of speaker labels is not punished. Overlap is native to that formulation, which is why it exists.

The metric is diarization error rate: DER = (false alarm + missed detection + confusion) / total reference speech. False alarm is non-speech called speech, missed detection is speech called non-speech, confusion is speech given to the wrong speaker. Two evaluation conventions decide what the number means: a forgiveness collar, conventionally 250 ms either side of each reference boundary (500 ms total), excluded because human annotators cannot mark a boundary to the sample; and whether overlapped regions are scored at all. Skip the overlap and you have deleted the hardest part of the problem from your own exam.

Published DERs for a single well-regarded open pipeline, across corpora — the spread is the point:

Read the first two rows together. Those are the same AMI meetings — same people, same words, same room. The only variable is whether each speaker wore a headset microphone or the room was captured by one distant mic, and DER moves from 12.9% to 15.6%. On a first-person Ego4D recording the same pipeline sits near 39%. Nothing about the model changed. How the audio was captured constrains the ceiling more than which model you pick, and no amount of post-processing recovers a voice the microphone did not resolve.

Why Accents and Overlap Are Hard

Accents: an acoustic gap, not a vocabulary gap

A 2020 PNAS study measured five commercial ASR systems — Amazon, Apple, Google, IBM and Microsoft — and found an average WER of 0.35 for Black speakers against 0.19 for white speakers; Apple's system scored 0.45 versus 0.23, Microsoft's 0.27 versus 0.15. The interesting part is the diagnosis. Lexical coverage was roughly equivalent between the two groups at around 98–99%, and the language model component actually performed better on the Black speakers' transcripts — so it was not a vocabulary problem. The gap persisted on identical phrases spoken by both groups, which localizes it to pronunciation and prosody, and behind that to which voices were in the acoustic training data.

The language model then compounds it. When the acoustics are ambiguous, the decoder resolves toward the most probable word sequence — which is the majority-dialect one — and returns it fluently, with no signal that a guess was made. An unusual vowel does not produce a low-confidence output; it produces a confident wrong word.

Overlap: one channel, one token stream

When two people talk at once, a standard ASR model does not report a conflict. It emits a single sequence, transcribes whichever voice is more prominent, and drops the other — so the transcript reads as complete while an entire turn is missing. Humans solve this with two ears and a lifetime of practice at the cocktail-party problem; a single-channel model has neither. Recovering both voices requires separate microphones, source separation, or a diarization architecture that models concurrent speakers.

The practical consequence for anyone recording: overlap is a capture decision, and it is made before you press record. Two people on two microphones, recorded to two tracks, is a solved problem. The same two people on one room mic is a research problem you have handed to a tool that will not tell you it lost.

What a Transcript Unlocks in Editing

A transcript with only sentence-level times is a document you read. A transcript with per-word times is an index you address the media through, and that difference changes what editing is. Here is what it looks like on a real file:

{
  "language": "en",
  "duration": 3187.4,
  "segments": [
    { "start": 742.11, "end": 746.83,
      "text": " So the pricing question — um, we moved to usage-based in March." }
  ],
  "words": [
    { "word": "pricing", "start": 742.55, "end": 743.02, "probability": 0.991 },
    { "word": "um",      "start": 744.18, "end": 744.39, "probability": 0.884 },
    { "word": "we",      "start": 744.71, "end": 744.83, "probability": 0.996 }
  ]
}

You have a 53-minute two-person recording and you need the answer about pricing. Searching the transcript for "pricing" returns 742.55 s — not approximately, exactly, and you can extend backwards to the breath before it and cut in the silence. That is one operation instead of ten minutes of scrubbing. Then you ask for the fillers to be removed: a single "um" is about 210 ms, there might be two hundred of them across the recording, and every one is a separate splice that has to land on a word boundary. Without word timing that job is not tedious, it is impossible — there is nothing to address the words by.

The same index then pays for itself three more times. Text-based editing becomes possible at all, because deleting a sentence resolves to deleting a source range. Captions come free and correct, because the words and their times are already computed. And clip selection stops being a skim: you read 8,000 words in five minutes and find the three moments worth cutting, which you cannot do watching at 2x. Note the honest detail in the sample too — "um" scores 0.884 against 0.99+ for real words, and it is in the transcript only because the system was asked to transcribe verbatim.

Common Misconceptions

How This Works in Valmera

Valmera is an agentic video editor, and the transcript is the substrate its agent works on rather than a feature bolted onto it. Every upload is indexed once — a word-accurate transcript with word-level timestamps, plus silence detection, shot detection and labeled frame tiles — and every later edit reads that index instead of re-analyzing the file.

• The transcript is addressable, not just readable. Transcript search, a word list and a kept-only transcript (what the current edit still contains, as opposed to what was recorded) are tools the agent calls, and the same tools are published on the MCP server so Claude can drive the edit with them.
• Every cut snaps to a word boundary. Range cuts, silence removal, filler-word removal (um, uh, er, hmm, plus custom words) and repeated-take detection all resolve to word-timed ranges rather than to a scrubbed guess.
• Captions come from the same timings — word-accurate, 1–16 words per caption, karaoke word-pop up to 6 words per line, per-word emphasis. Editing the transcript in the studio re-renders them.
• Language is auto-detected from the audio. English is the best-tested path.

The honest limits, since this page is about what ASR does and does not deliver. Valmera has no speaker diarization output and no per-speaker leveling — it will not hand you a speaker-labelled transcript or balance two voices independently, so a two-mic interview is better levelled before upload. There is no SRT or VTT import or export: captions are burned into the video, so the transcript drives the edit and is readable through the agent's tools, but it does not leave the project as a subtitle file. There is no denoise or studio-sound processing and no separating music from speech in an already-baked track, which means a noisy recording stays a noisy recording — and, as this page has argued, that limit is the ceiling on the transcription too. Everything above about capture quality applies to Valmera exactly as it applies to anything else.

Related Terms

Other entries in the Valmera video editing glossary that touch this one:

• Word-level timestamps — The per-word start and end times that turn a transcript into an index.
• Text-based video editing — Deleting a sentence deletes a source range — the workflow ASR exists to enable here.
• Silence detection — The other half of a clean cut. Words say what survives; silence says where to join.
• Burned-in captions vs subtitles — What happens to the transcript after it becomes pixels rather than a file.
• Edit decision list — Where the resolved ranges are written, in source seconds.
• Jump cut — What a transcript-driven splice looks like on the picture side.
• J-cut and L-cut — Deliberately offsetting the sound edit from the picture edit.
• Shot detection — The picture-side index. ASR indexes the sound.

Frequently Asked Questions

Edit From the Transcript

Upload real footage and it is transcribed to the word before you type anything — then search it, cut it and caption it by describing what you want. 50 free credits, no card.