← Home
GLOSSARY

Published · Updated

Word-Level Timestamps

Word-level timestamps are the start and end times of each individual spoken word in a recording, measured in that recording's own clock and usually accompanied by a per-word confidence score. They are produced either by an automatic speech recognition (ASR) model that emits timings together with the text, or by forced alignment, which takes an already-known transcript and finds where each word sits in the audio; typical resolution is tens of milliseconds. Their significance is structural. Once a transcript is timed word by word rather than sentence by sentence, the text becomes an index into the media: a phrase search returns a range you can cut, a caption can highlight one word at a time, and every edit point can be snapped to a boundary between words rather than landing inside one.

How a Word Gets a Time

There are two families of method, and they solve different problems. ASR-native timestamps come out of the recognition model itself, which produces the words and their timings in one pass. Forced alignment starts from a transcript that is already known and only has to find where each word sits in the audio. Alignment is the easier problem — the answer is constrained to spelling out a text you have — and it is generally the more accurate of the two, at the cost of needing that text first.

The grid underneath

Speech models do not see waveforms. Whisper's front end takes 16 kHz mono audio, computes a log-mel spectrogram with a 25 ms analysis window and a 10 ms hop — 100 frames per second — and a convolution of stride 2 halves that to 50 encoder positions per second. One position is therefore 20 ms, and its input is a fixed 30-second window: 3,000 mel frames in, 1,500 positions out. Whisper's vocabulary carries a matching set of 1,500 timestamp tokens, <|0.00|> through <|29.98|>, in 0.02-second steps. That 20 ms grid is the floor. Any timestamp printed finer than it is interpolation, not measurement.

ASR-native: dynamic time warping over cross-attention

Whisper natively emits segment-level timestamps — a phrase or sentence at a time. Word timing is a separate pass over the decoder's cross-attention, the weights that record which part of the audio the decoder was looking at as it emitted each token. Three steps: take the attention matrices from a chosen subset of heads (the alignment heads — specific [layer, head] pairs found empirically to track time, rather than all of them); median-filter them to denoise; then run dynamic time warping to find the monotonic path of highest cumulative attention through the token-by-frame matrix. Where that path steps from one token to the next becomes a boundary. Tokens are then merged back into words, with punctuation attached to its neighbour.

The word "monotonic" is doing real work there. Attention is fuzzy and can look in several places at once; DTW forces it into a single ordering in which time only moves forward, which is what converts a soft-focus heat map into hard boundaries.

Forced alignment: CTC over a known transcript

The alternative runs a phoneme or character recognizer — a wav2vec2 model is the usual choice — over the audio to get per-frame posterior probabilities, then uses CTC alignment to find the highest-scoring monotonic path through those frames that spells the transcript you already have. CTC emits the same symbol on every frame the sound occupies, so consecutive identical emissions are collapsed to give each character a span, and characters are merged up into words. This is what WhisperX does: transcribe with Whisper, then discard Whisper's own timings and re-derive them by alignment.

How Accurate They Actually Are

A word boundary is not right or wrong; it is off by some number of milliseconds, and the honest way to report it is a mean error plus a hit rate against a stated tolerance. Real published figures for the Montreal Forced Aligner, which is the reference point most systems are measured against:

  • Mean word-boundary error 21.9 ms on TIMIT and 27.8 ms on Buckeye.
  • F1 of roughly 65% at a 20 ms tolerance on both corpora — that is, only two thirds of boundaries land within a fiftieth of a second.
  • Loosen the tolerance to 50 ms and word-boundary accuracy on Buckeye rises to about 88%. MFA 3.0 reports mean boundary errors below 15 ms.
  • Papers evaluating ASR-native timestamps typically score with a 200 ms collar, which is itself a statement about how much slop is considered normal in that setting.

Scale matters here. The average English word runs about 440 ms, and conversational speech sits around 120–150 words per minute, so a 25 ms error is roughly 6% of a word. A video frame is 33.3 ms at 30 fps and 41.7 ms at 24 fps — so a good aligner's average error is smaller than a single frame, and the 20 ms grid is finer than a frame at 24, 25 or 30 fps though coarser than one at 60. That is why word-level timing is good enough to edit with.

The failure that costs you is not the average, it is the systematic one. DTW-based timestamps absorb pauses into the adjacent word. The cause is a tokenizer detail: Whisper glues the leading space onto the following token, so "This is a long pause." becomes ['This', ' is', ' a', ' long', ' pause'] and there is no token that represents the silence. DTW has nowhere to put it, so it lands inside a word's span. A reported word "end" can therefore include several hundred milliseconds of following silence. The fix demonstrated by CrisperWhisper is to retokenize so a space is its own token and the pause becomes an event DTW can attribute to itself. If your cuts feel late by a beat, this is usually why.

What the Data Looks Like

There is no standard, but every system converges on the same shape: a word string, a start, an end, and a confidence.

"words": [
  { "word": "we",      "start": 12.437, "end": 12.556, "probability": 0.994 },
  { "word": "shipped", "start": 12.556, "end": 12.921, "probability": 0.981 },
  { "word": "it",      "start": 12.921, "end": 13.024, "probability": 0.997 },
  { "word": "on",      "start": 13.024, "end": 13.160, "probability": 0.992 },
  { "word": "Friday",  "start": 13.512, "end": 14.005, "probability": 0.989 }
]

Two things in that snippet are worth reading carefully. First, inside the phrase, each word's end is exactly the next word's start — not because the aligner is confident, but because there is no gap there to measure. Continuous speech has no silence between "shipped" and "it". Second, there is a 352 ms gap before "Friday", and it belongs to neither word: it is a breath. That gap is the most useful number on the line, because it is where a cut can go.

Also note the false precision. Three decimal places implies microsecond resolution from a system whose native grid is 20 ms. The digits are real arithmetic; they are not real measurement.

Why Cuts Must Snap to Word Boundaries

Text-based editing is a mapping: delete a sentence in the transcript, delete the corresponding range from the source. The range comes entirely from the timestamps, so the quality of the edit is the quality of the timing. Two rules follow, and the second one is the one tools get wrong.

A word boundary is necessary

Cut inside a word and you truncate a phoneme. A word-final stop like /t/ or /k/ is a silent closure followed by a release burst — cut between the two and the word ends on a hole the ear reads as a swallowed syllable. A fricative like /s/ runs 100–150 ms and clips into an audible hiss-cut. Nothing downstream rescues this: no crossfade hides a missing consonant, because the information was never recorded in the range you kept.

A word boundary is not sufficient

Inside a phrase, the boundary between two words is notional rather than acoustic. Speakers coarticulate — the end of "shipped" is already shaping the vowel of "it" — so a splice there is clean at the sample level and still wrong at the sentence level, because it joins two different pitch contours mid-breath. The listener hears the prosody jump even though no audio is damaged. The genuinely safe cuts are at pauses, which is why word timing is used with silence detection and not instead of it: the timestamps decide which words survive, the silences decide where the join can hide.

Budget for the boundary error

Because ends carry 20–30 ms of error on average and can swallow trailing silence, cutting exactly at the reported end risks clipping the release. Practical systems keep a short tail. The mirror-image mistake is removing every millisecond of silence: speech with no breath in it reads as an unnatural rush, so silence removal targets a floor — leave a short gap — rather than zero.

Picture is quantised; audio is not

An audio edit can land anywhere. A picture edit lands on a whole frame. A cut computed at 12.556 s becomes frame 377 at 30 fps — 12.567 s — an 11 ms shift the sound does not take unless you make it. Sound and picture have to be quantised together at the same edit point, or every cut drifts by a fraction of a frame.

A Scenario Where It Shows

You record a 47-minute interview and you want the two minutes about pricing. Searching the transcript for "pricing" finds it at 21:14. With segment timing you get an utterance that begins mid-answer; you trim by ear, keep the tail of the previous question, and lose the final consonant of the last word. With word timing you get the exact span — first word's start, last word's end — then extend backwards to the breath before it and cut in the silence.

Then you ask for the fillers to be removed. A single "um" is about 210 ms. Across 47 minutes there might be a couple of hundred of them, which is only a minute of running time — but it is a couple of hundred separate splices, every one of which has to land on a word boundary and preferably inside an existing pause. That job is not tedious without word-level timestamps; it is impossible, because there is nothing to address the words by.

Then you ask for karaoke captions. The highlight advances on each word's start. Since the average word is ~440 ms, a systematic 100 ms error is a quarter of a word — small enough to survive a spec sheet and large enough that viewers see the highlight lagging the voice. Caption timing is the most visible consumer of word-level accuracy, because the error is on screen next to its own ground truth.

Common Misconceptions

"Whisper gives you word-level timestamps out of the box."

It gives you segment-level timestamps out of the box. Word-level timing is an additional pass — either DTW over the model's own cross-attention, or a separate forced aligner run afterwards. This is why wrappers like whisper-timestamped, stable-ts and WhisperX exist at all, and why two tools built on the same Whisper checkpoint can disagree about where a word starts.

"The timestamps are exact."

They are estimates with an error distribution. Even a strong forced aligner puts only about two thirds of word boundaries within 20 ms, and a "word boundary" is not a crisp physical event in the first place — trained human annotators disagree with each other about where one is. Treat a word timestamp as a measurement with a tolerance, not a fact.

"The end of one word is the start of the next."

Inside a phrase, yes, and only because there is nothing between them to measure. Between phrases, no: the pause belongs to neither word, and how a system reports it varies. DTW-based timestamps often hand the silence to the preceding word, which is exactly the pause-absorption problem above. If you compute "gap" as start[i+1] − end[i] and get zero everywhere, your pauses are hiding inside your words.

"If it's a word boundary, it's a safe cut."

It is a safe cut acoustically. It is not automatically a good cut. Mid-phrase word boundaries splice two prosodic contours, and the result sounds edited even when nothing is damaged. Word timing constrains where a cut can go; pauses, breaths and sentence structure decide where it should.

"SRT files can carry word timing."

SRT has no word-level facility whatsoever — a cue is a time range and a block of text. Writing one word per cue is a workaround that produces flashing single words, not a highlighted line. WebVTT does support it, via cue timestamp tags inside the payload (<00:00:12.556>shipped), and so do ASS and LRC. Everything else is exchanged as JSON, because subtitle formats have nowhere to put confidence or speaker fields.

"High word confidence means good timing."

The confidence score is about the identity of the word, not the placement of its edges. A word can be transcribed with 0.99 probability and still have a boundary 200 ms out — which is common precisely at pauses, where the transcription is easy and the timing is hardest.

How This Works in Valmera

Valmera is an agentic video editor, and word-level timing is the unit its agent edits in. Every upload is indexed once — a word-level transcript from a Whisper-family ASR model, plus silence detection and shot detection — and every later edit reads that same index rather than re-analysing the file.

  • All cuts snap to word boundaries. Range cuts, silence removal, filler-word removal (um, uh, er, hmm, plus custom words) and repeated-take detection all resolve to word-timed ranges rather than to a scrubbed guess.
  • Captions come from the same timings. Word-accurate, 1–16 words per caption, karaoke word-pop up to 6 words per line, per-word emphasis and per-word highlight colour. Editing the transcript re-renders them.
  • Emphasis is word-timed too. Automatic punch-ins land on the most vocally stressed words, which requires knowing where each word is, not just what was said.
  • Times are floats in source seconds, written to an edit decision list — not into a rendered file. Uploads arrive at whatever frame rate the camera chose, so seconds are the honest common unit and frame rounding happens once, at render. Because decisions are anchored in source time, a later cut earlier in the video does not invalidate a caption or a zoom placed after it.
  • The agent can read the timings directly — a word list, transcript search and a kept-only transcript are tools it calls, and the same tools are published on the MCP server so Claude can drive the edit with them.

The honest limits. Captions are burned into the video, and there is no SRT or VTT import or export — so the word timings drive the edit and are readable through the agent's tools, but they do not leave the project as a subtitle file or a data dump. English is the best-tested path. And the timings are model estimates carrying the same boundary error described above; Valmera does not claim frame-exact alignment, and when a cut lands a beat late the fix is to say so and have it moved, not to assume the number was ground truth.

Related Terms

Other entries in the Valmera video editing glossary that touch this one:

  • Transcription and ASR — The system that produces the words these timestamps hang off.
  • Text-based video editing — Deleting a sentence deletes a source range — the mapping word timings make possible.
  • Silence detection — Where a cut can hide. Word timing says which words survive; silence says where to join them.
  • Burned-in captions vs subtitles — Why karaoke word-pop is a rendering decision and not a subtitle file.
  • Edit decision list — The document the resolved ranges are written into, in source time.
  • Jump cut — What a word-boundary splice looks like on the picture side.
  • J-cut and L-cut — Audio and picture edit points deliberately offset from each other.
  • Shot detection — The picture-side index. Word timing indexes the sound.

Frequently Asked Questions

Word-level timestamps are the start and end times of each individual spoken word in a recording, measured in that recording's own clock and usually accompanied by a per-word confidence score. They are produced either by an automatic speech recognition (ASR) model that emits timings together with the text, or by forced alignment, which takes an already-known transcript and finds where each word sits in the audio; typical resolution is tens of milliseconds. Once a transcript is timed word by word rather than sentence by sentence, the text becomes an index into the media: a phrase search returns a range you can cut, a caption can highlight one word at a time, and every edit point can be snapped to a boundary between words rather than landing inside one.
Accuracy is measured as boundary error — how far the reported edge of a word sits from where a human annotator would put it — and it is reported against a tolerance. A strong forced aligner such as the Montreal Forced Aligner reports mean word-boundary errors around 21.9 ms on TIMIT and 27.8 ms on Buckeye, with roughly 65% of boundaries landing within a 20 ms tolerance and about 88% within 50 ms on Buckeye; MFA 3.0 reports mean boundary errors below 15 ms. Papers evaluating ASR-native timestamps commonly use a 200 ms collar, which tells you how much slop that field considers acceptable. For scale, the average English word is roughly 440 ms long, and a video frame is 33.3 ms at 30 fps. So a good aligner's average error is smaller than one frame — but individual words can be much further out, and the error is not random: systems that infer timing from attention tend to absorb pauses into the adjacent word.
Not directly. Whisper natively emits segment-level timestamps for whole utterances, from timestamp tokens in its vocabulary — 1,500 of them spanning <|0.00|> to <|29.98|> in 0.02-second steps across its fixed 30-second window. Word-level timing is a separate post-pass over the decoder's cross-attention. A chosen subset of attention heads, the alignment heads, are known empirically to track time; their attention matrices are median-filtered to denoise them, then dynamic time warping (DTW) finds the monotonic path of highest cumulative attention through the token-by-frame matrix. Where that path crosses gives each token a start and end, and tokens are merged back into words with punctuation attached. The underlying grid is 20 ms — Whisper's audio front end takes 16 kHz mono with a 25 ms window and a 10 ms hop, giving 100 frames per second, and a convolution of stride 2 halves that to 50 encoder positions per second.
A segment-level timestamp covers a whole utterance — typically a phrase or sentence, and in Whisper's case up to 30 seconds. It tells you when a block of text starts and stops, which is enough to build a subtitle file and nothing else. Word-level timing subdivides that block, giving each word its own span. The difference is not a matter of degree: segment timing lets you display text, word timing lets you address the media. Searching for a phrase, cutting a specific sentence out of an answer, removing one 'um', highlighting a word as it is spoken and snapping an edit to a word boundary all require the finer unit, and none of them are approximable from segment times.
Not meaningfully. SRT has no word-level facility at all — a cue is a time range plus a block of text, and the common workaround of writing one word per cue produces a file that flashes single words rather than a highlighted line. WebVTT does support it properly, through cue timestamp tags placed inside the cue payload (for example <00:00:12.556>shipped), which is the W3C's paint-on or karaoke mechanism; the tags must fall inside the cue's own range and increase strictly. ASS and LRC also carry per-word timing. In practice, though, word timings are exchanged as JSON, because you usually want the per-word confidence and speaker fields too, and no subtitle format carries those.
Because cutting inside a word truncates a phoneme, and the ear notices immediately. A word-final stop such as /t/ or /k/ is a closure followed by a release burst; cut between the two and the word ends on silence that reads as a swallowed syllable. A fricative like /s/ runs 100–150 ms and clips into an obvious hiss. So a word boundary is necessary. It is not sufficient: inside a phrase there is no acoustic gap between words — 'shipped it' is coarticulated, and the boundary is notional rather than audible — so splicing there joins two pitch contours mid-sentence and reads as a jump in prosody even when the audio itself is clean. The good cuts are at pauses, which is why word timing is used alongside silence detection rather than instead of it: the timestamps say which words survive, the silences say where the join can hide.
Valmera uses them for everything it does with speech, and its agent can read them, but it does not export them as a file. Every upload is indexed once into a word-level transcript alongside silence detection and shot detection, and that index is what every later edit reads. All cuts snap to word boundaries. Silence removal, filler-word removal, repeated-take detection, transcript search and automatic punch-ins on vocally emphasised words all resolve to word-timed ranges. Captions are word-accurate from the same timings, including karaoke word-pop up to 6 words per line and per-word emphasis, and editing the transcript re-renders them. The honest limit: captions are burned into the video, and there is no SRT or VTT import or export — so the word timings drive the edit and are readable through the agent's tools, but they do not leave the project as a subtitle or data file.

Edit by the Word

Upload real footage, and the transcript becomes the timeline — cuts on word boundaries, captions on word starts. 50 free credits, no card.

Start free →
See pricing →

Related Articles

Text-Based Video Editing
Edit the transcript, and the video follows — the workflow word timings exist to enable.
Karaoke Captions
Word-by-word highlighting, driven directly by each word's start time.
Captions
Presets, fonts, animations, per-word emphasis and the editable transcript.
Remove Filler Words
Hundreds of sub-second cuts, each one placed on a word boundary.
Timeline & Transcript
Two views onto the same timed document — scrub it, or read it as text.