How accurate are word-level timestamps?

Good forced aligners place most word boundaries within 20–50 ms of where a human annotator would, with average errors in published evaluations of roughly 15–30 ms, less than one video frame at 30 fps (33 ms). Individual words can be much further out, especially around pauses, and timestamps taken from a recognizer's attention tend to absorb a following pause into the previous word.

How does Whisper produce word-level timestamps?

Not natively. Whisper emits segment-level timestamps for whole phrases. Word timing comes from a second pass: either dynamic time warping over the decoder's cross-attention, which finds the most likely monotonic path between words and audio frames, or a separate forced aligner such as the one WhisperX uses. Its underlying grid is 20 ms per encoder position.

What is the difference between word-level and segment-level timestamps?

A segment timestamp covers a whole phrase or sentence: enough to display a subtitle, not enough to edit with. Word-level timing gives each word its own span, which is what you need to cut one sentence out of an answer, remove a single "um", highlight a word as it is spoken, or snap an edit to a word boundary.

Can SRT files carry word-level timing?

No. An SRT cue is one time range and a block of text; one word per cue just flashes single words. WebVTT can mark per-word timing inside a cue, and ASS supports per-syllable karaoke timing. Word timings are usually exchanged as JSON so confidence and speaker fields come along.

Why do cuts have to snap to word boundaries?

Cutting inside a word truncates a sound: a final /t/ loses its release, an /s/ clips into a hiss, and no crossfade can restore audio that was cut away. A word boundary is necessary but not always sufficient: inside a phrase, words run together, so the cleanest cuts land in the pauses between phrases.

Word-Level Timestamps

Word-level timestamps are the start and end times of each spoken word in a recording, usually with a confidence score per word. Speech-recognition models produce them directly or by forced alignment against a known transcript, typically accurate to a few tens of milliseconds. They turn a transcript into an index of the media: you can search it, cut on word boundaries, and time captions word by word.

How a Word Gets a Time

ASR-native timestamps come from the recognition model itself, which outputs words and timings in one pass, or from a second pass over its internal attention.
Forced alignment starts from known text and finds where each word sits in the audio, typically with a phoneme or character model and CTC alignment. It is the easier problem and usually the more accurate one; WhisperX, for example, transcribes with Whisper and then re-aligns.
The grid underneath. Speech models analyze audio in short frames: Whisper computes a spectrogram with a 25 ms window every 10 ms and halves that in its encoder, so its native resolution is 20 ms. Digits printed beyond that are arithmetic, not measurement.

What the Data Looks Like

"words": [
  { "word": "we",      "start": 12.437, "end": 12.556, "probability": 0.994 },
  { "word": "shipped", "start": 12.556, "end": 12.921, "probability": 0.981 },
  { "word": "it",      "start": 12.921, "end": 13.024, "probability": 0.997 },
  { "word": "on",      "start": 13.024, "end": 13.160, "probability": 0.992 },
  { "word": "Friday",  "start": 13.512, "end": 14.005, "probability": 0.989 }
]

Inside a phrase, each word ends exactly where the next begins, because continuous speech has no gap to measure. The 352 ms before “Friday” belongs to neither word: it is a breath, and it is where a cut can hide.

How Accurate They Are

Accuracy is reported as boundary error: how far a reported word edge sits from where a human annotator would place it. Good forced aligners put most boundaries within 20–50 ms, and average errors in published evaluations run roughly 15–30 ms. For scale, an average English word lasts around 400 ms and a video frame is 33 ms at 30 fps, so typical error is smaller than a frame. That is why word timing is good enough to edit with.

The error that matters is systematic. Timestamps inferred from a recognizer's attention tend to absorb a following pause into the previous word, so a reported word end can include hundreds of milliseconds of silence. If cuts feel late by a beat, that is usually why.

Why Cuts Snap to Word Boundaries

A word boundary is necessary. Cutting inside a word truncates a sound, and nothing downstream can restore it.
It is not always sufficient. Mid-phrase, words run together, so a splice there joins two pitch contours and sounds edited. The cleanest cuts land in pauses, which is why word timing works with silence detection.
Leave a little tail. Word ends carry error, so editors keep a short margin of air around speech instead of cutting exactly at the reported end.
Picture rounds to frames. Audio can cut anywhere; picture cuts land on whole frames. Both have to be rounded together, or sound drifts from picture by a fraction of a frame per cut.

Common Misconceptions

“Whisper gives word timestamps out of the box.” It gives segment timestamps; word timing is an extra pass, which is why tools built on the same model disagree about where a word starts.
“The timestamps are exact.” They are measurements with a tolerance, and human annotators disagree about boundaries too.
“High confidence means good timing.” Confidence is about which word it is, not where its edges are.

How Valmera Uses It

Word-level timing is the unit Valmera's agent edits in. Every upload is transcribed once with word timestamps and speaker labels, and filler words such as “um” and “uh” are kept with their own timings so they can be cut. Silence removal, filler removal, retake cuts and transcript-based cuts all resolve to word-timed ranges and snap to whole words. Captions are built from the same timings, which is what lets a karaoke highlight move word by word, and automatic emphasis lands on the words the speaker stressed.

Decisions are stored in source seconds in an edit decision list, so cutting something early never shifts a caption or zoom placed later. The agent reads word lists, transcript search and the kept transcript as tools, and the same tools are available to Claude, ChatGPT and other assistants through the Valmera MCP server. Timings are not exported as a subtitle file (no SRT or VTT), and English is the best-tested language. If a cut lands a beat late, say so and the agent moves it.

Related Terms

Transcription and ASR — The system that produces the words these timestamps hang off.
Text-based video editing — Editing a video through its timed transcript.
Silence detection — Words say which speech survives; pauses say where a cut can hide.
Burned-in captions vs subtitles — Why word-by-word highlights are rendered, not shipped as a file.
Edit decision list — Where resolved ranges are written, in source time.

Frequently Asked Questions

Made with Valmera: real exports

Edit by the Word

Upload real footage and the transcript becomes the map: cuts on word boundaries, captions on word starts. Creating an account and uploading are free; editing requires a subscription.

Create an account →

See pricing →