Shot Detection

Shot, Scene, Take: The Words Are Not Interchangeable

A shot is a mechanical unit — one unbroken run of frames from one camera, bounded by the camera starting and stopping or by two splices in an edit. A scene is a narrative unit: the shots that share a place and a time. A take is one attempt at a shot, most of which never reach the edit.

Nearly every tool advertising "scene detection" detects shots. PySceneDetect returns a list it calls scenes; they are shots. The naming is harmless until you act on the output. "Put a transition at each scene change" is a reasonable instruction; applied to a shot list on a two-camera interview it puts a whip pan in the middle of every sentence. Grouping shots into scenes is a genuinely different problem, because it needs an understanding of content rather than a measurement of difference.

How the Measurement Works

The classical algorithm is three steps and has barely changed since the 1990s. Reduce each frame to something comparable. Score the difference between consecutive frames. Declare a boundary where the score crosses a threshold.

Here are the real defaults, across the implementations most tools are built on:

Two rows of that table are the same idea on incompatible scales, and it is the commonest misconfiguration in the field. ffmpeg's scdet filter takes a threshold from 0 to 100 and defaults to 10. The scene variable inside the select filter runs from 0 to 1, and the expression everyone copies is select='gt(scene,0.4)'. Paste that 0.4 into scdet and you have asked for a threshold forty times more sensitive than intended — which returns a boundary every few frames and reads as a broken filter rather than a unit error.

Cuts Are Solved. Dissolves Are Not.

A hard cut is the easy case by construction: one frame belongs to the old shot, the next belongs to the new one, and the difference score spikes in a single step. A gradual transition — dissolve, fade, wipe, iris — is engineered to be the opposite. A one-second dissolve at 30 fps spreads the entire change across 30 frames, so each consecutive pair differs by about 3% of it. That is far below any threshold set high enough to survive an ordinary camera pan, so the detector returns nothing and the two shots are reported as one.

You cannot fix this by lowering the threshold, because you are already between two failure modes. Below roughly 20 on ContentDetector's scale you start firing on handheld motion, on a subject crossing the lens, and on sensor noise in dark footage. Above roughly 35 you start missing real cuts between two shots that happen to look alike — which is exactly the two-camera interview, the most common footage anyone owns.

The field's partial answers are all about looking at more than two frames. PySceneDetect ships a separate ThresholdDetector that watches average frame brightness against a default of 12, so it catches fades to and from black — but not a dissolve between two shots, where brightness never dips. Its AdaptiveDetector compares each frame's score to a rolling average of its neighbours and fires when the ratio exceeds 3.0, with a floor of 15.0 so noise cannot produce a large ratio out of two small numbers; that reframes the question usefully — a cut is a spike relative to its own neighbourhood, a pan is a plateau — but a dissolve is a plateau too.

Neural detectors close most of the remaining gap by seeing the whole event at once. TransNet V2 runs a dilated 3D convolutional network over a sliding window of 100 frames downscaled to 48×27 pixels — 1,296 pixels per frame, which is a deliberate statement that boundary detection is about global change and not about detail — and emits a per-frame boundary probability. Its published F1 scores tell you exactly how much of the problem remains: 96.2 on BBC Planet Earth and 93.9 on RAI, both broadcast material, against 77.9 on ClipShots, which is web video. The eighteen-point gap is what messy real-world footage costs. Shot boundary detection was a formally evaluated task at TRECVID from 2001 to 2007, where gradual transitions were scored with their own metrics — frame-recall and frame-precision — for the reason above: a dissolve has no single correct frame to be right about.

What a Shot List Gives an Editing Agent

A shot list is four numbers per shot at most, and it is disproportionately useful — not because it describes the video, but because it is the only thing that says where the picture is continuous.

• It separates scene changes from jump cuts. After a silence pass, a talking-head video has a junction per removed pause and nearly all of them sit inside one continuous shot. Those are jump cuts and they work by being invisible. Without a shot list, "add transitions" means a full-screen effect every couple of seconds through footage that never changed scene.
• It bounds anything with duration. A zoom, a speed ramp, a text overlay or a punch-in that crosses a cut reads as a mistake, because the viewer sees the effect survive a change it should not have survived. Shot boundaries are the safe start and end points.
• It is the unit for framing and colour. A crop that follows a subject has to reset when the camera changes, and a grade is matched shot by shot. Both are per-shot decisions in every manual workflow, for the same reason.
• It classifies the footage before anything is edited. One shot plus dense speech is a talking head. Forty shots plus almost no speech is montage material that needs a voiceover or music to carry it. Those two want opposite edits, and the shot count is half of what distinguishes them.
• It makes long video navigable. On a 40-minute recording the shot list is a table of contents for the picture, in the same way that word-level timestamps are one for the speech.

What a shot list does not give is the picture. It carries timings, not content — it can say that the camera changed at 41.2 s and cannot say what it changed to. Anything that needs to know what is on screen has to look at frames.

A Scenario Where It Shows

You record a 25-minute interview with two cameras, cut between them in a rough assembly, and export a single file with a half-second dissolve on every camera change. Then you hand that file to a tool and ask for a highlight reel.

The detector reports far fewer shots than you cut. Every dissolve was spread over 15 frames, none of the consecutive pairs crossed the threshold, and adjacent camera angles were merged into one long shot. Meanwhile it reports boundaries you never made: the moment your guest gestured across the frame, the moment the ring light flickered, and a run of six "shots" during a slow zoom, because the score sat just above the threshold for two seconds.

Every downstream decision then inherits both errors. Clip boundaries land mid-dissolve, so a clip opens on a half-faded frame. Transitions are placed at the flicker and skipped at the real angle change. A per-shot grade applies one look across two cameras with different white balance. None of that is a threshold problem; the threshold that would catch the dissolves fires on the gesture. It is the shape of the frame-difference method, and the two real answers are a detector that sees more than two frames at a time, or — far better — detecting shots on the footage before the dissolves were baked into it.

Common Misconceptions

How This Works in Valmera

Valmera is an agentic video editor, so shot detection is part of the index the agent edits from rather than a panel you open. Every upload is analysed once and the result is reused by every later edit.

• PySceneDetect ContentDetector at a threshold of 27.0, run at index time on the 540p proxy rather than the master. The downscale averages out sensor noise and is far faster; the boundary timings are the same either way, because a cut is a cut at any resolution.
• The shot list is cut geometry, not the picture. What the agent looks at is a set of labeled frame tiles covering the whole video, refreshed into its context on every message, plus the ability to request frames at exact times. The shot list answers when the camera changed; the frames answer what is there. Keeping those separate is deliberate — the failure it prevents is aiming a zoom or placing text from a timestamp nobody looked at.
• Transitions are bound to real shot changes. The default scope for transitions keeps only the junctions where the two sides come from different indexed shots, or where a clip has been spliced in. If every junction in the edit is a jump cut inside one continuous shot — the normal outcome after a silence pass — the agent does not apply the transition at all, and says why. Setting a transition on every cut is available, but you have to ask for it.
• Shot count feeds how the footage is classified. Dense speech across very few shots is treated as a talking head; many shots with little speech is treated as montage material. That classification changes what the agent proposes before you have asked for anything.
• Failure is visible, not silent. If detection cannot run on a file, the whole video is treated as one shot and the degradation is recorded as a warning rather than presented as a result. One shot is the honest answer when nothing could be measured; it is also the correct answer for a locked-off talking-head recording, and the two look identical unless the failure is reported.
• Readable directly. get_shots returns the boundaries for any time range, in the studio and over the MCP connector, so Claude can ask the same question the in-house agent asks.

The honest limits. Valmera runs content detection only. Gradual transitions already baked into an uploaded file — dissolves, fades, wipes from a previous edit — are not reliably found, for the reason described above, and the shot list carries no label saying whether a boundary was a cut or something softer. The threshold is not exposed as a per-project setting. Shot boundaries are also not editable by hand: if the detector merges two similar camera angles, you cut at the right moment by describing it or by using the timeline, not by correcting the shot list. And there is no shot-list export in an interchange format such as CMX 3600 or FCPXML — the boundaries are readable in the product and over MCP, and the deliverable is a rendered MP4. Valmera also has no true crossfade of its own; its transitions are duration-preserving junction effects, so it neither creates nor detects a real dissolve.

Related Terms

Other entries in the Valmera video editing glossary that touch this one:

• Edit decision list — Where a detected boundary becomes an actual cut — a range in a keep list rather than a change to the file.
• Jump cut — What you get when you cut inside a shot instead of between two. Shot detection is how a tool tells the two apart.
• B-roll — Splicing a clip in creates a new shot boundary that no detector had to find — the edit made it.
• J-cut and L-cut — Deliberately putting the picture boundary and the audio boundary at different times.
• Silence detection — The audio-side sibling: the same threshold-and-minimum-duration shape, applied to level instead of pixels.
• Proxy editing — Why detection normally runs on a small proxy — a downscale removes sensor noise and is an order of magnitude faster.
• Auto-reframe — Reframing is decided per shot, because a crop that follows a subject has to reset when the camera changes.
• Color grading — Grades are matched shot by shot; the shot list is the unit a colourist works in.

Frequently Asked Questions

Let the Agent Read the Shot Structure for You

Upload real footage and describe the edit. Shot boundaries, transcript and frames are indexed once, and transitions land where the picture actually changes. 50 free credits, no card.