What Is Shot Detection?

Shot, Scene and Take

A shot is one unbroken run of frames from one camera. A scene is the set of shots that share a place and a time. A take is one attempt at a shot. Nearly every tool sold as “scene detection” detects shots, which matters once you act on the output: a transition at every “scene change” of a two-camera interview lands mid-conversation.

How Shot Detectors Work

1. Reduce each frame

Raw pixels change with every camera move, so detectors compare a reduced version of each frame: a color histogram (which ignores position, so a pan barely moves it), HSV values that separate color from brightness, edges, or a perceptual hash.

2. Score the change

PySceneDetect’s ContentDetector averages the frame-to-frame change in hue, saturation and brightness into one score on a 0–255 scale. Its default threshold is 27, roughly a 10% average change.

3. Apply a threshold and a minimum shot length

A boundary goes wherever the score crosses the threshold, and a minimum shot length (15 frames by default) drops boundaries that arrive too close together. Counted in frames, that minimum is 0.25 s at 60 fps but 0.625 s at 24 fps, so the same settings can return different shot lists at different frame rates.

Adaptive thresholds. A fixed threshold low enough to catch a cut between two similar shots also fires on fast camera moves. An adaptive detector compares each frame’s score with the average of its neighbors and fires only on a spike; PySceneDetect’s AdaptiveDetector needs a ratio of 3.0 and a minimum score of 15. A cut is a spike against its own neighborhood; a pan is a plateau.

Watch the scales: ffmpeg’s scdet takes 0–100 and defaults to 10, while the scene value in its select filter runs 0–1 and is usually written gt(scene,0.4). Paste that 0.4 into scdet and you get a threshold 25 times below its default and a boundary every few frames.

Hard Cuts vs Gradual Transitions

A hard cut is the easy case: one frame belongs to the old shot, the next to the new one, and the score spikes in a single step. Gradual transitions are built to avoid that spike. A one-second dissolve at 30 fps spreads the change over 30 frames, so each pair differs by only about 3% of it, far below any threshold that survives ordinary camera motion, and the two shots come back as one. A wipe slips past the same way, one strip of the frame at a time. Fades are easier: PySceneDetect’s ThresholdDetector watches average brightness, so it catches a fade to or from black, but not a dissolve, where brightness never dips.

Neural detectors close much of the gap by looking at many frames at once. TransNet V2 reads 100-frame windows shrunk to 48×27 pixels and outputs a boundary probability per frame; its published F1 scores are 96.2 on BBC Planet Earth and 93.9 on RAI (broadcast material) but 77.9 on ClipShots (web video). Shot boundary detection was a TRECVID benchmark task from 2001 to 2007, where gradual transitions were scored separately because a dissolve has no single correct frame.

Common Failure Modes

• Camera flashes and strobes. A photographer’s flash, lightning or a light switching on changes every pixel for a frame or two, exactly like a cut.
• Fast motion. A whip pan, a fast handheld move or someone walking past the lens scores as high as a real cut.
• Lighting drift and noise. A cloud crossing a window creeps the score up; dark, high-ISO footage adds noise to every frame.
• Look-alike shots. Two cameras with similar framing on the same subject can differ by less than the threshold, so a real cut is missed.
• Baked-in transitions. Dissolves and fades already in the file merge two shots into one.
• Keyframes mistaken for cuts. Encoders place I-frames on a fixed schedule, so most keyframes sit mid-shot.

The usual fixes: an adaptive detector for flashes and motion, a lower threshold or a histogram detector for look-alike cuts, and detecting shots before transitions are added. When two runs disagree, check the frame rate before the threshold.

How Valmera Uses Shot Detection

Valmera runs shot detection once, at upload, as part of the index its AI agent edits from: PySceneDetect’s ContentDetector at its default threshold of 27, on the proxy Valmera builds for analysis and preview. The shot list then shapes several edits:

• Transitions land only on real scene changes. By default a transition goes where the shot actually changes or a clip is spliced in, never on the jump cuts a silence pass leaves inside one continuous shot, and the agent reports how many junctions it used and skipped.
• Reframing is decided per shot. Auto reframe measures the subject in each camera shot and re-aims the vertical crop at each cut; within one shot the crop holds still.
• Highlights and b-roll are timed to the picture. Shot boundaries inform where highlights and shorts start and end, and where b-roll comes in.
• The agent reads them directly. get_shots returns the boundaries for any time range; filmstrip frames show what is actually on screen. Claude, ChatGPT and other assistants call the same tool through Valmera’s MCP server.
• Failures are visible. If detection can’t run on a file, the video is treated as one shot and the failure is logged.

Limits

Valmera runs content detection only, so dissolves and fades already baked into an upload can merge two shots. The threshold isn’t a per-project setting and the shot list can’t be edited by hand; if two similar angles merge, describe the cut you want instead. There’s no shot-list export for other editing software; the deliverable is an MP4. Valmera’s own transitions are junction effects such as dip to black and whips; true crossfades aren’t supported.

Sources

• PySceneDetect detector reference, checked October 6, 2026
• FFmpeg filters documentation (scdet and select), checked October 6, 2026
• Souček and Lokoč, “TransNet V2: An effective deep network architecture for fast shot transition detection” (2020), checked October 6, 2026
• Smeaton, Over and Doherty, “Video shot boundary detection: Seven years of TRECVid activity,” Computer Vision and Image Understanding (2010), checked October 6, 2026

Frequently Asked Questions

Made with Valmera: real exports

Let the Agent Read the Shot Structure

Upload real footage and describe the edit; transitions land where the picture actually changes. Creating an account and uploading are free. Editing requires a subscription.