Auto Reframe
Auto reframe is the automatic conversion of a video from one aspect ratio to another, in which software decides where the new frame sits over the original picture instead of an editor placing and keyframing the crop by hand. It finds the important content in each shot (usually a face, otherwise an object or the most detailed region), places the crop window over it and, in some tools, moves the window as the subject moves. It is a crop, not generation: a 9:16 window taken from 16:9 footage keeps 31.6% of the width. Better implementations also decide when not to crop, and fit the whole frame over bars or a blurred backdrop instead.
What a Crop Keeps
Keep the full height of a 16:9 frame and the widest window you can take for each target is fixed by arithmetic. The share of width kept is simply the target ratio divided by the source ratio.
| Conversion | Crop window from 1920×1080 | Width kept | Width discarded |
|---|---|---|---|
| 16:9 → 9:16 | 608×1080 | 31.6% | 68.4% |
| 16:9 → 4:5 | 864×1080 | 45% | 55% |
| 16:9 → 1:1 | 1080×1080 | 56.25% | 43.75% |
Fitting is the opposite trade: the whole picture survives, but a 16:9 shot fitted into a 9:16 frame fills only 31.6% of it. Nothing outside the original frame was ever recorded, so every conversion gives up either picture or screen area. The full ratio math is in the aspect ratio entry.
The job itself is old. When widescreen films were shown on 4:3 television, a telecine operator chose, shot by shot, which part of the frame to keep and slid the window when the action moved. That was pan and scan, and 2.39:1 to 4:3 keeps about 56% of the width. Going from 16:9 to 9:16 keeps 31.6%, so the modern version is harsher.
How Auto Reframe Works
Implementations differ, but the serious ones share four stages. Google’s AutoFlip, open-sourced on MediaPipe in 2020, is the clearest published example.
1. Split the video into shots
A crop must never slide across a cut, or the viewer sees camera movement with no cause, so every decision is made per shot. AutoFlip detects shot changes by comparing each frame’s color histogram with the frames before it. Reframing is only as good as its shot detection.
2. Find what matters in the frame
Detectors for faces, objects and on-screen text run on sampled, downscaled frames; AutoFlip analyses one frame every 200 ms at 480 pixels wide. Each signal is marked required (it must stay in frame) or best-effort. For dialogue, the hard part is active-speaker detection: a face detector says where the faces are, not who is talking.
3. Place the crop, or plan a path
Re-centering on the subject every frame makes the picture jitter, because detections are noisy and heads move constantly. Good implementations hold still unless the subject really travels (AutoFlip keeps the window fixed when the salient content stays within half the frame width across the shot) and otherwise solve a smooth path of holds, pans and eases, the way a camera operator would.
4. Decide whether to crop at all
If the content that matters can’t fit inside the crop window at any position, cropping fails, and the right answer is to fit the whole frame over bars or a blurred copy of itself. Many tools skip this stage, and it is where most bad vertical edits come from: game HUDs, interfaces and slides cut in half.
Crop or Fit: When Each Is Right
| Mode | What it does | What it costs | Right when |
|---|---|---|---|
| Static crop | One fixed window for the shot | 68.4% of the width (16:9 → 9:16) | A subject who holds position: a seated talking head, a locked-off product shot |
| Tracking crop | The window follows the subject | The same width, plus the risk of camera moves the shot never had | A subject who travels: a presenter walking a stage, a ball in play |
| Pad (bars) | Whole frame fitted with black bars | 68.4% of a 9:16 frame is empty | Archive or client footage whose composition must stay intact |
| Blurred pad | Whole frame over a blurred copy of itself | Same area, but the screen reads as full | Content that runs edge to edge: gameplay, screen recordings, slides, wide scenes |
Tracking is not automatically better than a static crop. A window that re-centers on every small head movement adds camera motion the shot never had; the test is whether the subject actually leaves the window. Between the two pads, bars read as a letterbox, while a blurred copy of the frame reads as a full screen. Neither adds real picture.
Where Auto Reframe Goes Wrong
- The wrong person. The largest face is often the listener, not the speaker. Speaker turns from audio or the transcript, or a manual override, fix it.
- Content that runs to the edges. Cropping gameplay, a screen recording or a slide doesn’t reframe it; it truncates it. Fit those shots.
- Two people too far apart. In 1920×1080 the 9:16 window is 608 pixels wide. Two people seated a third and two thirds across the frame are 640 pixels apart, so no crop holds both. The answers are edits: cut between a crop on each speaker, or fit the shot with room for a headline and captions.
- Captions burned in too early. Burned-in captions added before the reframe get cropped with the picture. Add them after, laid out for the new frame.
- One pass treated as final. Every serious tool presents the result as a starting point to review shot by shot.
How Valmera Does It
In Valmera you type “make it 9:16” and the agent reframes the footage you kept. It finds faces in the pixels, confirmed across up to about 180 indexed frames, and aims the crop at them. It measures the subject in each camera shot and re-aims the crop at every cut. When a crop would keep less than 55% of the picture’s detail, as with gameplay, slides or screen recordings, it fits the shot over a blurred copy of itself instead. When two people share a shot, it fits and flags it, then sets a crop position for each speaker section from the speaker-labeled transcript and the frames, for you to check in the preview.
The crop never pans to follow movement within a shot: these are per-section crop positions, not continuous motion tracking. Padding is always bars or your own footage blurred. Each choice is stored in the edit decision list, so any crop can be moved or undone. The modes, prompts and resolution details are on the auto reframe tool page.
Frequently Asked Questions
Made with Valmera: real exports
Reframe by Describing the Platform
Upload a landscape video, ask for 9:16, and the agent decides shot by shot whether to crop or fit. Creating an account and uploading are free. Editing requires a subscription.
Create an account →