Auto-reframe
Auto-reframe is the automatic conversion of video from one aspect ratio to another, in which software decides where the new, differently shaped output frame should sit over the original picture — instead of an editor positioning and keyframing that crop by hand. It analyses the footage to find the salient content (usually faces, otherwise a detected object or the region carrying the most visual detail), places the output window over it, and where the subject moves, moves the window with it. It is a cropping operation, not a generative one: converting 16:9 to 9:16 leaves a window 31.6% as wide as the source, so auto-reframe is choosing which two thirds of the picture to discard. The better implementations also decide whether to crop at all, and fit the whole frame inside the new one — over bars or a blurred backdrop — when the content runs edge to edge.
The problem, and the fact that it is not new
Almost everything is still recorded wide — cameras, webcams, screens, conference rooms — and most of it is now watched tall. Converting between the two is not a setting, because the two frames do not contain the same picture. Keep the full height of a 1920x1080 frame and the widest 9:16 window you can take out of it is 608 x 1080: 31.6% of the width. Two thirds of every frame has to go somewhere, and the only questions are which two thirds and whether losing them is acceptable.
This job used to be done by a person, and it had a name. In the 1950s the studios adopted CinemaScope and Todd-AO precisely because television could not show them; a decade later those same films had to be broadcast on 4:3 sets anyway. Pan and scan was the answer: a telecine operator watched the film and chose, shot by shot, which part of the 2.35:1 frame to keep, sliding the window within a shot when the action demanded it. Reducing 2.35:1 to 4:3 discards roughly 45% of the image, and across formats the reported losses ran as high as 75%. It was widely, and correctly, regarded as vandalism.
Two things about that are worth carrying forward. The decision was made per shot, not per film — and it was made by a human who had watched the whole thing, and it still ruined compositions. Auto-reframe is that job automated, and the modern version is harsher: 2.35:1 into 4:3 keeps 57% of the width, while 16:9 into 9:16 keeps 32%. The tooling got better. The arithmetic got worse.
How it actually works
Implementations differ, but the serious ones are all the same four stages. Google's AutoFlip, open-sourced in 2020 on MediaPipe, is the clearest published example and the numbers below are its defaults — useful because they show what the design is actually worried about.
1. Cut the video into shots
Reframing is a per-shot decision because a crop path must never cross a cut — slide the window across an edit and you get camera movement with nothing motivating it. AutoFlip signals a shot change by computing a colour histogram for each frame and comparing it against a sliding window of the previous ones (defaults: window_size: 15 frames, min_shot_span: 0.2 seconds, so nothing shorter than a fifth of a second is treated as its own shot). Everything downstream then runs inside a shot. Shot detection is the dependency, and a reframe is only ever as good as it.
2. Find what matters in the frame
Several detectors run over the picture and their outputs are fused: face detection, generic object detection, text and logo detection, and a learned saliency map for footage with no recognisable subject in it. The structural idea worth stealing from AutoFlip is that every detected feature is either required — a hard constraint the output window must contain — or best-effort, a weighted preference. That single flag is what lets one framework express both "keep the speaker's eyes and mouth in frame whatever else is lost" and "include the product if you can". AutoFlip even splits faces into tiers for this: core landmarks, all landmarks, and the full head, each with its own confidence threshold around 0.8–0.9.
For dialogue the important detector is active-speaker detection. A face detector finds every face in every frame; an audio-visual model then correlates lip motion against the audio track to decide which of them is talking, so the crop follows the person speaking rather than the largest face. This is the capability that separates tools which handle a two-person interview from tools which do not, and it is a different and harder problem than face detection.
All of this runs small and cheap. AutoFlip scales the video to target_width: 480 and thins the detection stream to one frame every 200ms — about 5fps. Running detectors on every frame of a 4K master buys nothing, because the resulting path is going to be smoothed anyway.
3. Solve a camera path, not a per-frame position
The naive implementation centres the window on the subject in every frame. The result jitters, because detection is noisy and because people's heads move constantly by small amounts that a real camera operator would ignore. So the first question is whether to move at all: AutoFlip's motion_stabilization_threshold_percent: 0.5 holds the window completely still when the salient content stays within half the frame width for the whole shot, and only builds a tracking path when it does not. The paths themselves are approximated with constant, linear and parabolic segments — the model Google used in its L1-optimal stabilization work (Grundmann et al., CVPR 2011) to make an algorithmic camera path resemble a professional one: a hold, a pan, an ease. A snap-to-centre rule usually finishes the job, so footage that was framed centrally comes out exactly centred rather than one percent off.
4. Decide whether to crop at all
This is the stage most products skip, and it is where most bad output comes from. If the required content cannot fit inside the target window at any position, cropping cannot succeed, and the correct answer is to fit the whole frame into the output instead — over bars, or over a blurred, scaled-up copy of the same frame (AutoFlip's padding defaults are a blur_cv_size: 200 kernel at overlay_opacity: 0.6). Padding costs screen area: a 16:9 shot fitted into 9:16 fills about 32% of the output. Cropping costs picture. Something has to be surrendered, and which loss is cheaper is a property of the footage rather than a preference.
Four possible answers, and when each is right
| Mode | What it does | What it costs | Right when |
|---|---|---|---|
| Static crop | One fixed window for the whole shot | ~68% of the width, going 16:9 → 9:16 | A centre-weighted subject that holds position: a seated talking head, a locked-off product shot |
| Tracking crop | The window moves with the subject | The same picture, plus the risk of unmotivated camera movement | A subject that genuinely travels: a presenter who walks, a two-hander where the speaker changes |
| Pad (bars) | Whole frame letterboxed into the new one | About 68% of the output area is empty | Archive or client footage whose original composition has to survive intact |
| Blurred pad | Whole frame kept; bars filled with a blurred copy of it | Same area cost, but the frame reads as full | Content that runs edge to edge: screen recordings, gameplay with a HUD, wide scenes, on-screen text |
The two crop rows are frequently presented as a quality ladder, with tracking as the advanced option. They are not a ladder. A static crop is the correct answer for the large majority of talking-head footage, and a tracking crop applied to it makes the video worse by introducing movement the original camera never made. The test is whether the subject leaves the window, not whether the tool is capable of following.
The two pad rows differ only in what fills the empty area, and the choice is about attention rather than fidelity. Bars are honest and read as a letterbox — right for archive material and for anything where the viewer should understand they are seeing a wide frame. A blurred copy of the same frame reads as a full screen and keeps the phone's display from looking half-empty, which is why it has become the default treatment for wide footage in vertical players. Neither adds a single pixel of real picture.
The scenario where this bites
A founder records a twelve-minute product walkthrough in a single 1920x1080 take. The first two minutes are her on camera, seated slightly left of centre. The next nine are a full-screen recording of the app: a navigation sidebar down the left edge, a data table filling the right. The last minute is back to camera. She wants a 9:16 cut for Reels.
Run one auto-reframe over the whole file and whichever answer it picks is wrong for most of the runtime. A crop aimed at her face is right for three minutes; for the other nine it takes a 608px slice out of the middle of a user interface — sidebar gone on one side, half the table gone on the other, and what survives is unreadable on a phone anyway because it was drawn to be read at 1920 wide. Fit the whole frame instead and the screen recording stays intact and legible, but her talking-head sections now sit in a 1080x608 letterbox in the middle of a phone screen for three minutes.
There is no setting that resolves this, because the conflict is editorial. The real answers are all edits: convert the two camera sections with a crop aimed at her and the screen-recording section with a blurred pad; or crop the screen recording to the one region of the interface that matters, which is a framing decision about the app rather than about the video; or re-record the demo in a vertical browser window. Auto-reframe has a reputation for both saving hours and producing garbage for one reason — it is excellent at the mechanical part, deciding exactly where to put the window, and holds no view whatsoever on the editorial part, whether this shot should have been cropped.
Common mistakes
“It's AI, so it fills in the parts of the frame that get cut off.”
Auto-reframe is a crop. Nothing outside the original frame was ever recorded, and no auto-reframe control in Premiere, Final Cut, Resolve or any mainstream web editor invents picture. Generative outpainting does exist as a separate feature in some tools, it is a different operation with different failure modes, and it is not what the auto-reframe button does.
“Auto-reframe is an export setting.”
It is an editorial pass wearing an export setting's clothes. The right answer changes per shot, the crop position is a composition decision, and it survives into every downstream use of the file — thumbnails, someone else's re-crop, a second conversion later. A file that was auto-reframed badly cannot be un-reframed; only the original can be reframed again.
“Tracking is better than a static crop.”
A window that re-centres on a face every frame turns every small head movement into camera movement, which is more distracting than a fixed frame. Good implementations spend most of their effort deciding not to move — AutoFlip holds the window still whenever the subject stays inside half the frame width. A tool that never holds still is not more advanced; it is unsmoothed.
“It picked the wrong person, so the model is bad.”
It picked the largest or most confident face, because face detection answers where the faces are and not who the subject is. The person listening is often nearer the camera than the person speaking. Fixing it needs active-speaker detection — correlating lip motion with the audio — or the manual override every NLE ships for this exact case: Resolve's Reference Point, Premiere's editable keyframes, Final Cut's Transform.
“The reframe worked, so my captions are fine.”
Burned-in text lives inside the picture, so order of operations decides everything. Captions applied before a reframe are cropped or shrunk along with the frame; captions applied after are laid out against the new frame. Text that sat comfortably in a 16:9 frame can land under TikTok's caption block or behind its right-hand action rail in 9:16, because the safe area changed even though the text did not.
“One pass over the whole file and it's done.”
Every serious implementation is per-clip or per-shot, and all of them present the result as a starting point: Premiere writes keyframes you are expected to inspect, Resolve offers a Reference Point for when Auto gets it wrong, Final Cut leaves the Transform open for adjustment. Treating a single automatic pass as final is the most common way a bad reframe ships.
How this works in Valmera
You ask for a platform or a ratio in plain English and the agent converts the output frame to 16:9, 9:16, 1:1, 4:5 or the source ratio. It can crop, pad with bars, pad with a blurred copy, or measure the footage and choose between cropping and the blurred fit. An explicit instruction always wins — asking for a crop gets a crop.
When it chooses, it measures rather than assumes. Five frames are sampled evenly across the kept footage — the current edit, not the raw upload, so framing follows what the viewer will actually see. Faces are detected in the pixels on each sample, and the focus point is lifted by 12% of the face box's height, because portrait framing sits on the eyeline rather than the geometric middle of a head. A face has to appear in at least two of the five samples before it is believed: one detection used to be enough, and a face-shaped patch of a game HUD once aimed an entire 9:16 crop at a corner. The final point is the median of the accepted samples, not the mean, so one wide shot cannot drag the crop off every close-up.
When no face is found — a game capture, a screen recording, a wide scene — the question becomes whether to crop at all, and that is measured too. The agent integrates the picture's gradient energy over the exact window the renderer would take; when less than 55% of it would survive, the whole frame is fitted into the new ratio over a blurred copy of itself instead, and the reply quotes the measured share and says why. Aiming that fallback falls to a vision model asked for a coordinate per frame, or, where no vision model is configured, to the gradient-energy centroid — and the reply names which route was used, so "it is aimed at a measured face" and "it is an estimate" are never the same sentence. Output resolution never exceeds the source's pixel budget: resizing a 1280x720 file to 9:16 renders 720x1280, not an upscale dressed as 1080x1920.
What it does not do. The focus is a single fixed point for the whole video. There is no crop path, no keyframes and no motion tracking, so it will not follow a presenter who walks across a room; when the measured subject drifts more than 0.18 of the frame across the samples, the agent says so and offers the fitted conversion, which is a warning rather than a solution. There is no active-speaker detection and no automatic split-screen, so a two-person interview shot wider than the crop window is a case to fit rather than crop. Reframing applies to the whole video rather than shot by shot — there is a separate operation for shifting the visible aspect mid-video, but you place it deliberately, it is not solved from the footage. There are no arbitrary ratios and no safe-area guides. And padding is bars or a blurred copy of your own picture, never generated content. Burned-in captions are re-laid out against the output frame and held clear of the bottom edge in vertical outputs; the details are in the reframing docs.
Related terms
- Aspect ratio — the shapes auto-reframe converts between, and the pixel arithmetic behind every figure on this page.
- Shot detection — the dependency: reframing is a per-shot decision, so a crop path is only as good as the shot boundaries under it.
- Proxy editing — why the analysis runs on a downscaled copy while the final crop is taken from the original file.
- Burned-in captions vs. subtitles — burned text is inside the frame, so a reframe moves it and a platform's UI can then cover it.
- Edit decision list — where the output frame and its focus point are stored as a reversible decision rather than baked into pixels.
- B-roll — inserted footage that rarely shares the main frame's ratio, and has to be fitted or cropped on the way in.
- All glossary terms
Frequently Asked Questions
Reframe by describing the platform
Upload a landscape edit, ask for 9:16, and the agent measures whether a crop or a fit is the honest conversion — then says which it chose. 50 free credits, no card.
Try it free →