Frame-by-Frame Redaction: Filling the Gaps Between Analyzed Frames

An analyst who has keyframed a face by hand through a two-minute clip knows that frame-by-frame redaction seldom means drawing a box on every frame. It means placing boxes on some frames and trusting something, a keyframe curve or a tracker, to fill in the rest. That filling is one of the failure families in our guide to video redaction best practices. How it works decides whether a face slips out at the edge of a turn, flashes into view at an occlusion, or vanishes under a smear that withholds half the scene.

Keyframing by hand is slow, and automated tracking changes who fills the gaps more than whether gaps exist. The rule defended here is that when a mask could be in two places it should cover both, and it comes with a limit on how far over-coverage can go.

Where the gaps between analyzed frames come from

Every method of video redaction leaves gaps between the frames where a subject's position is actually known. An analyst working by hand sets the mask's position on keyframes and lets the editing software move it in between. That is slow precisely because every turn, stop or change of pace needs another keyframe to keep the mask on the face.

Automated redaction has gaps of its own, and they come from several directions at once. A detector can miss a frame where the face is blurred by motion or half hidden. Some systems read on-screen text at a lower rate than they look for faces, which leaves longer gaps for text masks to bridge, and the problem grows when the text scrolls. Text is also less forgiving than a face, since a single character left uncovered at the edge of a box can be read outright, where a sliver of a face usually cannot be recognized.

The frames being written can also carry timing that differs from the frames detection saw, which our article on variable frame rate video explains. Short gaps still matter, because a person who crosses the full width of the frame in two seconds moves about a sixtieth of the width on every frame at 30 frames per second. A gap of ten frames lets that person travel a sixth of the frame, far enough to leave any mask that stayed where it was.

Straight lines, and where they slip

The simplest way to fill a gap is to move the box in a straight line from its position on one known frame to its position on the next. That works while a subject moves steadily, and it fails when the subject turns, stops or speeds up in between, because the straight-line box lags behind the real path.

A straight-line box fails quietly when a subject turns between two frames where its position is known. The interpolated box is shown for the whole interval and trails the turn, so the edge of the face slips out of it for part of the interval. Nothing looks wrong at normal speed, and only stepping frame by frame through the turn shows the exposed edge.

VIDIZMO Redactor closes that gap by drawing, for each interval between two known positions, the box that encloses both the start position and the end position. The subject is then covered wherever it went in between, at the price of a mask somewhat larger than the face in the middle of the interval. On screen the mask widens a little during a fast move and returns to the size of the face once the subject settles. Over longer gaps with curved or erratic motion even the enclosing box is a guess, which is one reason gaps should stay short and review should look hardest where motion is fastest.

A worked example with round numbers shows what the enclosing box costs and what it buys. Take a face 80 pixels square that sits at one position on a known frame and 40 pixels further right on the next known frame, ten frames later. The straight-line box moves 4 pixels a frame. If the subject made most of that move in the first three frames, the box trails by more than 20 pixels at frame three and uncovers over a quarter of the face's width. The enclosing box for the interval is 120 pixels wide and 80 tall, half again as wide as the face, and it covers every position between the two.

Analysts keyframing by hand can apply the same idea without any change of software. Placing keyframes at the moments a subject turns, stops or starts, instead of at even intervals, puts the known positions where straight lines fail, and widening the box slightly across a turn covers the rest.

Gaps left by lost tracking

People drop out of tracking for reasons that have nothing to do with interpolation, most often by walking behind something, and the frame before such a gap and the frame after it both need masks. Our article on tracking through occlusion explains why those edge frames are easy to leave bare, and how trackers and reviewers handle the gap itself. Analysts working by hand can cover the same edges by setting a keyframe on the last frame the face is visible and another on the first frame it returns. Leaving the mask switched on through the frames where the face is hidden does the rest. Keeping a mask on over a parked van costs the requester nothing, whereas switching it off one frame early can expose the face as the person steps out.

Padding is a common answer at the ends of a track, extending a mask briefly before the tracker picks a subject up and after it loses them, and holding the last known box in place. Because the held box stays put while a quick subject moves on, padding should cover only the moment of pickup or loss. How much padding is enough depends on how fast the subject moves, and no single setting suits every recording, so it is a judgment to check on real footage. A padding rule written in frames also changes meaning with the frame rate. Ten frames last a third of a second at 30 frames per second, a sixth at 60 and two thirds of a second in a CCTV export recorded at 15.

When over-coverage goes wrong

Covering both positions errs toward safety within an interval, and over-coverage can still go too far. One failure mode applies padding meant for the ends of a track to every frame's box instead, so each frame draws every box inside the padding window. The mask becomes a band smeared along the subject's whole path, and on a face moving across the frame it can come out nearly three times as wide as the unpadded mask.

Redactor pads only the two ends of each continuous run and holds the boundary box there, so padding covers the pickup and the loss and nothing in between. A smeared band is easy to spot once you know to look for it, and hard to notice when you are checking whether the face is covered. In selective redaction, where some people must stay visible, an oversized mask can also swallow the person who is meant to be seen, the situation our article on selective video redaction describes. Beyond the subject's possible path, any extra coverage withholds part of a record the requester is owed under the disclosure rules our guide quotes.

Two safety rules at the edges of the arithmetic

The step that fills gaps works against a list of frame times, and a missing or empty list is a failure mode to plan for. With nothing to interpolate against, a naive step produces no positions at all, and the video goes out without masks and without an error, so nothing prompts anyone to look. When the timing needed to fill gaps is missing, Redactor masks every raw detection on its own frame instead of none. Masks drawn only on the detected frames are imperfect, but a reviewer can see them and they cover the faces that were found. A quietly empty result would reach a requester with nothing masked at all.

Box edges rarely fall exactly on pixel boundaries, so every mask has to be rounded to whole pixels somewhere along the way. Rounding inward can leave a fringe of the face unmasked along the edge of the box, visible to anyone who zooms in. Redactor rounds outward, so rounding can only ever grow a mask. The difference is a single pixel along each edge, which matters most on small masks such as a distant face or a line of text.

Checks for the frames in between

Each failure above leaves a trace that a reviewer can find by stepping through the right frames, whichever tool produced the redaction. What fills each box, whether blur, pixelation or a solid fill, is a separate choice that our article on which mask leaves nothing behind compares.

For an acceptance test, run the same checks on a short set of clips with known turns, occlusions and track ends, and repeat them after every software update, since gap handling can change between versions. Keeping the clips and the results together makes the checks repeatable from one release to the next. Stepping through a turn goes fastest with the view zoomed on the mask's leading edge, watching for skin, hair or an ear outside the box rather than for the face as a whole.

For the same rules applied to body-worn camera footage from upload to release, see body camera redaction with Redactor.