How RedFlag scores a video

Last Updated: September 4, 2026

RedFlag turns a video into a single binary verdict per moment — 🟢 Full Harmony or 🔴 Red Flag. This page describes exactly how, with the real numbers. We publish the method because a behavioral-signal tool that hides its method cannot be questioned, and being questioned is the point.

1. Deterministic by construction

The analysis is a property of the video, not of the playback session. The timeline is cut into fixed 0.5-second slots; raw video frames are pooled into fixed 100-millisecond media-time buckets. Each slot's metrics are computed once from those buckets and then frozen, so a full watch, a pause, a seek, or a re-run of the same video produces the identical result. When the video stream is directly fetchable, RedFlag analyzes the stream itself at fixed sampling points, so two runs process byte-identical input.

2. What is measured

Thirteen signals are tracked and shown on the dashboard: blink rate, gaze stability, head stability, brow furrow, jaw tension, smile authenticity, voice volume, pitch frequency, verbal stability, vocal activity, speaking rate, filler ratio and filler count. Face landmarks come from MediaPipe's face landmarker; pitch and voicing come from the audio track.

Only three signals decide the verdict. Every candidate signal was tested against hand-labeled ground truth (see §6), and most were rejected — including some that look intuitively compelling. The dashboard's facial composite, for example, measured an AUC of 0.43 against the labels (below chance) and is therefore display-only, never a verdict input.

3. The gates — one speaker, speaking clearly

A moment is only scored when the footage can honestly support a measurement. A slot is left unscored (no verdict, no color) unless:

• a face is detected, at landmarker confidence ≥ 0.35;
• the face fills at least 10% of the frame height (podium and parliament screen captures pass);
• no second comparable face is in frame (a second face ≥ 55% of the subject's size and ≥ 13% of frame height gates the shot as multi-person);
• the surrounding 3-second window contains any voiced speech at all.

The blink channel keeps a stricter bar: eyelid blendshapes cannot be resolved on small faces, so blink data is recorded only when the face is at least 22% of the frame height. Below that, the verdict rides on the pitch signal alone, which is audio-based and does not degrade with face size.

4. The three verdict signals

The verdict measures how far the speaker sits outside normal speaking behavior:

1. Pitch monotony (absolute norm, weight 0.75). Normal conversation carries a pitch variability of roughly 2.3–4.5 semitones (standard deviation of F0, centered ±5 s window). Flagged moments are consistently flattened below that band. Deviation grows linearly the further the value sits outside 2.3–4.5 st.
2. Blink elevation (person-relative norm, weight 0.25). Blink rate over a centered 20-second window is divided by that speaker's own median rate across the whole video. Unflagged stretches sit near 0.88–0.98× the personal baseline; flagged moments at 1.3–2×. The deviation activates above a knee of 1.3× baseline. There is no absolute blinks-per-minute threshold, because baseline blink rates differ several-fold between people.
3. Voiced-continuity elevation (person-relative, additive). Flagged stretches are often more continuously voiced than the speaker's own norm — the speaker monologues through the tough spot. Voiced fraction over a centered ±2.5-second window is divided by the video's median, with the same 1.3× knee.

The fusion:

deviation = min(10, 0.75·pitchDev + 0.25·blinkDev + voicedDev)
RED ⇔ deviation > 2.5

The scorer is pitch-anchored: with no pitch data there is no verdict at all; with no blink data the pitch signal carries alone; with no voiced-continuity data that term contributes zero. No hard threshold separates "computed" from "default" anywhere in the pipeline — every gate is a ramp, so capture jitter cannot flip a verdict.

5. Smoothing and episodes

• Each slot's score is smoothed by a trailing median over 7 slots (~3.5 s).
• Hysteresis of 0.2 deviation units: to leave the current verdict band, the smoothed score must cross the cutoff by that margin. This kills flicker at the boundary.
• Flag episodes shorter than 3 seconds are dropped as measurement blips; same-color episodes separated by 3 seconds or less are merged.
• Unscored (gated) footage gets no color at all — RedFlag never holds a verdict over footage it did not analyze.

6. How it is tested

Every change to the scorer must pass a reference-label regression test before shipping. Four reference videos (of 15 labeled in total) were hand-labeled moment by moment; the current scorer's per-video results (alert recall / green precision) are 21/95, 47/88, 96/91 and 89/100. New signals are admitted only under a strict-dominance rule: across all labeled videos, no video may lose recall or precision compared with the shipped scorer, verified with leave-one-video-out calibration. The most consistent single signal we ever measured (a pitch-level drop, directionally consistent on 13 of 15 labeled videos) was still rejected under this bar.

7. Limitations

AI systems are probabilistic. RedFlag does not guarantee that every relevant signal is detected, that every detected signal is meaningful, or that results are comparable across recordings. Recording quality, lighting, microphone quality, compression, language and individual differences all affect measurement. RedFlag signals must not be treated as evidence of deception, credibility, intent, health, personality, or guilt, and the Terms of Service prohibit high-stakes use (hiring, legal, security, medical). Currently the analyzer supports a single speaker in frame.