Everyone training robot policies on captioned video is trusting the captions. The past few months have produced good benchmarks for whether AI can write those labels (WGO-Bench from Macrodata Labs: best end-to-end F1 of 0.168) and whether VLMs can understand procedural video (VidWork-Bench from Datoric: top composite score of 0.446). Both point the same direction: models aren't ready to generate labels unsupervised.
Nobody has published on the third question, which is the one that matters if you're buying or training on labeled video today: the labels already exist. Are they true?
We run quality control on egocentric manipulation data for robotics-model labs, so we measured it.
The setupWhat we audited, and how
The judge pulls only the short clip for each span (never the full 2 GB episode: ffmpeg range-requests against presigned URLs) and answers one scoped question: does the footage support this caption? Verdicts are split across scene, objects, and action. The human labels give us ground truth for both questions: how bad are the captions, and how good is the judge?
Finding 01About one in three captions is wrong
Roughly a third of the captions did not match the footage, and the defect rate varied about 4× between the cleanest and worst episodes. The dataset is uneven: some episodes are nearly fine, others need real attention.
The errors are not where you'd guess. Scene and objects are usually right. The verb is what fails. The caption says "pull the chair away" and nothing moves. The action almost happens, or happens differently, or happens to a different object.
The "where" and "what" can mostly be trusted. The "doing" cannot.
That pattern should worry anyone training manipulation policies, because the verb is the part the policy is supposed to learn.
Finding 02An AI judge is precise but incomplete
Against the human labels, the calibrated judge shows a sharp asymmetry. Precision ~90%: when it flags a caption, 9 times out of 10 a human agrees. It rarely cries wolf. Recall ~50%: it catches about half of the bad captions, and the misses are the subtle ones, the same "almost happens" action errors that dominate the defect distribution.
So the deployment guidance is asymmetric too. As a triage tool, it works today: point reviewers at flagged spans, rank episodes worst-first, stop watching everything. As an automatic gatekeeper, it is not ready: "no flag" does not mean "clean," and a pipeline that auto-approves unflagged data will ship bad verbs.
Good enough to make a reviewer several times faster. Not good enough to replace one.
The calibration arcWhat it took to make the judge trustworthy
The interesting engineering is not the final prompt, it's the failure modes on the way there. We ran the same 18 spans through four versions of the judge, and the flag rate moved from 28% to 50% to 22% for reasons worth knowing:
| Ver | Evidence | Verdict | Flag rate | Lesson |
|---|---|---|---|---|
| V1.0 | 2 still frames | agree y/n | 28% | Stills can't see motion. "Pick" and "place" are motion verbs; two frames conflate "can't see it happen" with "wrong." |
| V1.1 | native clip @ 4fps | 3-way | 50% | Video over-flags manner. "Linear rather than circular motion" is not a defect; neither is finishing just past the span boundary. |
| V1.2 | native clip @ 4fps | content-scoped | 22% | Scope to content. Core scene, objects, and action category; explicitly ignore manner and completion. |
| V1.3 | + 3-vote majority | content-scoped | 22% · unanimous | Borderline verdicts flip at temperature 0. A flag needs a strict majority; after voting, every verdict was unanimous. |
Each step fixed a specific, reproducible failure mode. Anyone building a VLM-as-judge pipeline over video will hit all three.
CostIt's cheap
About $3 per hour of footage to check every span (roughly half with batch inference); $0.003 per span single-pass, $0.01 with a 3-vote majority. At these prices, the constraint on dataset quality is not money. It's whether anyone runs the check.
ImplicationsIf you buy or sell labeled video
- Audit before you train. A third of spans mislabeled, concentrated in verbs, is enough to distort a policy. A few dollars of judging per hour of footage is cheap insurance against thousands of GPU-hours on wrong labels.
- Demand defect rates per episode, not dataset-level claims. A 4× spread between episodes means averages hide the problem. Quality is a distribution.
- Treat AI verification as triage, not certification. Today's judges are precise enough to route human attention and nowhere near complete enough to replace it. Any vendor claiming fully automated label QC is shipping the misses.
This is also, transparently, how we operate: every Synjuku delivery ships with a QC report, labels are verified against calibrated human gold sets, and labels that can't be confirmed are removed rather than shipped. The judge makes our reviewers faster; the humans make the numbers trustworthy.
Numbers in this post come from a 240-span human-labeled sample over 20 episodes, single annotator. We're expanding the gold set with a second rater; treat the dataset-level rates as a strong warning sign, not a final measurement.