If you train a policy on captioned video, you are trusting the captions. We took 240 of them and checked each one against the footage. The question is simple. Are they right?
About a third were not. The misses cluster in the verbs, which is the part a manipulation policy is supposed to learn. A calibrated judge can route human attention to those spans. It cannot replace the human. That combination is what we put behind every delivery.
The setupWhat we audited, and how
The judge sees only the short clip for each span and answers one question: does the footage support this caption? Verdicts are split across scene, objects, and action. The human labels are the ground truth for both questions: how often the captions hold up, and how well the judge agrees.
Finding 01About one in three captions is wrong
Roughly a third of the captions did not match the footage, and the defect rate varied about 4× between the cleanest and worst episodes. The dataset is uneven: some episodes are nearly fine, others need real attention.
Scene and objects are usually fine. The verb is what fails. The caption says "pull the chair away" and nothing moves. Sometimes the action almost happens. Sometimes it happens to a different object.
The "where" and "what" can mostly be trusted. The "doing" cannot.
If you are training a manipulation policy, that is the part it is supposed to learn.
Finding 02An AI judge is precise but incomplete
Against the human labels, the judge is lopsided. Precision ~90%: when it flags a caption, a human agrees 9 times out of 10. It does not cry wolf much. Recall ~50%: it catches about half of the bad captions. The ones it misses are the subtle "almost happens" cases, which is also where most of the defects live.
That changes how you use it. As a triage tool, it works: send reviewers to flagged spans, rank the hardest episodes first, stop watching everything. As an automatic gate, it is not enough. No flag does not mean clean. The labels that slip through are the verbs a policy would learn from.
Good enough to make a reviewer several times faster. Not good enough to replace one.
The calibration arcWhat it took to make the judge trustworthy
A judge is only useful if it is calibrated. We ran the same 18 spans through four versions. The flag rate went 28% to 50% to 22%:
| Ver | Evidence | Verdict | Flag rate | Lesson |
|---|---|---|---|---|
| V1.0 | 2 still frames | agree y/n | 28% | Stills can't see motion. "Pick" and "place" are motion verbs; two frames conflate "can't see it happen" with "wrong." |
| V1.1 | native clip @ 4fps | 3-way | 50% | Video over-flags manner. "Linear rather than circular motion" is not a defect; neither is finishing just past the span boundary. |
| V1.2 | native clip @ 4fps | content-scoped | 22% | Scope to content. Core scene, objects, and action category; explicitly ignore manner and completion. |
| V1.3 | + 3-vote majority | content-scoped | 22% · unanimous | Borderline verdicts flip. A flag needs a strict majority; after voting, every verdict was unanimous. |
Each step fixed one failure we could reproduce. Stills miss motion. Raw video over-flags manner. Scope and a majority vote make the flag something a reviewer can trust.
What you getA QC report with every delivery
- Audit before you train. A third of spans mislabeled, concentrated in verbs, is enough to distort a policy. The check is what makes the hour worth training on.
- Defect rates per episode, not a dataset-level claim. A 4× spread between episodes means an average hides the hard hours.
- A judge for triage, a human for the number. The judge is precise enough to route attention and not complete enough to replace it. That is the stack behind a Synjuku QC report.
Every delivery includes that report. Labels are checked against a human gold set, and anything we cannot confirm comes out before it ships. The judge makes reviewers faster. The humans make the numbers trustworthy.
Numbers in this post come from a 240-span human-labeled sample over 20 episodes, single annotator. We're expanding the gold set with a second rater; treat the dataset-level rates as a strong warning sign, not a final measurement.