Evaling Video Slop — Maor Bril, Character.ai
AI Engineer2 momentsb_PmGocP4rc
- 4:42spokenbenchmark on how we test video that we can rerun over and over and over again. So that combines both metrics, as I said earlier, that knows how to view individual frames, but also consistent LLM as a judge, right, where we also use human annotation to calibrate the LLM as a judge. So for every report that we generate with that harness, we're able to have humans annotated and basically feed that fe…youtu.be/b_PmGocP4rc?t=285 ↗
- 18:02spoken…ideos and games and books. Fair. So how would you construct and align sort of like any human judges? Yeah. So this is actually solved at first at the Judge Judy part. where every report it'll generate, a human can go and annotate it. And we actually, we do that. We periodically have sessions where everyone spends 10 to 15 minutes just annotating videos. And that usually happens on, multiple axes. …youtu.be/b_PmGocP4rc?t=1113 ↗