Videos IJXjTLPzvAU
The Miranda Hypothesis: How Hamilton Poisoned Persona Evals - Jacob E. Thomas, Results Gen
Scene timeline
137 shot(s).
keyframes kept every frame deduplicated
What was stored
- cues
- 570
- whisperx 570
- chunks
- 103
- from 570 cues
- keyframes
- 49
- kept of 137 captured
- frames with text
- 49
- 833 lines read
- chapters
- 0
- from the source metadata
- keyframe bytes
- 12.8 MB
- word timings on 570 cues
Provenance
| stage | state | model | started | took |
|---|---|---|---|---|
fetch |
done | — | 2026-08-11 10:17 | 1m 40s |
stt |
done | — | 2026-08-11 10:18 | 53s |
chunk |
done | — | 2026-08-11 10:19 | 0s |
text_embed |
done | — | 2026-08-11 10:19 | 1s |
keyframe |
done | — | 2026-08-11 10:19 | 5m 19s |
ocr |
done | — | 2026-08-11 10:25 | 24s |
frame_embed |
done | — | 2026-08-11 10:25 | 15s |
Frames, and what the machine read
-
- The Miranda Hypothesis1.00
- How Hamilton (the Musical)1.00
- Poisoned Your Persona Evals1.00
- AI ENGINEER WORLD'S FAIR· 20260.96
- Jacob E. Thomas, PhD . with Rick Halpern, University of Toronto0.99
- archival & theological review . Shawn Martin, Washington College0.97
-
- (character.ai)1.00
- Plan a trip1.00
- Write a story1.00
- Play a game0.96
- Help me make0.98
- with Trip Planner1.00
- with Creative Helper0.99
- with Space Adventure Game0.99
- with DecisionHel1.00
- Create1.00
- Voices1.00
- Discover1.00
- Indian Girl Accent1.00
- Robot1.00
- The Narrator0.97
- Galen1.00
- A Hindi girl's accent0.99
- Just a robot :)0.99
- Illuminates dark1.00
- Feed1.00
- wisdom, lore an0.97
- Charms1.00
- Filter by category1.00
- é0.77
- Labs1.00
- Anime1.00
- Assistant1.00
- Creative1.00
- Family0.97
- Fantasy1.00
- Gaming1.00
- History1.00
- Human1.00
- Humor1.00
- Learning1.00
- Lifestyle1.00
- Mafia1.00
- Powerful1.00
- Q0.93
- Search1.00
- EPIC - Hermes0.98
- EPIC - Odysseus0.96
- Telemachus1.00
- Odysseut0.99
- By @demonicghoul0.99
- By (@aeternumus0.97
- By (@CharacterUser170371480.0.98
- By (@Chara0.94
- Today1.00
- wouldn't you like?1.00
- * you're far too gullible.0.96
- You stand up for him against0.98
- the suitors1.00
- | Full Spe0.91
- Odysseus1.00
- O1.1m0.85
- 8.2m1.00
- 4.4m0.90
- 2.1m1.00
- ALEXANDER HAMILTOI -..0.90
- Try saying1.00
- A While Ago1.00
- The founding fathers1.00
- @landon • 210.3m interactions · 98.0k likes0.95
- Character Assistant1.00
- WhoWouldWin1.00
- @greg - 39.4m interactions - 12.2k likes0.96
- Elon Musk1.00
- @Bolonwhisperer - 46.8m interaction0.96
- What type of fish is Dory from Finding Nemo?0.99
- Batman vs. Superman0.98
- Why did you buy twitter?0.99
- Create an ad campaign for a new video game1.00
- Knight vs. Samurai0.99
- What do you think about Jeff Bezos' Blue0.99
- Decide between the Macbook Air and Macbook Pro1.00
- Lebron James vs. Michael Jordan1.00
- If you could time travel, when/where wou0.99
- Your Privacy Choices0.99
- Upgrade to (c.ai+)1.00
- ImpromptuDhole92331.00
-
- ELLO1.00
- HISTORY1.00
- FOR EDUCATION0.98
- FOR RESEARCH0.98
- FOR EXHIBITIONS0.99
- ABOUT US0.96
- CONTACTS1.00
- Chat With Anyone From1.00
- The Past1.00
- Marcus Aurelius0.99
- An Al powered app that lets you have life-like conversations with1.00
- historical figures.1.00
- AppStore0.98
- For schools and institutions →0.97
- Powered by:humy0.93
-
- BEFORE THE FORMAL TALK0.98
- A GROUNDING0.98
- What is a role-playing language0.99
- agent?1.00
- Character.ai1.00
- Hello History1.00
- companion apps1.00
- AI tutors · historical figures0.98
- LIVE DEMO0.98
- A0.90
-
- HASNTHGTON0.63
- COL. RANILTOM0.86
- NR. JEFFERSON0.77
- DR. FRAERLTH0.75
- GEORGE MASHINGTON0.92
- ALEXANDER HAMILTON0.91
- BENJAMIH FRAHNLIN0.74
- Sessings0.86
- THE COMMITTEE1.00
- This0.93
- Export this transcript 40.97
-
- WE BUILT BENCI0.96
- These systems are deployed and1.00
- used for things that matter. So1.00
- we measure them. The question0.99
- that leads this talk is — what is0.99
- the eval actually measuring?1.00
-
- YOUR EVAL PIPELINE REPORTS1.00
- 80.7%1.00
- personalityfidelity1.00
- InCharacter benchmark · state-of-the-art role-playing agents0.97
-
- WHO'S TELLING YOU THIS0.99
- Epidemiologist1.00
- Data Scientist1.00
- AI Engineer1.00
- One question — how information0.98
- environments shape populations — from three0.99
- Jacob E. Thomas1.00
- sides. The humanities are the instrument the1.00
- PhD · UT Austin | MA . Columbia0.92
- engineering is missing.0.99
- THE DOMAIN VOICES WHO HOLD THE WORK TO THE RECORD1.00
- Rick Halpern1.00
- Shawn Martin0.99
- Historian - University of Toronto0.99
- Librarian & Information Scientist · Washington College0.98
-
- THE FAILURE IS INVISIBLE0.98
- First, credit where it's due. The0.99
- field has made real progress.1.00
- STAGE 010.98
- STAGE 020.99
- STAGE 030.98
- Templates1.00
- Style Imitation1.00
- Cognitive1.00
- Simulation1.00
- Canned responses keyed to1.00
- Voice, cadence, characteristic1.00
- inputs.1.00
- tics.1.00
- Personality models, memory,1.00
- motivation-situation chains.0.99
-
- Be precise about what that number measures.0.99
- IT MEASURES0.97
- IT DOES NOT MEASURE0.98
- Can the model reproduce his1.00
- Can it constrain him to his record,1.00
- personality profile?1.00
- at a moment in time?1.00
- His Big Five. His register. His motivational0.99
- What the figure could have known, believed, or0.99
- architecture.1.00
- argued at this point in his life.0.99
- 111.00
- 600.99
Transcript
570 cues· 6,975 words· 41,949 chars
- 0:05 Before I begin my formal talk, I want to show you something just so we're all on the same page about what we're even talking about.
- 0:16 This is a platform called Character AI.
- 0:20 It's a hybrid social media platform with role-playing language agents.
- 0:26 This is Hello History.
- 0:27 It's a more education-focused one where you can summon a persona such as Marcus Aurelius and be tutored by them.
- 0:36 Millions of people open these tools and have conversations with Napoleon, Cleopatra, or Marcus Aurelius, as you saw, with a fictional companion or with a tutor wearing a historical face.
- 0:49 The technical name for what's underneath these tools is role-playing language agent, a system built to instantiate a persona, real or invented, and reason and speak as them.
- 1:02 Yes, it's entertainment and it's companionship, but increasingly it's being proposed as civic and pedagogical infrastructure.
- 1:16 And here's one more.
- 1:17 This one's mine.
- 1:20 This is a frontier model, Claw Opus 4.7, same one you use, running an open source prompt framework that I built called Companion.
- 1:32 In this particular example, I summoned a collection of founding fathers and set them in a room with the Epstein files.
- 1:42 I asked them to counsel the soul of America.
- 1:45 That demo is live on our site if you want to play with it.
- 1:50 But I want to be clear that this is one of many attempts to do persona instantiation well.
- 1:57 The companies building the systems I just showed you have their own.
- 2:02 Mine is not better by default.
- 2:04 The one thing it is is open.
- 2:07 You can read every line of what shapes the persona.
- 2:15 I ask my companion system a real question that's highly relevant to the current sociopolitical moment.
- 2:22 And this is the exact question we'll come back to near the end of the talk.
- 2:26 So sit with it.
- 2:28 I instantiated Abraham Lincoln and I asked him, under what circumstances may a president take the country to war without Congress?
- 2:38 And here's what came back.
- 2:41 While Congress holds the power to declare war, the president as commander in chief possesses inherent executive authority to act decisively in moments of national emergency.
- 2:53 The executive must respond to the threats with the energy and dispatch the office requires.
- 2:59 And history has vindicated those who acted to preserve the union when circumstances demanded it.
- 3:07 This is a good answer.
- 3:09 It's fluent and it's plausible and it sounds like Lincoln.
- 3:13 You can replicate this exact exercise and I encourage you to.
- 3:17 The answers vary often, but the thesis rarely does.
- 3:24 So these systems are real.
- 3:26 They're deployed and they're being used for things that matter.
- 3:30 And our discipline did what our discipline does.
- 3:34 We built benchmarks.
- 3:35 We built evaluations.
- 3:38 We measure these things now rigorously at scale.
- 3:43 And that's exactly where this talk begins with a simple question that I think is profoundly under asked.
- 3:51 And I'll warn you now that this talk poses many more questions than it does answers.
- 3:56 But that principal question is this, what is the eval actually measuring?
- 4:04 And that's the formal talk.
- 4:06 Let me begin.
- 4:11 The in-character benchmark, which is a gold standard in the field, evaluates personality fidelity in RPLAs, and it reports state-of-the-art systems hitting 80.7% alignment with human perceived personalities of that target character.
- 4:28 80%.
- 4:30 It sounds like a passing grade, but here's the problem.
- 4:34 When the character is Alexander Hamilton, the same high scoring system is also rendering a Hamilton who sounds like he's read his own Broadway musical.
- 4:47 This is the full thesis.
- 4:49 If a dominant failure mode is anachronistic compositing and your evals measure fluency and personality consistency, then your evals cannot detect the dominant failure.
- 5:03 I want you to hold on to that for the next half hour.
- 5:06 Everything I show you is an argument that this is true, structurally, architecturally, and measurably.
loading