vidtheque.

read-only demo

Videos m24UKZomm7k

Don't Let the LLM Drive - Ornella Bahidika & Joel Allou, Microsoft

index_state ready data_status ok

AI Engineer· published 2026-07-20· 0:06:07· en-US· indexed 2026-08-10 19:48

Open on YouTube

Scene timeline

  1. Shot 0, 0:00 to 0:12, 1 of 1 keyframes kept
  2. Shot 1, 0:12 to 0:28, 1 of 1 keyframes kept
  3. Shot 2, 0:28 to 0:38, 1 of 1 keyframes kept
  4. Shot 3, 0:38 to 0:54, 1 of 1 keyframes kept
  5. Shot 4, 0:54 to 1:25, 1 of 1 keyframes kept
  6. Shot 5, 1:25 to 1:51, 1 of 1 keyframes kept
  7. Shot 6, 1:51 to 2:16, 0 of 1 keyframes kept
  8. Shot 7, 2:16 to 2:42, 0 of 1 keyframes kept
  9. Shot 8, 2:42 to 3:08, 0 of 1 keyframes kept
  10. Shot 9, 3:08 to 3:33, 0 of 1 keyframes kept
  11. Shot 10, 3:33 to 3:59, 1 of 1 keyframes kept
  12. Shot 11, 3:59 to 4:26, 0 of 1 keyframes kept
  13. Shot 12, 4:26 to 4:53, 1 of 1 keyframes kept
  14. Shot 13, 4:53 to 5:19, 0 of 1 keyframes kept
  15. Shot 14, 5:19 to 5:59, 1 of 1 keyframes kept
  16. Shot 15, 5:59 to 6:07, 1 of 1 keyframes kept

16 shot(s).

keyframes kept every frame deduplicated

What was stored

cues
57
whisperx 57
chunks
11
from 57 cues
keyframes
10
kept of 16 captured
frames with text
10
147 lines read
chapters
0
from the source metadata
keyframe bytes
1.1 MB
word timings on 57 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage state model started took
fetch done — 2026-08-10 04:37 1m 48s
stt done — 2026-08-10 04:38 6s
chunk done — 2026-08-10 04:39 0s
text_embed done — 2026-08-10 19:48 0s
keyframe done — 2026-08-10 04:39 16s
ocr done — 2026-08-10 04:39 3s
frame_embed done — 2026-08-10 19:48 2s

Frames, and what the machine read

  • 0:09 #0 done9 line(s)

    shot 0·sharpness 798.9

    1. A C E0.85
    2. Joel Allou0.99
    3. HARNESS ENGINEERING1.00
    4. Don't Let the LLM Drive.0.99
    5. Ornella Bahidika0.99
    6. Joel Allou0.98
    7. PRODUCT1.00
    8. ENGINEERING1.00
    9. aceactprep.com1.00

  • 0:23 #1 done9 line(s)

    shot 1·sharpness 820.6

    1. THE FELT PROBLEM0.99
    2. Joel Allou0.99
    3. It ended the lesson halfway0.98
    4. through.0.97
    5. In the demo0.98
    6. In production1.00
    7. Runs clean, start to finish.0.99
    8. Declares itself done. Skips. Loops.0.99
    9. Demos hide it. Real users find it.0.98

  • 0:37 #2 done8 line(s)

    shot 2·sharpness 692.2

    1. THE TRAP0.94
    2. Joel Allou1.00
    3. rety0.79
    4. plateau1.00
    5. Just prompt it1.00
    6. harder.1.00
    7. more prompt rules →0.98
    8. Reliability was never a prompting problem.0.99

  • 0:49 #3 done7 line(s)

    shot 3·sharpness 1180.4

    1. THE REFRAME0.98
    2. Ornella Bahidika0.99
    3. Joel Allou0.98
    4. The model is the talent. The0.99
    5. harness is the director.1.00
    6. Brilliant at delivering a line. Terrible at remembering it's on step three of six. So0.99
    7. we stopped asking it to.0.99

  • 1:01 #4 done15 line(s)

    shot 4·sharpness 1300.8

    1. THE ARCHITECTURE1.00
    2. Ornella Bahidika0.98
    3. Joel Allou0.95
    4. A lesson is a small state machine. The harness drives it.1.00
    5. intro1.00
    6. teach1.00
    7. check1.00
    8. grade1.00
    9. advance1.00
    10. wrap1.00
    11. advance n loops back to teach if not mastered0.98
    12. HARNESS - MODEL0.99
    13. MODEL - HARNESS0.98
    14. Narrow contract: do this one thing, return that.0.98
    15. Returns language. Never decides where we are.1.00

  • 1:35 #5 done20 line(s)

    shot 5·sharpness 856.2

    1. THE CATCH1.00
    2. LIVE1.00
    3. English Foundations: Parts of Speech, Sentence Structure, and Verb Tense1.00
    4. CH. GETTING STARTED S0.95
    5. LIVE1.00
    6. □ END0.87
    7. 回∅÷0.60
    8. top0.88
    9. @0.83
    10. hamnes0.85
    11. Default levele0.89
    12. issues:1期22 2 hiden0.91
    13. Joel Allou1.00
    14. cmd to turn on code0.82
    15. suggestions. Oon't_ show again0.92
    16. HEY JORDAN0.99
    17. Warming up Pax...0.95
    18. Type a question1.00
    19. Tap mic to talk, or type bolow0.94
    20. The model proposes. The harness decides.1.00

  • 1:59 #6 skipped

    shot 6·duplicate of #5

  • 2:22 #7 skipped

    shot 7·duplicate of #5

  • 2:48 #8 skipped

    shot 8·duplicate of #5

  • 3:30 #9 skipped

    shot 9·duplicate of #5

  • 3:56 #10 done53 line(s)

    shot 10·sharpness 1288.2

    1. THE1.00
    2. CATCH1.00
    3. LIVE1.00
    4. Ornella Bahidika1.00
    5. English Foundations: Parts of Speech, Sentence Structure, and Verb Tense1.00
    6. CH. GRAMMAR FOUND...0.97
    7. TEACHING1.00
    8. END0.90
    9. ∅÷0.69
    10. top0.93
    11. @0.94
    12. hames0.93
    13. Default levels0.92
    14. 60.86
    15. 00.51
    16. a0.55
    17. 80.54
    18. Today we're building your grammar0.99
    19. Hey Jordan, welcome back.0.99
    20. 5issues:122 2hidden0.87
    21. [harness] signel = section0.96
    22. sse-livekit-lesson,ts: 3780.79
    23. Joel Allou0.95
    24. tooit for the A C T. We'l start0.80
    25. with the eight parts of speech-the0.98
    26. Sentence Structure, and verb0.90
    27. building blocks of every sentence.0.99
    28. Today: Parts of Speech → Sentences → Verb Tense0.98
    29. Got it. Before we wrap. let me ask:0.99
    30. ready to end the lesson0.99
    31. [harness signal + claar_queue0.90
    32. (types 'clear_queue')0.80
    33. ase-livekt-lesson,tsi3780.63
    34. The A C T tests grammar through reading passages. You'll identify errors ir0.98
    35. isn't working0.96
    36. are you feeling okay, or is there1.00
    37. something about the lesson that1.00
    38. ( (0.55
    39. (type: 'clar_quee')0.54
    40. from the milky. We can wrap the0.97
    41. cad i to turn on code0.84
    42. suggestions, Oon't show again0.89
    43. No problem, Jordan. Youve got a0.99
    44. solid foundation to build on with1.00
    45. grammar—keep working through0.99
    46. those parts of speech and sentence0.99
    47. structure, and you'll spot errors on0.99
    48. Type a question1.00
    49. 100%1.00
    50. ①0.65
    51. 00:341.00
    52. -00:090.98
    53. The model proposes. The harness decides.0.98

  • 4:18 #11 skipped

    shot 11·duplicate of #5

  • 4:42 #12 done10 line(s)

    shot 12·sharpness 461.7

    1. WHAT THE LLM SHOULD NEVER OWN0.98
    2. Ornella Bahidika0.99
    3. Joel Allou0.96
    4. Is the lesson done?0.99
    5. 011.00
    6. Did theygetitright?0.99
    7. 021.00
    8. What comes next?0.99
    9. 031.00
    10. These belong to code. In any flow agent.1.00

  • 5:01 #13 skipped

    shot 13·duplicate of #12

  • 5:35 #14 done6 line(s)

    shot 14·sharpness 749.7

    1. THE RULE0.95
    2. Ornella Bahidika1.00
    3. When reliability is a coin flip, take1.00
    4. control flow out of the model.0.99
    5. Let it talk. Don't let it drive.1.00
    6. coding agents ops runbooks · onboarding flows0.98

  • 6:01 #15 done10 line(s)

    shot 15·sharpness 990.8

    1. ACE1.00
    2. Ornella Bahidika0.99
    3. Joel Allou0.95
    4. Let the model talk. You keep the1.00
    5. wheel.1.00
    6. Ornella Bahidika1.00
    7. Joel Allou0.97
    8. PRODUCT1.00
    9. ENGINEERING1.00
    10. aceactprep.com1.00

Transcript

57 cues· 964 words· 5,209 chars

  1. 0:00 Hi, I'm Ornella, that's Joelle, and we built Ace, a live AI voice tutor that runs a full lesson, start to finish, reliably.
  2. 0:09 The trick is LLM is not in charge.
  3. 0:13 If you have shipped a multi-step agent, you know this moment.
  4. 0:17 It's near the demo, then a real user gets in and halfway through, the agent decides it's done, or skip a step, or even loops.
  5. 0:26 The demo never shows you that.
  6. 0:30 And the first fix everyone reaches for is prompt is harder, add more rules.
  7. 0:35 But reliability was never a prompting problem.
  8. 0:39 It's a control problem.
  9. 0:40 Think of it this way.
  10. 0:42 The model is the talent and the harness is the director.
  11. 0:46 The model is brilliant at delivering a line, but it's really terrible at remembering if it's on step three of six.
  12. 0:53 So we stop asking it too.
  13. 0:56 A lesson is a small state machine with intro, teach, check, grade, advance, and wrap.
  14. 1:04 Each step sends a model a new contract.
  15. 1:07 Do this one thing, return it.
  16. 1:10 The harness validates what comes back, advance the state, and decide what's next.
  17. 1:16 The model never decide where we are, that they decide.
  18. 1:20 Joelle is going to show you the harness thing.
  19. 1:24 Yeah, so when we think about the frontier models of today, let's take, for example, Opus 4.7 Cloud from Anthropic, you will see that oftentimes people leverage the model for essentially everything, for the thinking, for the processing, right, and for everything in between.
  20. 1:46 Well, that can be good.
  21. 1:47 It's not always effective in situations like ours, where we are building a live AI tutor that is speaking back and forth with students.
  22. 1:57 For something like this, we have a need to actually build something that is reliable, something that is cost effective, and something that is fast.
  23. 2:08 So this is where the idea of leveraging the concept of RNS engineering has come in, where instead of having a model that is really intelligent sort of go through everything for us, we will build all of these steps that are needed and provide only the input required for the model to execute a specific scenario.
  24. 2:30 So when we were building ACE, we actually thought very deeply about state machines, right?
  25. 2:35 What is the step right now and what is the possible steps that could come after?
  26. 2:40 And within each of the steps, what are concrete things that we can provide to the model so that it is confined to that specific action?
  27. 2:49 It is confined to that specific step at that particular moment and only execute what needs to be done.
  28. 2:56 So by doing this, instead of having a very heavy model like a 4.7, we are actually able to rely on something like a Haiku 4.5, which is a much smaller model, doesn't have as much reasoning capabilities, but because of the harnessing around it, is still able to perform at the level in which we expect, saving money, saving time, and saving latency.
  29. 3:19 So let's go ahead and play this recording, which will show us logs about a particular lesson.
  30. 3:26 So as you can see in this video, especially on the right side, we see logs on all of the different harnessing that are happening.
  31. 3:36 For example, we see that there is harnessing for a section which provides input to the model about exactly what to speak about, what to do.
  32. 3:45 We have harnessing about drawing on the whiteboard.
  33. 3:48 We have harnessing that deals with clearing the queue.
  34. 3:52 We have steps to how to end the lesson and everything in between.
  35. 3:57 So everything that would allow us to actually build the lesson in a way that is reliable, even if there's a new scenario that comes in, we try to incorporate that in our state machine.
  36. 4:09 We try to incorporate that within the lesson.
  37. 4:11 So again, the model, all it worries about is giving the input.
  38. 4:16 It knows which action to take, and it provides the output of that action.
  39. 4:21 And so the model never really has to think.
  40. 4:25 It proposes, but ultimately it is the hardest that decides.
  41. 4:29 And so for Hayes specifically, there are three things that we wanted to think about.
  42. 4:33 Like when is the lesson done is one.
  43. 4:34 Did the student actually get it right?
  44. 4:37 Like did they actually learn in the way they were supposed to?
  45. 4:40 And what comes next?
  46. 4:41 And so everything that comes within those three categories,
  47. 4:45 all of the different questions or the different actions that the models needs to take we have engineered that outside of the model right so again it's an input the model receives it and gives us an output right and so this is very very important and we have found this to be very remarkable
  48. 5:02 So again, this is applicable to really everything.
  49. 5:06 It's applicable to something like ACE that is a voice model.
  50. 5:11 It's applicable to coding agents, to apps, runbooks.

Open at this second