vidtheque.

read-only demo

Videos BInpv7lGp1o

Your Agent Didn't Fail. Your Harness Did. — Vinoth Govindarajan, OpenAI

index_state ready data_status ok

AI Engineer· published 2026-07-29· 0:18:25· en-US· indexed 2026-08-10 19:38

Open on YouTube

Scene timeline

  1. Shot 0, 0:00 to 0:03, 1 of 1 keyframes kept
  2. Shot 1, 0:03 to 0:05, 1 of 1 keyframes kept
  3. Shot 2, 0:05 to 0:12, 1 of 1 keyframes kept
  4. Shot 3, 0:12 to 0:28, 1 of 1 keyframes kept
  5. Shot 4, 0:28 to 0:31, 1 of 1 keyframes kept
  6. Shot 5, 0:31 to 0:49, 1 of 1 keyframes kept
  7. Shot 6, 0:49 to 1:14, 1 of 1 keyframes kept
  8. Shot 7, 1:14 to 2:00, 1 of 1 keyframes kept
  9. Shot 8, 2:00 to 2:37, 1 of 1 keyframes kept
  10. Shot 9, 2:37 to 3:05, 0 of 1 keyframes kept
  11. Shot 10, 3:05 to 3:35, 1 of 1 keyframes kept
  12. Shot 11, 3:35 to 4:09, 1 of 1 keyframes kept
  13. Shot 12, 4:09 to 4:37, 0 of 1 keyframes kept
  14. Shot 13, 4:37 to 5:05, 0 of 1 keyframes kept
  15. Shot 14, 5:05 to 5:41, 1 of 1 keyframes kept
  16. Shot 15, 5:41 to 6:17, 1 of 1 keyframes kept
  17. Shot 16, 6:17 to 6:37, 1 of 1 keyframes kept
  18. Shot 17, 6:37 to 7:11, 1 of 1 keyframes kept
  19. Shot 18, 7:11 to 7:40, 1 of 1 keyframes kept
  20. Shot 19, 7:40 to 8:10, 1 of 1 keyframes kept
  21. Shot 20, 8:10 to 8:25, 1 of 1 keyframes kept
  22. Shot 21, 8:25 to 8:56, 1 of 1 keyframes kept
  23. Shot 22, 8:56 to 9:32, 1 of 1 keyframes kept
  24. Shot 23, 9:32 to 9:53, 0 of 1 keyframes kept
  25. Shot 24, 9:53 to 10:11, 0 of 1 keyframes kept
  26. Shot 25, 10:11 to 10:43, 1 of 1 keyframes kept
  27. Shot 26, 10:43 to 11:17, 1 of 1 keyframes kept
  28. Shot 27, 11:17 to 11:34, 1 of 1 keyframes kept
  29. Shot 28, 11:34 to 12:06, 0 of 1 keyframes kept
  30. Shot 29, 12:06 to 12:32, 1 of 1 keyframes kept
  31. Shot 30, 12:32 to 12:57, 1 of 1 keyframes kept
  32. Shot 31, 12:57 to 13:12, 1 of 1 keyframes kept
  33. Shot 32, 13:12 to 13:40, 0 of 1 keyframes kept
  34. Shot 33, 13:40 to 14:07, 1 of 1 keyframes kept
  35. Shot 34, 14:07 to 14:28, 1 of 1 keyframes kept
  36. Shot 35, 14:28 to 14:41, 1 of 1 keyframes kept
  37. Shot 36, 14:41 to 15:01, 1 of 1 keyframes kept
  38. Shot 37, 15:01 to 15:26, 1 of 1 keyframes kept
  39. Shot 38, 15:26 to 15:51, 0 of 1 keyframes kept
  40. Shot 39, 15:51 to 16:16, 1 of 1 keyframes kept
  41. Shot 40, 16:16 to 16:53, 1 of 1 keyframes kept
  42. Shot 41, 16:53 to 17:15, 1 of 1 keyframes kept
  43. Shot 42, 17:15 to 17:40, 1 of 1 keyframes kept
  44. Shot 43, 17:40 to 17:58, 1 of 1 keyframes kept
  45. Shot 44, 17:58 to 18:08, 1 of 1 keyframes kept
  46. Shot 45, 18:08 to 18:25, 0 of 1 keyframes kept

46 shot(s).

keyframes kept every frame deduplicated

What was stored

cues
259
whisperx 259
chunks
32
from 259 cues
keyframes
37
kept of 46 captured
frames with text
37
704 lines read
chapters
11
from the source metadata
keyframe bytes
4.8 MB
word timings on 259 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage state model started took
fetch done — 2026-08-09 22:03 0s
stt done — 2026-08-09 05:28 23s
chunk done — 2026-08-09 05:29 0s
text_embed done — 2026-08-10 19:38 1s
keyframe done — 2026-08-09 05:29 2m 19s
ocr done — 2026-08-09 05:31 14s
frame_embed done — 2026-08-10 19:38 6s

Frames, and what the machine read

  • 0:02 #0 done2 line(s)

    shot 0·sharpness 456.7

    1. AlEngineer0.96
    2. World's Fair0.97

  • 0:03 #1 done2 line(s)

    shot 1·sharpness 663.3

    1. AlEngineer0.95
    2. World's Fair0.99

  • 0:10 #2 done24 line(s)

    shot 2·sharpness 2742.3

    1. LAB & PLATINUM SPONSORS0.99
    2. Amazon AGI Lab0.98
    3. ANTHROP\C1.00
    4. Google DeepMind1.00
    5. MINIMAX0.93
    6. OpenAI0.92
    7. Akamai1.00
    8. arize0.92
    9. aws1.00
    10. Braintrust bright data0.98
    11. B1.00
    12. Browserbase1.00
    13. docker1.00
    14. :neo4j0.90
    15. ORACLE1.00
    16. PayPal1.00
    17. qodo1.00
    18. reducto1.00
    19. Sonar1.00
    20. Makers of1.00
    21. togetherai1.00
    22. Unblocked1.00
    23. WorkOS1.00
    24. SonarQube1.00

  • 0:17 #3 done2 line(s)

    shot 3·sharpness 197.0

    1. AlEngineer0.99
    2. World's Fair0.99

  • 0:31 #4 done11 line(s)

    shot 4·sharpness 1713.9

    1. OpenAI0.93
    2. AlEngineer0.99
    3. World's Fair0.96
    4. Your Agent Didn't Fail.1.00
    5. Your Harness Did.1.00
    6. PRESENTED·BY0.96
    7. Microsoft1.00
    8. Vinoth Govindarajan1.00
    9. Infrastructure1.00
    10. Engineering the future of Al0.99
    11. World's Fair0.97

  • 0:35 #5 done9 line(s)

    shot 5·sharpness 1503.4

    1. AlEngineer0.99
    2. World's Fair0.97
    3. PRESENTED·BY0.96
    4. The user saw the reply.1.00
    5. Microsoft1.00
    6. The system forgot it happened.0.99
    7. OpenAl0.97
    8. Engineering the future of Al0.99
    9. World'sFair1.00

  • 1:11 #6 done31 line(s)

    shot 6·sharpness 1916.1

    1. AlEngineer0.99
    2. Opening incident0.98
    3. World's Fair0.96
    4. PRESENTED BY0.97
    5. Support Assistant1.00
    6. Bot1.00
    7. Backend / memory store0.97
    8. A0.95
    9. NOT PERSISTED1.00
    10. Microsoft1.00
    11. Conversation: 9f3c2a1e1.00
    12. Remember the customer1.00
    13. asked for a refund on0.99
    14. order 4819.1.00
    15. User ID1.00
    16. u_7b9d2f4a1.00
    17. Session ID1.00
    18. sess_01HV9ZK31.00
    19. MISSING1.00
    20. Got it. I've recorded that1.00
    21. No record found.1.00
    22. and I'll remember it for0.99
    23. Last Turn1.00
    24. 11:31:021.00
    25. Nothing was persisted.1.00
    26. the next turn.1.00
    27. Durable Record1.00
    28. OpenAl0.95
    29. TRACK 1· JUNE 30, 20260.97
    30. World's Fair0.98
    31. Claws & Personal Agents1.00

  • 1:46 #7 done19 line(s)

    shot 7·sharpness 1875.8

    1. AlEngineer0.99
    2. Why this matters1.00
    3. World's Fair0.98
    4. PRESENTED BY0.95
    5. Microsoft1.00
    6. A crash gives1.00
    7. Silent success1.00
    8. 70.87
    9. Future context1.00
    10. you a boundary1.00
    11. gives you a lie1.00
    12. has a hole0.97
    13. Failure is explicit1.00
    14. The reply looks fine0.98
    15. Nothing looked broken1.00
    16. OpenAl0.97
    17. World's Fair0.96
    18. TRACK 1· JUNE 30, 20260.97
    19. Claws & Personal Agents0.99

  • 2:26 #8 done19 line(s)

    shot 8·sharpness 1585.9

    1. Hi, I'm Vinoth0.97
    2. AlEngineer0.99
    3. World's Fair0.97
    4. I bring a distributed-systems lens to agent infrastructure1.00
    5. PRESENTED BY0.96
    6. OpenAl0.95
    7. Apple, Uber1.00
    8. The Agent Stack0.99
    9. Microsoft1.00
    10. Core data1.00
    11. Previously, building1.00
    12. Systems writing1.00
    13. and Al infrastructure1.00
    14. distributed systems1.00
    15. on production agents1.00
    16. OpenAl0.96
    17. World's Fair0.97
    18. TRACK 1· JUNE 30,20260.95
    19. Claws & Personal Agents0.98

  • 2:59 #9 skipped

    shot 9·duplicate of #7

  • 3:32 #10 done17 line(s)

    shot 10·sharpness 2370.6

    1. AlEngineer0.99
    2. Three things to remember1.00
    3. Own the state0.98
    4. World'sFair1.00
    5. Who owns the truth?0.97
    6. Every failure mode in this talk reduces0.99
    7. to one of these boundaries1.00
    8. Order the mutation1.00
    9. Who committed first?1.00
    10. PRESENTED BY0.95
    11. Prove the action1.00
    12. Microsoft1.00
    13. Who can show what happened?1.00
    14. OpenAl0.96
    15. World's Fair0.94
    16. TRACK 1· JUNE 30, 20260.97
    17. Claws & Personal Agents0.99

  • 3:49 #11 done25 line(s)

    shot 11·sharpness 2232.9

    1. The model is the engine0.97
    2. AlEngineer0.99
    3. World'sFair1.00
    4. The harness is everything that lets the engine move safely in the real world0.99
    5. SPEED1.00
    6. LIMIT1.00
    7. 551.00
    8. Model1.00
    9. Dashboard1.00
    10. Black box1.00
    11. Observability1.00
    12. Run receipt1.00
    13. PRESENTED·BY0.96
    14. Microsoft1.00
    15. Brakes1.00
    16. Steering1.00
    17. Transmission1.00
    18. Road rules0.99
    19. Approval boundaries1.00
    20. State ownership1.00
    21. Tool routing1.00
    22. Policy1.00
    23. World's Fair1.00
    24. TRACK 1· JUNE 30,20260.93
    25. Claws & Personal Agents0.96

  • 4:26 #12 skipped

    shot 12·duplicate of #7

  • 4:46 #13 skipped

    shot 13·duplicate of #7

  • 5:20 #14 done37 line(s)

    shot 14·sharpness 2509.0

    1. Context is assembled, not remembered.1.00
    2. AlEngineer0.99
    3. World's Fair0.98
    4. Session transcript1.00
    5. 口1.00
    6. PRESENTED·BY0.95
    7. recent events1.00
    8. Working1.00
    9. Model1.00
    10. Response /0.98
    11. Persist new events1.00
    12. Session state1.00
    13. set1.00
    14. call1.00
    15. tool calls1.00
    16. + update receipt0.99
    17. Microsoft1.00
    18. scratchpad + metadata1.00
    19. Memory/summaries1.00
    20. A concise, relevant0.99
    21. Generate1.00
    22. Model output1.00
    23. Durably store0.99
    24. durable context1.00
    25. bundle of context0.96
    26. output1.00
    27. and actions1.00
    28. and advance1.00
    29. for this turn1.00
    30. the session1.00
    31. Policy + tools0.99
    32. available capabilities1.00
    33. The model only sees the working set the harness supplies0.99
    34. OpenAl0.97
    35. TRACK 1· JUNE 30,20260.95
    36. World's Fair1.00
    37. Claws & Personal Agents1.00

  • 5:52 #15 done28 line(s)

    shot 15·sharpness 2243.8

    1. Why old failures get sharper in agents?1.00
    2. AlEngineer0.97
    3. World'sFair1.00
    4. Familiar reliability bugs sit around a probabilistic planner1.00
    5. PRESENTED·BY0.95
    6. {f}0.98
    7. Dynamic1.00
    8. Probabilistic1.00
    9. More event0.98
    10. More action1.00
    11. Microsoft1.00
    12. plans1.00
    13. output1.00
    14. sources1.00
    15. surfaces1.00
    16. the next step is1.00
    17. the path may differ0.97
    18. users, tools, timers,0.98
    19. messages, files, APls,0.98
    20. chosen at runtime1.00
    21. between turns1.00
    22. webhooks, subagents0.98
    23. browsers, workflows0.97
    24. The agent setting makes them easier to trigger and harder to explain1.00
    25. OpenAl0.97
    26. TRACK 1·JUNE 30,20260.97
    27. World's Fair0.99
    28. Claws & Personal Agents0.99

  • 6:29 #16 done11 line(s)

    shot 16·sharpness 1820.2

    1. AlEngineer0.99
    2. World'sFair1.00
    3. FAILURE 1: STATE0.96
    4. PRESENTED·BY0.98
    5. Delivered is not remembered1.00
    6. Microsoft1.00
    7. The user sees success. The source of truth cannot replay it.1.00
    8. OpenAl0.96
    9. TRACK 1· JUNE 30,20260.95
    10. World'sFair1.00
    11. Claws & Personal Agents0.99

  • 7:07 #17 done33 line(s)

    shot 17·sharpness 2212.5

    1. AlEngineer0.99
    2. 目0.90
    3. FAILURE 1: STATE1.00
    4. World's Fair0.96
    5. The reply worked.1.00
    6. Replay broke.1.00
    7. openclaw / openclaw0.95
    8. Code1.00
    9. Issues 3.3k1.00
    10. Pull requests0.99
    11. 3.2k1.00
    12. Agents1.00
    13. Actions1.00
    14. More1.00
    15. PRESENTED BY0.96
    16. New issue0.97
    17. Microsoft1.00
    18. "Your turn was submitted."0.99
    19. User sees success1.00
    20. claude-cli backend skips runContextEngineMaintenance0.99
    21. — LCM/transcript persistence broken for all CLl-routed0.98
    22. Durable record1.00
    23. turns #815581.00
    24. Missing1.00
    25. ©Closed0.94
    26. #818690.99
    27. Delivery is not durable state.1.00
    28. rasimme opened on May 131.00
    29. Source: OpenClaw #815580.98
    30. OpenAl0.97
    31. TRACK 1· JUNE 30,20260.95
    32. World's Fair1.00
    33. Claws & Personal Agents1.00

  • 7:37 #18 done21 line(s)

    shot 18·sharpness 2021.3

    1. AlEngineer0.96
    2. 目0.96
    3. FAILURE1: STATE0.98
    4. World's Fair0.99
    5. By owner, I mean the system of record.0.99
    6. Storage tells you where bytes live. Ownership tells you who can reconstruct reality.1.00
    7. PRESENTED·BY0.95
    8. Microsoft1.00
    9. 日0.98
    10. Calendar event1.00
    11. Support status1.00
    12. Code change1.00
    13. Conversation turn0.97
    14. Calendar system1.00
    15. Ticketing system1.00
    16. Repository/workspace1.00
    17. Session transcript0.99
    18. OpenAl0.97
    19. TRACK 1· JUNE 30,20260.95
    20. World's Fair0.99
    21. Claws & Personal Agents0.98

  • 7:55 #19 done24 line(s)

    shot 19·sharpness 2177.9

    1. AlEngineer0.99
    2. 目0.96
    3. FAILURE 1: STATE0.97
    4. World's Fair0.97
    5. Reliable memory has three moves1.00
    6. If no owner can replay the fact, the system never truly remembered it0.99
    7. PRESENTED BY0.97
    8. Microsoft1.00
    9. 010.97
    10. 020.99
    11. 031.00
    12. Persist the turn1.00
    13. Name the owner1.00
    14. Make replay possible1.00
    15. Write every turn to durable1.00
    16. Define and document the1.00
    17. Capture inputs and events1.00
    18. storage as the source of truth0.99
    19. owner of record for the state0.99
    20. so state can be rebuilt0.99
    21. OpenAl0.98
    22. TRACK 1· JUNE 30,20260.94
    23. World's Fair0.99
    24. Claws & Personal Agents0.99

  • 8:11 #20 done12 line(s)

    shot 20·sharpness 1983.1

    1. AlEngineer0.99
    2. World'sFair1.00
    3. FAILURE 2: ORDER0.96
    4. PRESENTED·BY0.97
    5. Last writer wins is not1.00
    6. Microsoft1.00
    7. a consistency model1.00
    8. Two correct writes can still produce one wrong outcome0.99
    9. OpenAl0.96
    10. TRACK 1· JUNE 30,20260.96
    11. World'sFair1.00
    12. Claws & Personal Agents1.00

  • 8:52 #21 done36 line(s)

    shot 21·sharpness 2645.0

    1. AlEngineer0.99
    2. FAILURE 2: ORDER0.97
    3. World'sFair1.00
    4. openclaw / openclaw0.99
    5. Two writers.1.00
    6. Code1.00
    7. Issues 3.3k1.00
    8. Pull requests 3.2k0.97
    9. Agents1.00
    10. Actions1.00
    11. More1.00
    12. One stale snapshot.1.00
    13. New issue1.00
    14. C0.54
    15. PRESENTED BY0.94
    16. [Bug]: commitments store has no write serialization —0.99
    17. concurrent markCommitmentsStatus/1.00
    18. Microsoft1.00
    19. Load → mutate → save0.98
    20. markCommitmentsAttempted calls silently lose0.98
    21. dismiss/sent/attempt updates (data-loss class; can re-deliver1.00
    22. Two callers load the same old state.1.00
    23. Each changes a different record.0.99
    24. an already-sent commitment) #811450.99
    25. The second save silently erases the first.0.99
    26. Closed0.99
    27. 1#863260.93
    28. Missing boundary: one ordered commit path.0.99
    29. Labels1.00
    30. YB0y opened on May 120.99
    31. Contributor1.00
    32. Source: OpenClaw #81145 · PR #863260.97
    33. OpenAl0.97
    34. TRACK 1· JUNE 30, 20260.94
    35. World's Fair1.00
    36. Claws & Personal Agents0.99

  • 9:24 #22 done34 line(s)

    shot 22·sharpness 2584.7

    1. FAILURE 2: ORDER0.97
    2. AlEngineer0.99
    3. World'sFair1.00
    4. Serialize the mutation,1.00
    5. not the whole system1.00
    6. Serialize writes, detect stuck lanes,0.98
    7. PRESENTED BY0.95
    8. and acknowledges after commit1.00
    9. Session A1.00
    10. One active1.00
    11. Microsoft1.00
    12. lane1.00
    13. writer1.00
    14. Incoming1.00
    15. Global1.00
    16. →Session B0.99
    17. →One active0.97
    18. →State0.99
    19. events1.00
    20. throttle1.00
    21. lane1.00
    22. writer1.00
    23. commits1.00
    24. System-wide cap1.00
    25. Tools1.00
    26. Audit1.00
    27. Session C0.98
    28. One active1.00
    29. lane1.00
    30. writer1.00
    31. OpenAl0.97
    32. TRACK 1· JUNE 30, 20260.97
    33. World'sFair1.00
    34. Claws & Personal Agents0.99

  • 9:43 #23 skipped

    shot 23·duplicate of #19

Transcript

259 cues· 2,444 words· 14,109 chars

  1. 0:13 Thank you for choosing to spend this session with me.
  2. 0:16 My goal today is simple.
  3. 0:19 I want to convince you all that most of the agent failures are not model failures.
  4. 0:27 Those are harness failures.
  5. 0:29 So let's start with one production incident.
  6. 0:32 The user saw the reply.
  7. 0:34 The system forgot it happened.
  8. 0:36 This is the failure shape I want to start with.
  9. 0:39 Not a hallucination, not a crash, not a bad answer.
  10. 0:43 The user-visible edge looked healthy, while the durable record had a hole.
  11. 0:50 In this example, the user asked the agent to remember a refund for a customer.
  12. 0:58 The assistant said it recorded the fact for the next turn.
  13. 1:02 The interface looked normal, no red screen, no obvious failures.
  14. 1:06 But the next turn cannot reconcept the fact.
  15. 1:09 The user experienced success.
  16. 1:11 The system inherited incomplete reality.
  17. 1:16 Why this matters?
  18. 1:18 The crash is annoying, but at least it gives you a boundary.
  19. 1:22 You know something stopped.
  20. 1:24 You usually see an error.
  21. 1:26 You can often start from last known good point.
  22. 1:30 Silent success gives you a lie.
  23. 1:32 Delivery can succeed while the persistent fails.
  24. 1:36 The user has no reason to doubt the reply.
  25. 1:40 The operator has no obvious alarm.
  26. 1:43 The next turn can still sound confident because the model is coherent, but it is coherent over a broken history.
  27. 1:53 That is why agent reliability matters, and agent reliability cannot stop at model quality.
  28. 2:01 Hi, I'm Vinod.
  29. 2:04 core data and AI infrastructure at OpenAI.
  30. 2:07 Before that, I worked on distributed systems at Apple and Uber.
  31. 2:12 Outside of the work, I write the agent stack, where I try to explain how the production, AI, and data systems work under the hood.
  32. 2:20 I'm a daughter and mentor of OpenClaw, and this is not an OpenClaw product pitch.
  33. 2:25 This is a pure system design talk.
  34. 2:27 I'm using OpenClaw as a public case study because its issues, code, docs,
  35. 2:33 makes the harness around the agent unusually visible.
  36. 2:39 Here's the production contract for the talk.
  37. 2:41 A model proposes, the harness commits, and the receipts proceed.
  38. 2:46 The model may suggest a message, a tool, an edit, or a command, but model is not the production boundary.
  39. 2:54 The harness owns the state transition, the authority check, the ordered commit, and the receipt is the evidence that survives the turn.
  40. 3:02 OpenClaw is the case study, and the contract is the takeaway.
  41. 3:07 If you remember only three things from this talk, make it these.
  42. 3:10 Own the state, order the mutation, and prove the action.
  43. 3:16 A fact needs only one owner and one replay path.
  44. 3:19 Shared mutable state needs one ordered commit path.
  45. 3:22 And transcript is not the proof.
  46. 3:25 A transcript tells you what the agent said.
  47. 3:27 A receipt tells you what the system allowed, attempted, executed, and what the user visible edge confirmed.
  48. 3:37 To create a simple mental model, I created this car analogy of the harness.
  49. 3:42 The model is the engine.
  50. 3:44 It matters a lot.

Chapters

  1. 0:00 Introduction: harness failures vs model failures
  2. 1:32 Delivery can succeed while the truth fails
  3. 2:46 A model proposes, the harness commits, the receipt proves
  4. 4:14 How events enter and state is rehydrated
  5. 5:48 Idempotency, locks, and ordering
  6. 7:22 Ownership: who persists the turn
  7. 8:28 Single writer lanes and overlapping writes
  8. 10:09 Time, deadlines, and cancellation
  9. 11:23 Approval drift and bounded authority
  10. 13:05 Internal success vs user-visible proof
  11. 14:08 The run receipt audit: five questions

Open at this second