vidtheque.

read-only demo

Videos 3_gYbhABcAE

Why (Senior) Engineers Struggle to Build AI Agents — Philipp Schmid, Google DeepMind

index_state ready data_status ok

AI Engineer· published 2026-05-30· 0:10:39· en-US· indexed 2026-08-10 19:50

Open on YouTube

Scene timeline

  1. Shot 0, 0:00 to 0:05, 1 of 1 keyframes kept
  2. Shot 1, 0:05 to 0:09, 1 of 1 keyframes kept
  3. Shot 2, 0:09 to 0:14, 1 of 1 keyframes kept
  4. Shot 3, 0:14 to 0:35, 1 of 1 keyframes kept
  5. Shot 4, 0:35 to 1:19, 1 of 1 keyframes kept
  6. Shot 5, 1:19 to 1:34, 1 of 1 keyframes kept
  7. Shot 6, 1:34 to 2:04, 1 of 1 keyframes kept
  8. Shot 7, 2:04 to 2:05, 0 of 1 keyframes kept
  9. Shot 8, 2:05 to 2:31, 1 of 1 keyframes kept
  10. Shot 9, 2:31 to 2:58, 0 of 1 keyframes kept
  11. Shot 10, 2:58 to 3:24, 0 of 1 keyframes kept
  12. Shot 11, 3:24 to 3:50, 0 of 1 keyframes kept
  13. Shot 12, 3:50 to 4:23, 1 of 1 keyframes kept
  14. Shot 13, 4:23 to 5:05, 1 of 1 keyframes kept
  15. Shot 14, 5:05 to 5:34, 1 of 1 keyframes kept
  16. Shot 15, 5:34 to 6:03, 1 of 1 keyframes kept
  17. Shot 16, 6:03 to 6:31, 0 of 1 keyframes kept
  18. Shot 17, 6:31 to 6:51, 1 of 1 keyframes kept
  19. Shot 18, 6:51 to 7:18, 1 of 1 keyframes kept
  20. Shot 19, 7:18 to 7:46, 1 of 1 keyframes kept
  21. Shot 20, 7:46 to 8:23, 1 of 1 keyframes kept
  22. Shot 21, 8:23 to 9:05, 1 of 1 keyframes kept
  23. Shot 22, 9:05 to 9:32, 1 of 1 keyframes kept
  24. Shot 23, 9:32 to 9:58, 0 of 1 keyframes kept
  25. Shot 24, 9:58 to 10:24, 0 of 1 keyframes kept
  26. Shot 25, 10:24 to 10:39, 1 of 1 keyframes kept

26 shot(s).

keyframes kept every frame deduplicated

What was stored

cues
105
whisperx 105
chunks
18
from 105 cues
keyframes
19
kept of 26 captured
frames with text
19
342 lines read
chapters
0
from the source metadata
keyframe bytes
2.4 MB
word timings on 105 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage state model started took
fetch done — 2026-08-10 14:02 1m 40s
stt done — 2026-08-10 14:04 11s
chunk done — 2026-08-10 14:04 0s
text_embed done — 2026-08-10 19:50 0s
keyframe done — 2026-08-10 14:04 52s
ocr done — 2026-08-10 14:05 7s
frame_embed done — 2026-08-10 19:50 3s

Frames, and what the machine read

  • 0:03 #0 done2 line(s)

    shot 0·sharpness 658.4

    1. AlEngineer0.98
    2. EUROPE1.00

  • 0:08 #1 done2 line(s)

    shot 1·sharpness 829.1

    1. PRESENTINGSPONSOR1.00
    2. Google DeepMind1.00

  • 0:13 #2 done3 line(s)

    shot 2·sharpness 907.7

    1. PLATINUM SPONSORS0.98
    2. # Braintrust0.96
    3. WorkOS OpenAI0.95

  • 0:25 #3 done7 line(s)

    shot 3·sharpness 437.2

    1. Why (Senior) Engineers0.98
    2. Struggle to Build Al0.98
    3. Agents1.00
    4. 5 mental model collisions0.99
    5. engineering. and how to fix them0.99
    6. AlEngineer0.99
    7. EUROPE1.00

  • 0:40 #4 done25 line(s)

    shot 4·sharpness 2421.5

    1. TOP0.85
    2. Engineering Instincts1.00
    3. PHILIPP SCHMID· GOOGLE DEEPMIND0.98
    4. Strict Types, Determinism1.00
    5. Why (Senior) Engineers1.00
    6. ★1.00
    7. AIE1.00
    8. Struggle to Build Al0.99
    9. BOTECK0.75
    10. ★1.00
    11. ★1.00
    12. ★1.00
    13. Agents1.00
    14. Mental Model Collision1.00
    15. Fighting the probability of LLMs1.00
    16. 5 mental model collisions from traditional0.99
    17. engineering, and how to fix them.1.00
    18. BT0.88
    19. Agent Reality1.00
    20. Loops, Recovery, Contextual Meaning0.99
    21. 1/90.99
    22. AlEngineer0.97
    23. AlEngineer0.98
    24. EUROPE1.00
    25. EUROPE1.00

  • 1:26 #5 done18 line(s)

    shot 5·sharpness 3105.2

    1. THE PARADOX1.00
    2. Traffic Controller vs. Dispatcher0.99
    3. Traffic Controller Mindset: Traditional software engineering is Deterministic.1.00
    4. You own the roads, signals, and laws.1.00
    5. Dispatcher Mindset: Agent engineering is Probabilistic. You give instructions1.00
    6. to a driver tracking ambiguity.1.00
    7. ★1.00
    8. ★1.00
    9. AIE1.00
    10. The Trap: The better the engineer, the harder they fight the model. They try to0.99
    11. "code away" non-determinism with if-statements and rigid schemas.1.00
    12. ★1.00
    13. ★1.00
    14. 3/90.97
    15. AlEngineer0.98
    16. AlEngineer0.98
    17. EUROPE1.00
    18. EUROPE1.00

  • 1:49 #6 done4 line(s)

    shot 6·sharpness 359.9

    1. Traffic1.00
    2. vs. Dispatcher1.00
    3. AlEngineer0.99
    4. EUROPE1.00

  • 2:04 #7 skipped

    shot 7·duplicate of #5

  • 2:28 #8 done27 line(s)

    shot 8·sharpness 2790.8

    1. PART 10.98
    2. Text is the New State0.97
    3. The Trap: Treating the real world as enums1.00
    4. and booleans because it feels safe.1.00
    5. The Collision: Forcing natural language0.99
    6. intent into discrete booleans lobotomizes1.00
    7. AIE1.00
    8. ★1.00
    9. the context.1.00
    10. #Software Eng: Lobotomizing intent0.99
    11. ★1.00
    12. The Fix: Preserve semantic meaning1.00
    13. {"plan_id": "123", "status": "APPROVED"}0.99
    14. ★1.00
    15. ★1.00
    16. through raw strings so the agent can adapt0.98
    17. #Agent Eng: Preserving semantic meaning0.99
    18. intelligently downstream.1.00
    19. {"plan_id": "123", "text": "Approved, but focus on the US market and ignore CA."}1.00
    20. # Another example: boolean vs. semantic preference0.99
    21. is_celsius: true1.00
    22. #One-size-fits-all1.00
    23. "I prefer Celsius for weather, Fahrenheit for cooking." # Task-aware0.99
    24. 4/90.92
    25. Engineering the future of Al1.00
    26. AlEngineer0.98
    27. EUROPE1.00

  • 2:52 #9 skipped

    shot 9·duplicate of #8

  • 3:21 #10 skipped

    shot 10·duplicate of #8

  • 3:37 #11 skipped

    shot 11·duplicate of #8

  • 4:03 #12 done28 line(s)

    shot 12·sharpness 2347.8

    1. PART 21.00
    2. Hand Over Control0.99
    3. The Trap: In microservices, user intent maps to a route. We instinctively hard-0.99
    4. Customer Support0.97
    5. code the flow.1.00
    6. The Collision: Real interactions loop, backtrack, and pivot. Hard-coded paths0.98
    7. I want to cancel my subscription.0.99
    8. lose customers.1.00
    9. ★1.00
    10. INTENT: CHURN0.98
    11. AIE1.00
    12. ★1.00
    13. The Fix: Trust the agent to navigate. You're a dispatcher, not a traffic0.99
    14. ★1.00
    15. controller.0.98
    16. for the next 3 months.1.00
    17. I understand. Before you go, I can offer you 50% off0.98
    18. ★1.00
    19. ★1.00
    20. Key insight: Describe what you want, not the path to get there. Provide constraints, not0.99
    21. procedures.1.00
    22. Actually, yeah, that works.1.00
    23. INTENT: RETENTION0.99
    24. If you hard-coded the churn flow, you just lost a customer.1.00
    25. 5/90.83
    26. Engineering the future of Al0.98
    27. AlEngineer1.00
    28. EUROPE1.00

  • 4:48 #13 done3 line(s)

    shot 13·sharpness 349.8

    1. Hand1.00
    2. AlEngineer0.99
    3. EUROPE1.00

  • 5:17 #14 done31 line(s)

    shot 14·sharpness 2750.5

    1. PART 31.00
    2. Errors Are Just Inputs1.00
    3. The Trap: Failing fast and crashing the entire execution on a minor schema0.99
    4. fault.1.00
    5. LOOP0.93
    6. Agent Loop Running0.99
    7. The Collision: An agent run takes 5 minutes and costs $0.50. Crashing at0.99
    8. step 4 of 5 is unacceptable.1.00
    9. ★1.00
    10. AIE1.00
    11. The Fix: Catch the error and feed it back so the agent self-corrects.0.99
    12. ★1.00
    13. ★1.00
    14. ★1.00
    15. FAULT0.78
    16. Tool Execution Error Occurs1.00
    17. #Traditional: Fail Fast0.99
    18. raise RuntimeError(f"Failed: {e}") # Crash0.98
    19. #Agent: Feedback Loop0.98
    20. Traditional Way0.99
    21. Agent Way0.99
    22. return f"Error: {e}. Try a different approach." # Recover0.98
    23. Crash run1.00
    24. Feed back as String1.00
    25. $0.50 wasted0.96
    26. Agent Retry & Succeed0.99
    27. 6190.99
    28. AlEngineer0.97
    29. AlEngineer1.00
    30. EUROPE1.00
    31. EUROPE1.00

  • 5:43 #15 done31 line(s)

    shot 15·sharpness 2769.5

    1. PART 31.00
    2. Errors Are Just Inputs1.00
    3. The Trap: Failing fast and crashing the entire execution on a minor schema0.99
    4. fault.1.00
    5. LOOP0.96
    6. Agent Loop Running0.99
    7. The Collision: An agent run takes 5 minutes and costs $0.50. Crashing at0.99
    8. step 4 of 5 is unacceptable.0.98
    9. ★1.00
    10. AIE1.00
    11. The Fix: Catch the error and feed it back so the agent self-corrects.0.99
    12. ★1.00
    13. ★1.00
    14. ★1.00
    15. ★1.00
    16. FAULT0.87
    17. Tool Execution Error Occurs1.00
    18. #Traditional: Fail Fast1.00
    19. raise RuntimeError(f"Failed: {e}") # Crash0.99
    20. #Agent: Feedback Loop0.98
    21. Traditional Way1.00
    22. Agent Way1.00
    23. return f"Error: {e}. Try a different approach." # Recover0.98
    24. Crash run1.00
    25. Feed back as String1.00
    26. $0.50 wasted0.97
    27. Agent Retry & Succeed0.99
    28. 6190.96
    29. Engineering the future of Al0.99
    30. AlEngineer1.00
    31. EUROPE1.00

  • 6:12 #16 skipped

    shot 16·duplicate of #15

  • 6:36 #17 done4 line(s)

    shot 17·sharpness 351.4

    1. From1.00
    2. Evals1.00
    3. AlEngineer0.99
    4. EUROPE1.00

  • 7:10 #18 done39 line(s)

    shot 18·sharpness 2663.0

    1. PART 41.00
    2. From Unit Tests to Evals0.99
    3. Evaluate, Don't Assert: Output is nondeterministic. Run 3-5 trials per0.99
    4. prompt and measure the distribution.0.99
    5. EVAL FRAMEWORK1.00
    6. TESTS1.00
    7. EVALS1.00
    8. Negative Cases Matter: Test that your agent ignores requests outside its0.99
    9. scope. Triggering blindly degrades everything.1.00
    10. Question1.00
    11. "Did it work?"1.00
    12. "How often?"1.00
    13. ★1.00
    14. ★1.00
    15. ★0.94
    16. AIE1.00
    17. ★1.00
    18. Grade Outcomes, Not Paths: Did the agent yield the correct result? Don't1.00
    19. Method1.00
    20. Binary assertion0.98
    21. Pass^k (50x)1.00
    22. ★1.00
    23. ★1.00
    24. grade which tools it chose to get there.1.00
    25. Quality1.00
    26. N/A1.00
    27. LLM-as-Judge1.00
    28. ★1.00
    29. ★1.00
    30. ★1.00
    31. Debug1.00
    32. Stack trace1.00
    33. Tracing1.00
    34. Ready: 45/50 passes with quality score 4.5/5. You're managing risk, not eliminating variance.0.99
    35. 7190.99
    36. Braintrust1.00
    37. WorkOS OpenAI0.97
    38. AlEngineer1.00
    39. EUROPE1.00

  • 7:32 #19 done35 line(s)

    shot 19·sharpness 2577.7

    1. PART 41.00
    2. From Unit Tests to Evals0.98
    3. Evaluate, Don't Assert: Output is nondeterministic. Run 3-5 trials per0.99
    4. prompt and measure the distribution.0.99
    5. EVAL FRAMEWORK0.98
    6. TESTS1.00
    7. EVALS1.00
    8. Negative Cases Matter: Test that your agent ignores requests outside its1.00
    9. scope. Triggering blindly degrades everything.0.98
    10. Question1.00
    11. "Did it work?"0.97
    12. "How often?"0.96
    13. ★1.00
    14. AIE1.00
    15. Grade Outcomes, Not Paths: Did the agent yield the correct result? Don't0.99
    16. Method1.00
    17. Binary assertion0.99
    18. Pass^k (50x)0.99
    19. ★1.00
    20. grade which tools it chose to get there.1.00
    21. Quality1.00
    22. N/A1.00
    23. LLM-as-Judge1.00
    24. ★1.00
    25. ★1.00
    26. ★1.00
    27. Debug1.00
    28. Stack trace0.99
    29. Tracing1.00
    30. Ready: 45/50 passes with quality score 4.5/5. You're managing risk, not eliminating variance.0.98
    31. 7190.98
    32. AlEngineer0.98
    33. AlEngineer1.00
    34. EUROPE1.00
    35. EUROPE1.00

  • 8:15 #20 done34 line(s)

    shot 20·sharpness 2467.2

    1. PART 51.00
    2. Agents Evolve, APls Don't0.98
    3. A0.51
    4. Human-Grade API0.98
    5. ✓ Agent-Ready API0.96
    6. Agents are literalists and will hallucinate ambiguous parameters.1.00
    7. Explicit, self-documenting semantic interfaces.1.00
    8. ★1.00
    9. AIE1.00
    10. ★1.00
    11. delete_item(id)1.00
    12. delete_item_by_uuid(uuid: str)1.00
    13. No type, no description. Agent guesses id format.1.00
    14. "Deletes an item by UUID. Returns error if not found."1.00
    15. ★1.00
    16. ★1.00
    17. ★1.00
    18. ★1.00
    19. →0.99
    20. # Vague implementation1.00
    21. # Explicit, verbose, typed1.00
    22. def delete_item(id):1.00
    23. def delete_item_by_uuid(uuid: str):1.00
    24. pass1.00
    25. """Deletes an item by its UUID.1.00
    26. Returns error if not found.""0.99
    27. Semantic Naming0.98
    28. Verbose Docstrings1.00
    29. Just-In-Time Adaptation0.99
    30. 8/90.94
    31. AlEngineer0.97
    32. AlEngineer0.99
    33. EUROPE1.00
    34. EUROPE1.00

  • 8:32 #21 done8 line(s)

    shot 21·sharpness 367.1

    1. Agents1.00
    2. PIs Don't0.96
    3. Human-Grade API0.98
    4. Vague implementation0.96
    5. delete_item(id):0.75
    6. poss0.93
    7. AlEngineer0.99
    8. EUROPE1.00

  • 9:14 #22 done38 line(s)

    shot 22·sharpness 2112.5

    1. SUMMARY1.00
    2. Trust, but Verify1.00
    3. ★1.00
    4. AIE1.00
    5. ★1.00
    6. Stop Fighting1.00
    7. Preserve1.00
    8. Design for0.97
    9. Evaluate Don't1.00
    10. Build to1.00
    11. ★1.00
    12. the Model0.98
    13. Meaning1.00
    14. Recovery1.00
    15. Assert1.00
    16. Delete1.00
    17. ★1.00
    18. ★1.00
    19. ★1.00
    20. You're a dispatcher now. Give1.00
    21. instructions, let it pathfind.1.00
    22. Context is everything.1.00
    23. Text is greater than booleans.1.00
    24. Errors are inputs in a long-0.99
    25. running execution loop.1.00
    26. Manage risk through Pass^k0.97
    27. and LLM-as-judge.1.00
    28. harness gets rewritten. Make1.00
    29. The Bitter Lesson: every0.99
    30. yours modular.0.99
    31. 40.79
    32. "You cannot code away the probability. You must manage it through evals and self-correction."1.00
    33. Blog philschmid.de/why-engineers-struggle-building-agents1.00
    34. Philipp Schmid@_philschmid0.98
    35. 9190.97
    36. Engineering the future of Al0.99
    37. AlEngineer1.00
    38. EUROPE1.00

  • 9:52 #23 skipped

    shot 23·duplicate of #22

Transcript

105 cues· 1,748 words· 9,265 chars

  1. 0:15 OK, cool.
  2. 0:15 Awesome.
  3. 0:16 Hi, everyone.
  4. 0:17 My name is Filip.
  5. 0:18 I work at DeepMind, everything related to agents on Gemini or Gemini API.
  6. 0:22 So if you have some questions afterwards, some concerns, some bugs, some issue, please let me know.
  7. 0:28 We are going to talk today 10 minutes about why engineers struggle to build agents.
  8. 0:33 And I see this every day internally at Google, but also externally at Google.
  9. 0:37 And I brought five examples on what's really different to how we built traditional software a few years ago and to now how we build agents.
  10. 0:45 And if we, on a high level, compare them, when we wrote software, we created a spec, a PRD, wrote code,
  11. 0:53 sometimes create a test to make sure our code works.
  12. 0:56 We deployed it and then our user use it.
  13. 0:59 When building agents, things are a little bit different.
  14. 1:02 We define instructions on what we want our agent to do.
  15. 1:05 We run it, we observe what it does, we maybe adjust our prompts, maybe we adjust our tools.
  16. 1:11 We run it again and we have this iterative loop of how can we improve and make our agent way more reliable, which is very different to how we build software.
  17. 1:21 Something I like to compare it to is like traditional software is more like we acted as a traffic controller, right?
  18. 1:27 We had control over the street lights or how fast you can go, which road you can use, basically how the car drives.
  19. 1:34 And now with agents, we are more of a dispatcher.
  20. 1:36 We tell the agent, hey, I want to go to London and I'm from like Germany.
  21. 1:41 I could use the train, I could fly, I could use my car and go like under the water.
  22. 1:47 And it's more about, okay, we define the goal on what we want the agent to do, but we don't define the exact step the agent needs to take to achieve that goal.
  23. 1:55 And I mean, every one of you has probably seen in their coding agent that sometimes it does something very weird, but at the end it achieves the outcome.
  24. 2:03 And that's what we want to do.
  25. 2:05 So starting with the first example, text is our new state.
  26. 2:09 I mean, traditionally we had data structures and everything was kind of mapped to Boolean or to like flags we could check.
  27. 2:17 Initially, when we created, for example, a deep research agent, deep research agent returns a plan to you, okay, I'm going to research this and that.
  28. 2:27 In traditional software, we might have had an accept plan or deny plan, but we couldn't catch semantic meaning.
  29. 2:35 And now what we have with LLMs is they can understand the semantic meaning.
  30. 2:38 So, for example, if I have a deep research agent
  31. 2:42 request on like doing some market research I can approve the initial plan but I can also on the same time provide additional information so maybe I want to focus on like the US market and ignore California maybe I want to provide something additional and not have like this multiple steps right traditionally I would probably set decline and then it has a follow-up I might need it to provide more input create a new plan and continue and
  32. 3:07 And another good example is everything related to memory and personalization we do cannot really be mapped to data structures, right?
  33. 3:16 The example I have here is like, I'm from Europe, so I mostly use Celsius, but what if I would like to use Fahrenheit for cooking, right?
  34. 3:25 Previously, we might had some flex on like the user profile is Celsius or is Europe or use Fahrenheit, but I couldn't like dynamically adjust based on the user preference, based on what they provide.
  35. 3:39 So really it's all about text and context.
  36. 3:42 I mean, it could be images, video or audio as well, but we no longer are really operating in those clear structured data concepts.
  37. 3:51 The other thing is we should start handing over control and the trap or the example which we might have from like previous customer support is like when a user reached out, hey, I want to cancel my subscription, I might have had a classification model which kind of classified the attend, okay, the user wants to churn and then I had a predefined workflow of okay,
  38. 4:17 Do you try to sell it?
  39. 4:18 Do you cancel the subscription?
  40. 4:20 But there was no dynamic kind of option to react to it dynamically.
  41. 4:26 And maybe instead of like,
  42. 4:29 We're going through the subscription cancel flow.
  43. 4:31 What if your agent tries to understand the meaning and offers something except to the subscription and the user changes their mind and now you have a whole different intent?
  44. 4:43 And it's very hard to model all of those differences and uniqueness and to do all of those
  45. 4:51 stateful workflows we had before.
  46. 4:53 So we need to trust into the LLM or hand over control that we are no longer working in those purely deterministic environments.
  47. 5:04 The third one is errors are just inputs.
  48. 5:08 So if something in your agent flow fails, we need to treat it as a normal input, as very similar to a user input.
  49. 5:16 In Go, we already do this, right?
  50. 5:18 A function call can be an error or can be a value, and we treat them kind of equally.

Open at this second