vidtheque.

read-only demo

Videos 2JX6JYyQG4Y

Perception Agents — Antje Barth, Amazon AGI Lab

index_state ready data_status ok

AI Engineer· published 2026-07-23· 0:21:44· en· indexed 2026-08-10 19:41

Open on YouTube

Scene timeline

  1. Shot 0, 0:00 to 0:03, 1 of 1 keyframes kept
  2. Shot 1, 0:03 to 0:05, 1 of 1 keyframes kept
  3. Shot 2, 0:05 to 0:12, 1 of 1 keyframes kept
  4. Shot 3, 0:12 to 0:15, 1 of 1 keyframes kept
  5. Shot 4, 0:15 to 0:18, 1 of 1 keyframes kept
  6. Shot 5, 0:18 to 0:26, 1 of 1 keyframes kept
  7. Shot 6, 0:26 to 0:29, 1 of 1 keyframes kept
  8. Shot 7, 0:29 to 0:32, 1 of 1 keyframes kept
  9. Shot 8, 0:32 to 0:35, 1 of 1 keyframes kept
  10. Shot 9, 0:35 to 0:44, 1 of 1 keyframes kept
  11. Shot 10, 0:44 to 1:14, 1 of 1 keyframes kept
  12. Shot 11, 1:14 to 1:49, 1 of 1 keyframes kept
  13. Shot 12, 1:49 to 2:25, 0 of 1 keyframes kept
  14. Shot 13, 2:25 to 3:12, 1 of 1 keyframes kept
  15. Shot 14, 3:12 to 3:39, 1 of 1 keyframes kept
  16. Shot 15, 3:39 to 4:06, 0 of 1 keyframes kept
  17. Shot 16, 4:06 to 4:33, 0 of 1 keyframes kept
  18. Shot 17, 4:33 to 5:01, 1 of 1 keyframes kept
  19. Shot 18, 5:01 to 5:29, 0 of 1 keyframes kept
  20. Shot 19, 5:29 to 5:57, 0 of 1 keyframes kept
  21. Shot 20, 5:57 to 6:26, 1 of 1 keyframes kept
  22. Shot 21, 6:26 to 6:53, 1 of 1 keyframes kept
  23. Shot 22, 6:53 to 7:20, 1 of 1 keyframes kept
  24. Shot 23, 7:20 to 7:46, 1 of 1 keyframes kept
  25. Shot 24, 7:46 to 8:19, 1 of 1 keyframes kept
  26. Shot 25, 8:19 to 8:51, 0 of 1 keyframes kept
  27. Shot 26, 8:51 to 9:20, 1 of 1 keyframes kept
  28. Shot 27, 9:20 to 9:49, 0 of 1 keyframes kept
  29. Shot 28, 9:49 to 10:18, 0 of 1 keyframes kept
  30. Shot 29, 10:18 to 10:43, 1 of 1 keyframes kept
  31. Shot 30, 10:43 to 11:09, 0 of 1 keyframes kept
  32. Shot 31, 11:09 to 11:34, 1 of 1 keyframes kept
  33. Shot 32, 11:34 to 11:41, 1 of 1 keyframes kept
  34. Shot 33, 11:41 to 12:06, 1 of 1 keyframes kept
  35. Shot 34, 12:06 to 12:31, 1 of 1 keyframes kept
  36. Shot 35, 12:31 to 12:57, 1 of 1 keyframes kept
  37. Shot 36, 12:57 to 13:22, 0 of 1 keyframes kept
  38. Shot 37, 13:22 to 13:55, 1 of 1 keyframes kept
  39. Shot 38, 13:55 to 14:14, 1 of 1 keyframes kept
  40. Shot 39, 14:14 to 14:47, 1 of 1 keyframes kept
  41. Shot 40, 14:47 to 14:57, 1 of 1 keyframes kept
  42. Shot 41, 14:57 to 15:06, 0 of 1 keyframes kept
  43. Shot 42, 15:06 to 15:39, 1 of 1 keyframes kept
  44. Shot 43, 15:39 to 15:41, 1 of 1 keyframes kept
  45. Shot 44, 15:41 to 15:42, 1 of 1 keyframes kept
  46. Shot 45, 15:42 to 15:44, 1 of 1 keyframes kept
  47. Shot 46, 15:44 to 15:47, 1 of 1 keyframes kept
  48. Shot 47, 15:47 to 16:15, 1 of 1 keyframes kept
  49. Shot 48, 16:15 to 16:43, 0 of 1 keyframes kept
  50. Shot 49, 16:43 to 17:11, 0 of 1 keyframes kept
  51. Shot 50, 17:11 to 17:42, 1 of 1 keyframes kept
  52. Shot 51, 17:42 to 18:13, 0 of 1 keyframes kept
  53. Shot 52, 18:13 to 18:17, 1 of 1 keyframes kept
  54. Shot 53, 18:17 to 18:19, 1 of 1 keyframes kept
  55. Shot 54, 18:19 to 18:21, 0 of 1 keyframes kept
  56. Shot 55, 18:21 to 18:22, 1 of 1 keyframes kept
  57. Shot 56, 18:22 to 18:24, 0 of 1 keyframes kept
  58. Shot 57, 18:24 to 18:26, 1 of 1 keyframes kept
  59. Shot 58, 18:26 to 18:29, 1 of 1 keyframes kept
  60. Shot 59, 18:29 to 18:33, 0 of 1 keyframes kept
  61. Shot 60, 18:33 to 18:49, 0 of 1 keyframes kept
  62. Shot 61, 18:49 to 18:52, 1 of 1 keyframes kept
  63. Shot 62, 18:52 to 18:55, 1 of 1 keyframes kept
  64. Shot 63, 18:55 to 19:29, 1 of 1 keyframes kept
  65. Shot 64, 19:29 to 20:14, 0 of 1 keyframes kept
  66. Shot 65, 20:14 to 20:46, 1 of 1 keyframes kept
  67. Shot 66, 20:46 to 21:07, 1 of 1 keyframes kept
  68. Shot 67, 21:07 to 21:25, 1 of 1 keyframes kept
  69. Shot 68, 21:25 to 21:26, 1 of 1 keyframes kept
  70. Shot 69, 21:26 to 21:27, 1 of 1 keyframes kept
  71. Shot 70, 21:27 to 21:44, 0 of 1 keyframes kept

71 shot(s).

keyframes kept every frame deduplicated

What was stored

cues
229
whisperx 229
chunks
38
from 229 cues
keyframes
51
kept of 71 captured
frames with text
51
1,622 lines read
chapters
13
from the source metadata
keyframe bytes
9.3 MB
word timings on 229 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage state model started took
fetch done — 2026-08-09 23:07 0s
stt done — 2026-08-09 13:20 26s
chunk done — 2026-08-09 13:21 0s
text_embed done — 2026-08-10 19:41 0s
keyframe done — 2026-08-09 13:21 3m 28s
ocr done — 2026-08-09 13:24 28s
frame_embed done — 2026-08-10 19:41 9s

Frames, and what the machine read

  • 0:02 #0 done2 line(s)

    shot 0·sharpness 454.8

    1. AlEngineer0.96
    2. World's Fair0.97

  • 0:03 #1 done2 line(s)

    shot 1·sharpness 658.9

    1. AlEngineer0.95
    2. World's Fair0.99

  • 0:10 #2 done25 line(s)

    shot 2·sharpness 2731.8

    1. LAB & PLATINUM SPONSORS0.98
    2. Amazon AGI Lab0.97
    3. ANTHROP\C1.00
    4. Google DeepMind1.00
    5. MINIMAX0.97
    6. OpenAI0.92
    7. Akamai1.00
    8. arize1.00
    9. aws1.00
    10. Braintrust1.00
    11. bright data0.99
    12. B1.00
    13. Browserbase1.00
    14. docker1.00
    15. neo4j0.99
    16. ORACLE1.00
    17. PayPal1.00
    18. qodo1.00
    19. reducto1.00
    20. Sonar1.00
    21. Makers of0.99
    22. together.ai0.96
    23. Unblocked1.00
    24. WorkOS1.00
    25. SonarQube1.00

  • 0:14 #3 done2 line(s)

    shot 3·sharpness 173.1

    1. Amazon AGI Lab0.99
    2. AIE0.92

  • 0:16 #4 done2 line(s)

    shot 4·sharpness 181.9

    1. Amazon AGI Lab1.00
    2. AIE0.94

  • 0:25 #5 done4 line(s)

    shot 5·sharpness 406.2

    1. ANTJEBARTH1.00
    2. MEMBER OF TECHNICAL STAFF1.00
    3. Amazon AGI Lab0.99
    4. AIF1.00

  • 0:28 #6 done5 line(s)

    shot 6·sharpness 1009.0

    1. ANTJEBARTH0.98
    2. Lab1.00
    3. MEMBER OF TECHNICAL STAFF0.99
    4. Amazon AGI Lab0.99
    5. AIE1.00

  • 0:32 #7 done81 line(s)

    shot 7·sharpness 3279.0

    1. World'sFair1.00
    2. Engineer1.00
    3. World's Fai0.90
    4. Fair1.00
    5. THE VELOCITY ROOM0.97
    6. mezmo*0.88
    7. io0.88
    8. fiddler0.95
    9. air1.00
    10. INNGEST1.00
    11. Cane0.67
    12. e0.98
    13. World's Fair0.96
    14. PRIOR0.91
    15. rox0.62
    16. air1.00
    17. World's Fair0.97
    18. Z.AI0.94
    19. BAND1.00
    20. World's Fair0.97
    21. Keycard1.00
    22. AUTOMATTIC1.00
    23. :neo4j0.97
    24. World'sFair0.99
    25. Cleric1.00
    26. Meticulous0.95
    27. s0.75
    28. World's Fair0.97
    29. World's Fair0.94
    30. Google DeepMind1.00
    31. qodo0.99
    32. World's Fai0.90
    33. World's Fail0.92
    34. À ATLASSIAN0.96
    35. GGRAVITEE0.91
    36. Amazon AGI Lab1.00
    37. World's Fait0.92
    38. World's Fa0.96
    39. arize0.98
    40. Gradium0.89
    41. FACTORY1.00
    42. Braintrust1.00
    43. Microsoft1.00
    44. reducto1.00
    45. ANTHROPIC0.95
    46. World's Fair0.94
    47. World's Fair0.97
    48. OpenAl0.92
    49. World'sFai0.99
    50. World's Fair0.93
    51. MINIMAX1.00
    52. Worid'sFai0.90
    53. bright data0.91
    54. Microsoht0.94
    55. togetherai1.00
    56. World's Fair0.92
    57. Z.AI0.98
    58. World'sFair0.83
    59. ANTHROPIC0.98
    60. World's Fai0.95
    61. World's Fair0.96
    62. DATADOG1.00
    63. Jorkd'sFal0.76
    64. Arbyte0.92
    65. World's Fair0.96
    66. Resolve.ai1.00
    67. CODERB0.94
    68. Optiver0.98
    69. World's Fair0.98
    70. B0.58
    71. builder.io0.99
    72. World's Fair0.95
    73. Ravenna1.00
    74. World'sF0.99
    75. Col0.74
    76. Vorld'sFa0.90
    77. TOPK1.00
    78. World's Fair0.96
    79. cognee1.00
    80. World'sFai1.00
    81. BeNCORD0.88

  • 0:35 #8 done199 line(s)

    shot 8·sharpness 2596.8

    1. World's Fair0.95
    2. ORACLE1.00
    3. World's Fair0.96
    4. arize1.00
    5. World's Fair0.96
    6. Google DeepMind0.99
    7. World's Fair0.94
    8. :neo4j0.90
    9. World's Fair0.97
    10. Z.AI0.99
    11. World's Fair0.94
    12. bright data0.99
    13. World's Fair0.98
    14. Browserbase0.99
    15. paper compute co.0.98
    16. World's Fair0.94
    17. extend1.00
    18. World's Fair0.95
    19. vast.ai1.00
    20. World's Fair0.96
    21. Ref.1.00
    22. World's Fair0.98
    23. RedHat1.00
    24. World's Fair0.95
    25. mezmo+0.96
    26. World's Fair0.98
    27. stigg1.00
    28. World's Fair0.98
    29. World's Fair0.97
    30. RELAI0.96
    31. World's Fair0.91
    32. VAPI0.99
    33. World's Fair0.98
    34. Modal1.00
    35. World's Fair0.98
    36. promptql1.00
    37. World's Fair0.97
    38. THE VELOCITY ROOM0.97
    39. World's Fair0.98
    40. fiddler1.00
    41. World's Fair0.96
    42. Superconductor1.00
    43. Surreal80.90
    44. World'sFair0.99
    45. ZERO0.95
    46. World's Fair0.97
    47. AlEngineer0.96
    48. comet1.00
    49. World's Fair0.98
    50. authe0.99
    51. World'sFai0.99
    52. World'sFair1.00
    53. SOIOIO0.93
    54. World's Fair0.94
    55. granica1.00
    56. World's Fair0.94
    57. dash00.99
    58. World's Fair0.99
    59. PRIOR1.00
    60. World's Fair1.00
    61. DigitalOcean1.00
    62. World's Fair0.95
    63. Vence0.96
    64. World's Fair0.97
    65. POSTMAN0.98
    66. World's Fair0.89
    67. Composio1.00
    68. World's Fair0.98
    69. World's Fair0.98
    70. Modular1.00
    71. World's Fair0.97
    72. MERGE1.00
    73. World's Fair0.97
    74. AUTOMATTIC0.98
    75. World's Fair0.98
    76. Buildkite1.00
    77. yugabyteDB1.00
    78. World'sFair0.97
    79. Zed1.00
    80. World'sFair0.98
    81. Keycard1.00
    82. World's Fair0.95
    83. Meticulous1.00
    84. World's Fair0.96
    85. World's Fair0.98
    86. Daytona1.00
    87. World's Fair0.95
    88. twilio0.92
    89. World's Fair0.98
    90. GRAVITEE1.00
    91. World's Fair0.97
    92. PlanetScale0.99
    93. Temporal1.00
    94. World's Fair0.97
    95. W Llamalndex0.93
    96. World's Fair0.96
    97. INNGEST0.99
    98. World's Far0.96
    99. BAND1.00
    100. World's Fair0.97
    101. Cleric0.97
    102. World's Fair0.99
    103. À ATLASSIAN0.93
    104. World's Fair0.94
    105. FACTORY1.00
    106. World'sFair0.93
    107. World's Fair0.99
    108. baseten1.00
    109. World's Fair0.96
    110. Snorkel0.94
    111. World's Fair0.96
    112. Z.AI0.99
    113. World's Fair0.96
    114. qodo0.95
    115. World's Fair0.95
    116. PayPal1.00
    117. World'sFair0.97
    118. Gradium1.00
    119. World's Fair0.98
    120. ANTHROPIC1.00
    121. Amazon AGI Lab0.99
    122. World's Fair0.99
    123. World's Fair0.96
    124. :neo4j0.95
    125. World's Fair0.95
    126. Google DeepMind1.00
    127. World's Fair0.96
    128. arize1.00
    129. World's Fair0.95
    130. reducto1.00
    131. World's Fair0.96
    132. Microsoft0.95
    133. World's Fair0.93
    134. World's Fair0.96
    135. OpenAl0.97
    136. World's Fair0.92
    137. WorkOS0.97
    138. World's Fair0.97
    139. Amazon AGI Lab0.99
    140. World's Fair0.93
    141. Microsoft1.00
    142. World's Fair0.89
    143. ORACLE1.00
    144. World's Fair0.98
    145. bright data0.99
    146. World's Fair0.96
    147. Google DeepMind1.00
    148. Microsoft0.99
    149. World'sFair0.99
    150. docker1.00
    151. World'sFair0.94
    152. Braintrust1.00
    153. World's Fair0.94
    154. OpenAl0.97
    155. World's Fair0.94
    156. MINIMAX0.95
    157. World's Fair0.96
    158. Z.AI0.99
    159. World's Fair0.96
    160. ANTHROPIC1.00
    161. World's Fair0.94
    162. World'sFair0.99
    163. snyk1.00
    164. World's Fair0.94
    165. aws1.00
    166. World's Fair0.97
    167. togetherai0.93
    168. World's0.90
    169. World's Fair0.96
    170. Unblocked1.00
    171. Lightrun1.00
    172. World's Fair0.95
    173. CLOUDFLARE0.93
    174. World'sFair0.99
    175. DATAD0.97
    176. World's Fai0.95
    177. Resolve.ai1.00
    178. World's Fai0.93
    179. Worlds Fal0.79
    180. Worid s Fair0.91
    181. World'sFair0.97
    182. World's Fair0.95
    183. LanceDB1.00
    184. World's Fai0.93
    185. builderio0.95
    186. World'sF0.98
    187. Ravenna1.00
    188. World's Fal0.92
    189. CopilotKit1.00
    190. descupe0.90
    191. oft1.00
    192. World's Fair0.93
    193. TOPK1.00
    194. World's Fair0.94
    195. cogne1.00
    196. World'sFair0.99
    197. eNCORD0.93
    198. World'sFa1.00
    199. Vorld's Fair0.87

  • 0:41 #9 done10 line(s)

    shot 9·sharpness 662.8

    1. AlEngineer0.99
    2. Open/1.00
    3. World's Fa0.98
    4. AlEr0.93
    5. GEPT0.85
    6. Worl1.00
    7. Akamai1.00
    8. AlEngineer0.97
    9. DA1.00
    10. World's Fa0.99

  • 1:05 #10 done15 line(s)

    shot 10·sharpness 1938.5

    1. AlEngineer0.97
    2. World'sFair1.00
    3. Clicking was1.00
    4. the easy part.0.98
    5. Open1.00
    6. W0.99
    7. Amazon AGl Lab0.98
    8. © 2026 Amazon.com, Inc. or Its afilates.0.97
    9. AIEn0.90
    10. RGET0.77
    11. Norl0.99
    12. Perception Agents1.00
    13. DA1.00
    14. W1.00
    15. Antje Barth / Member of Technical Staff Amazon AGI Lab0.97

  • 1:38 #11 done18 line(s)

    shot 11·sharpness 2590.6

    1. AlEngireer0.95
    2. World's Fair0.97
    3. The agent can use every tool.1.00
    4. It still can't do the work.0.98
    5. AI0.83
    6. Vorld's Fi0.94
    7. AlEngineer0.97
    8. Amazon AGI Lab0.99
    9. © 2026 Amazon.com, Inc. or Its afillates.0.97
    10. Fair1.00
    11. RCEPTY0.96
    12. GENT1.00
    13. Akama1.00
    14. Perception Agents1.00
    15. AlEngineer1.00
    16. DOG1.00
    17. orld's Fi0.90
    18. Antje Barth / Member of Technical Staff Amazon AGI Lab0.97

  • 2:10 #12 skipped

    shot 12·duplicate of #11

  • 3:02 #13 done16 line(s)

    shot 13·sharpness 2067.1

    1. AIEngineer0.97
    2. World'sFair1.00
    3. We taught computers1.00
    4. to use computers.1.00
    5. - AlEngineer0.99
    6. orld'sFai0.99
    7. Amazon AGl Lab0.98
    8. © 2026 Amazon.com, Inc. or its afillates.0.98
    9. Akal0.94
    10. GENT1.00
    11. Wor1.00
    12. Perception Agents1.00
    13. - AlEngineer -0.93
    14. orld's0.97
    15. Re1.00
    16. Antje Barth / Member of Technical Staff Amazon AGI Lab0.98

  • 3:36 #14 done20 line(s)

    shot 14·sharpness 2183.3

    1. AlEngineer0.99
    2. World's Fair0.99
    3. Capability, we mostly figured out.0.98
    4. Reliability is the1.00
    5. hard part.0.98
    6. igineer0.98
    7. I's Fair0.90
    8. MMI0.78
    9. Amazon AGI Lab0.99
    10. © 2026 Amazon.com, Inc. or Its afilates.0.96
    11. RCEPTV0.98
    12. — AlEngin0.89
    13. GENT1.00
    14. orld'0.98
    15. tamai0.96
    16. Perception Agents1.00
    17. igineer1.00
    18. I's Fai0.91
    19. Reso0.96
    20. Antje Barth / Member of Technical Staff Amazon AGI Lab0.97

  • 3:43 #15 skipped

    shot 15·duplicate of #14

  • 4:15 #16 skipped

    shot 16·duplicate of #14

  • 4:36 #17 done16 line(s)

    shot 17·sharpness 1994.8

    1. AlEngineer0.96
    2. World'sFair1.00
    3. Coding already1.00
    4. made the jump.0.99
    5. AI0.96
    6. "'sF:0.75
    7. gineer1.00
    8. Amazon AGI Lab0.96
    9. © 2026 Amazon.com, Inc. or its afilates.0.97
    10. Fair0.99
    11. ama0.98
    12. Perception Agents1.00
    13. gineer1.00
    14. DOG0.98
    15. I'sFi0.93
    16. Antje Barth/ Member of Technical Staff Amazon AGI Lab0.98

  • 5:05 #18 skipped

    shot 18·duplicate of #17

  • 5:33 #19 skipped

    shot 19·duplicate of #17

  • 6:06 #20 done16 line(s)

    shot 20·sharpness 2162.2

    1. AlEngineer0.98
    2. World'sFair1.00
    3. Code crossed it first1.00
    4. because you can verify it.0.99
    5. Engineer1.00
    6. I'sFair0.92
    7. Amazon AGI Lab0.95
    8. © 2026 Amazon.com, Inc. or Its affillates.0.98
    9. RCEPTV0.96
    10. GENT1.00
    11. osoft0.86
    12. Perception Agents1.00
    13. te1.00
    14. sFair0.99
    15. neer1.00
    16. Antje Barth / Member of Technical Staff Amazon AGI Lab0.97

  • 6:32 #21 done18 line(s)

    shot 21·sharpness 2018.4

    1. AlEngineer0.96
    2. World'sFair1.00
    3. Knowledge work is messy. The real world is messy.0.99
    4. But most work1.00
    5. isn't verifiable.1.00
    6. AIEI0.86
    7. penAl0.91
    8. Worl0.98
    9. Amazon AGI Lab0.97
    10. ©2026 Amazon.com, Inc. or Its affililates.0.93
    11. IEngineer0.95
    12. 1d'sF0.96
    13. Mi1.00
    14. Perception Agents0.98
    15. AIEr0.80
    16. Buildr0.82
    17. Work0.99
    18. Antje Barth / Member of Technical Staff Amazon AGI Lab0.99

  • 7:09 #22 done19 line(s)

    shot 22·sharpness 2056.1

    1. AlEngineer0.99
    2. World's Fair0.98
    3. Knowledge work is messy. The real world is messy.0.99
    4. But most work1.00
    5. isn't verifiable.1.00
    6. gineer1.00
    7. I's Fair0.94
    8. do1.00
    9. Amazon AGI Lab0.99
    10. © 2026 Amazon.com, Inc. or its affillates.0.98
    11. AlEngin1.00
    12. cro1.00
    13. IGENT0.99
    14. orld's0.97
    15. Perception Agents1.00
    16. gineer1.00
    17. I'sFa0.98
    18. Deas1.00
    19. Antje Barth/ Member of Technical Staff Amazon AGI Lab0.99

  • 7:30 #23 done17 line(s)

    shot 23·sharpness 2046.7

    1. AlEngineer0.98
    2. World'sFair1.00
    3. So how do humans1.00
    4. handle the messy work?1.00
    5. AI0.89
    6. orld'sF1.00
    7. AlEngineer1.00
    8. Amazon AGI Lab0.98
    9. © 2026 Amazon.com, Inc. or Its afflates.0.94
    10. Fair1.00
    11. IGENT1.00
    12. cro:0.89
    13. Perception Agents1.00
    14. Engineer1.00
    15. dkite1.00
    16. id'sF0.90
    17. Antje Barth / Member of Technical Staff Amazon AGI Lab0.98

Transcript

229 cues· 2,842 words· 15,049 chars

  1. 0:12 Joining us on stage is a member of technical staff at Amazon AGI Lab, Anjie Barth.
  2. 0:36 Good morning.
  3. 0:37 It's so great to be back here at the AI Engineer World's Fair.
  4. 0:43 Just a year ago, the hard problem was getting an agent to find a button and click it on a screen, especially screens it had never seen before.
  5. 0:57 Now, agents can drive browsers, and they're starting to also drive desktop apps.
  6. 1:06 But what we figured out, clickered, clicking, was actually the easy part.
  7. 1:15 What we didn't solve is the actual work.
  8. 1:20 And what do I mean with this?
  9. 1:22 Let's take a very simple example.
  10. 1:25 A new team member starts on Monday.
  11. 1:28 And maybe your job is to set up their accounts, add them to your Slack channel, book intros with colleagues, order the laptops, et cetera.
  12. 1:40 And nobody really owns this end-to-end process in the company.
  13. 1:46 And it might be also touching five different systems.
  14. 1:51 Now, agents can most likely perform each single individual step of this workflow.
  15. 1:59 But agents still struggle to do this end to end because the real work lives within the seams of all of those different applications, of all of those different steps you have to take.
  16. 2:12 And this is mostly where it all falls apart.
  17. 2:17 The agent can use every single tool you give it, but it still can't do the full work.
  18. 2:26 So why do we see this gap?
  19. 2:30 Think about for a minute what we actually built.
  20. 2:34 We taught computers to use computers.
  21. 2:38 So what do I mean with this?
  22. 2:40 We started building out the basics.
  23. 2:42 We taught them clicking, scrolling, typing, calling an API, filling out a form.
  24. 2:50 And we got those steps really reliable.
  25. 2:54 And you can string them together in a workflow.
  26. 2:57 And agents these days are fairly good at operating those workflows.
  27. 3:03 So why can't you not just hand them more of your work and then literally just walk away and trust it to be completed?
  28. 3:14 So all the things I talked about, like using a tool, models itself, tool use, stringing agents together, this is all capabilities.
  29. 3:27 And we mostly figured out how to add capabilities to models.
  30. 3:33 Now the next hard part is really reliability.
  31. 3:38 And without reliability, we cannot really build up trust in those systems.
  32. 3:45 So here's a quick gut check, and maybe all of you can just think about an agent doing work in an end-to-end workflow.
  33. 3:54 How often do you think that actually succeeds these days?
  34. 3:59 Maybe 60, maybe 80% of the time?
  35. 4:03 And it sounds really fine, but if you look into this, if your agent one in four times deletes the database, you will never touch that agent again, right?
  36. 4:19 So when you need this reliability, you really need to be in the nines.
  37. 4:27 You need to have the trust that it actually can do the work successfully.
  38. 4:35 Now, there's actually one place where we made enormous progress on reliability and trust.
  39. 4:44 And this is coding, right?
  40. 4:47 Think about how fast coding evolved.
  41. 4:52 I still remember the first time when it started auto-completing for you, right?
  42. 4:56 You just tapped auto-complete, amazing.
  43. 5:00 Then, short time later, it started to write functions
  44. 5:04 And we thought that is amazing.
  45. 5:07 And now look at these days, coding agents write the code, they open up the pull request themselves, and we had it earlier this week, code keeps flying by.
  46. 5:19 So once in a time, we were able to just, every single line that it generated, we felt like the urge, we need to really read it and make sure it's correct, right?
  47. 5:29 I think most in the audience here can still relate to that.
  48. 5:33 These days, I think hardly anyone is still doing that.
  49. 5:38 Like we cannot even do that, right?
  50. 5:39 Code is generated at such a pace.

Chapters

  1. 0:00 Introduction to the AI Engineer World's Fair
  2. 0:43 The Evolution of AI Agent Capabilities
  3. 1:15 The Problem: Why Agents Struggle with Real Work
  4. 2:26 Understanding the Gap: Reliability and Trust
  5. 4:36 Why Coding Agents Succeeded: The Role of Verification
  6. 6:27 The Challenge of "Messy" Knowledge Work
  7. 7:29 How Humans Collaborate: The Power of Shared Context
  8. 9:22 Introducing Perception Agents: Perceive, Plan, Act
  9. 11:36 Why Perception Agents Matter: Closing the Loop
  10. 13:23 Open Source Harness: Annotation and Verification
  11. 16:48 Multimodal Perception: Beyond the Screen
  12. 19:30 Call to Action: Building Together
  13. 20:07 We want to build AI that makes all of us smarter together.

Open at this second