vidtheque.

read-only demo

Videos q2JrUKBMf0w

The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI

index_state ready data_status ok

AI Engineer· published 2026-07-24· 0:06:06· en-US· indexed 2026-08-10 19:40

Open on YouTube

Scene timeline

  1. Shot 0, 0:00 to 0:03, 1 of 1 keyframes kept
  2. Shot 1, 0:03 to 0:05, 1 of 1 keyframes kept
  3. Shot 2, 0:05 to 0:12, 1 of 1 keyframes kept
  4. Shot 3, 0:12 to 0:13, 1 of 1 keyframes kept
  5. Shot 4, 0:13 to 0:45, 1 of 1 keyframes kept
  6. Shot 5, 0:45 to 1:16, 1 of 1 keyframes kept
  7. Shot 6, 1:16 to 1:42, 1 of 1 keyframes kept
  8. Shot 7, 1:42 to 1:59, 1 of 1 keyframes kept
  9. Shot 8, 1:59 to 2:38, 1 of 1 keyframes kept
  10. Shot 9, 2:38 to 3:11, 1 of 1 keyframes kept
  11. Shot 10, 3:11 to 3:44, 0 of 1 keyframes kept
  12. Shot 11, 3:44 to 3:53, 1 of 1 keyframes kept
  13. Shot 12, 3:53 to 4:34, 1 of 1 keyframes kept
  14. Shot 13, 4:34 to 5:20, 1 of 1 keyframes kept
  15. Shot 14, 5:20 to 5:42, 1 of 1 keyframes kept
  16. Shot 15, 5:42 to 5:49, 1 of 1 keyframes kept
  17. Shot 16, 5:49 to 6:05, 0 of 1 keyframes kept

17 shot(s).

keyframes kept every frame deduplicated

What was stored

cues
69
whisperx 69
chunks
10
from 69 cues
keyframes
15
kept of 17 captured
frames with text
15
460 lines read
chapters
4
from the source metadata
keyframe bytes
2.2 MB
word timings on 69 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage state model started took
fetch done — 2026-08-09 22:53 0s
stt done — 2026-08-09 07:19 7s
chunk done — 2026-08-09 07:19 0s
text_embed done — 2026-08-10 19:40 0s
keyframe done — 2026-08-09 07:19 49s
ocr done — 2026-08-09 07:20 8s
frame_embed done — 2026-08-10 19:40 3s

Frames, and what the machine read

  • 0:02 #0 done2 line(s)

    shot 0·sharpness 449.6

    1. AlEngineer0.96
    2. World's Fair0.97

  • 0:04 #1 done2 line(s)

    shot 1·sharpness 661.2

    1. AlEngineer0.95
    2. World's Fair0.99

  • 0:10 #2 done26 line(s)

    shot 2·sharpness 2714.8

    1. LAB & PLATINUM SPONSORS0.98
    2. Amazon AGI Lab1.00
    3. ANTHROP\C1.00
    4. Google DeepMind1.00
    5. MINIMAX0.97
    6. OpenAl0.94
    7. 70.60
    8. Akamai1.00
    9. arize0.92
    10. aws1.00
    11. Braintrust1.00
    12. bright data0.98
    13. B1.00
    14. Browserbase1.00
    15. docker1.00
    16. :neo4j0.91
    17. ORACLE1.00
    18. PayPal1.00
    19. qodo1.00
    20. reducto1.00
    21. Sonar1.00
    22. Makers of1.00
    23. together.ai0.98
    24. Unblocked1.00
    25. WorkOS1.00
    26. SonarQube1.00

  • 0:13 #3 done12 line(s)

    shot 3·sharpness 296.4

    1. AlEnginee0.98
    2. Oper0.99
    3. ir1.00
    4. World's1.00
    5. AlEn0.77
    6. ai1.00
    7. Wor'0.98
    8. Akan1.00
    9. arize1.00
    10. AIEnginee0.95
    11. E口0.52
    12. World's1.00

  • 0:41 #4 done8 line(s)

    shot 4·sharpness 317.1

    1. AlEngineer0.98
    2. penAl0.95
    3. Id's Fair0.98
    4. -AlEngineer0.96
    5. orld's Fair0.99
    6. mai0.96
    7. DATADOG1.00
    8. I'sFair0.99

  • 1:00 #5 done10 line(s)

    shot 5·sharpness 415.0

    1. AlEngineer0.98
    2. OpenAI0.93
    3. Vorld's Fair0.96
    4. AlEngineer0.98
    5. World's Fair0.98
    6. arize1.00
    7. Akamai1.00
    8. AlEngineer0.97
    9. DATADOG1.00
    10. vorld's Fair0.97

  • 1:27 #6 done32 line(s)

    shot 6·sharpness 2381.8

    1. AlEngineer0.99
    2. Evals are growing quickly0.98
    3. World'sFair1.00
    4. 100M1.00
    5. reddit1.00
    6. Booking.com1.00
    7. Uber1.00
    8. evals run every month1.00
    9. Tripadvisor1.00
    10. AATLASSIAN0.99
    11. owayfair0.94
    12. 3,800+1.00
    13. eval jobs run by the top Al teams1.00
    14. DOORDASH1.00
    15. klaviyo"0.98
    16. PagerDuty1.00
    17. 12.31.00
    18. Typeform1.00
    19. priceline1.00
    20. Handshake1.00
    21. eval jobs the average team runs0.97
    22. AlEngineer-0.94
    23. World'sF1.00
    24. Λ arize0.89
    25. 02 / 100.91
    26. air1.00
    27. Akama1.00
    28. Evals Track0.96
    29. AlEngineer0.97
    30. orld's F0.92
    31. Apara Dhinakaran / Co-founder & Chief Product Officer0.98
    32. arize0.98

  • 1:57 #7 done29 line(s)

    shot 7·sharpness 2696.5

    1. Evals were going to solve everything0.99
    2. AlEngineer1.00
    3. World'sFair1.00
    4. Kevin Weil1.00
    5. Writing evals is going to become a core skill for1.00
    6. product managers.1.00
    7. Mike Krieger1.00
    8. Writing evals is probably the most important thing we1.00
    9. can teach people.1.00
    10. We added evals1.00
    11. So they'll catch the failures, right?0.99
    12. Garry Tan0.95
    13. Evals are emerging as the real moat for Al startups.1.00
    14. Greg Brockman1.00
    15. Al0.82
    16. AlEngineer -0.94
    17. evals are surprisingly often all you need1.00
    18. They'll catch the failures, right?1.00
    19. World'sF1.00
    20. arize0.97
    21. -air0.84
    22. arize0.82
    23. Akama0.98
    24. Evals Track0.96
    25. Engineer1.00
    26. DOG0.82
    27. forld'sF0.95
    28. Apama Dhinakaran / Co-founder & Chief Product Officer0.97
    29. arize1.00

  • 2:30 #8 done35 line(s)

    shot 8·sharpness 1919.9

    1. AlEngineer0.98
    2. Agents got more complex0.99
    3. World's Fair0.99
    4. Run sub-agents1.00
    5. over long horizons1.00
    6. Research & code on0.98
    7. their own1.00
    8. Operate a computer1.00
    9. Reason1.00
    10. step-by-step1.00
    11. Use tools & browse1.00
    12. the web1.00
    13. Call functions & run1.00
    14. code1.00
    15. Answer a prompt1.00
    16. GFT-40.62
    17. interpreter0.97
    18. Workflows0.99
    19. -air0.90
    20. Open/0.91
    21. 20230.95
    22. 20241.00
    23. 20251.00
    24. 20261.00
    25. TIME1.00
    26. arize0.96
    27. AlEngineer -0.95
    28. er.r0.81
    29. harlae0.71
    30. orld'sF1.00
    31. Evals Track1.00
    32. Fair0.90
    33. DATAD0.99
    34. Apama Dhinakaran / Co-founder & Chief Product Officer0.98
    35. arize1.00

  • 3:07 #9 done132 line(s)

    shot 9·sharpness 3023.1

    1. copilot-prod0.98
    2. Trace: c969d7d3... caa33138 Session:488250.96
    3. AlEngineer0.99
    4. Failures got more complex1.00
    5. Signal 27 Evals & Metrics 50.98
    6. Status OK Cost $0.364943 Start Time 6/26/2026, 11:51:26 AM0.97
    7. World's Fair0.97
    8. Arize Default0.96
    9. Q0.98
    10. Trace Tree 411.00
    11. Search by span name1.00
    12. Timeline Agent Graph1.00
    13. AGENT ROUTER-TRAC0.98
    14. Jalbreak Detection: saf0.94
    15. We saw this with our own agent Alyx.1.00
    16. 00 Alyx0.77
    17. Traces 4.13k Spans 52.92k0.98
    18. 80.98
    19. ROUTER-TRACE_AGENT1.00
    20. 1 eval 62.83s0.98
    21. Attributes1.00
    22. Input / Out0.94
    23. one_alyx_agent1.00
    24. Q0.98
    25. Search1.00
    26. 62.54s1.00
    27. Context1.00
    28. seen late or lost in long run1.00
    29. 4.13k traces1.00
    30. Orchestrator - Runlteration 10.99
    31. 3.1s, 30,229 tk, $0.1051580.99
    32. input 1 item0.98
    33. value 3 items0.98
    34. Tool wrong tool, bad args, stale schema1.00
    35. Status1.00
    36. Kind1.00
    37. AGENT1.00
    38. get_trace_data1.00
    39. 0.15s1.00
    40. GQL. query GqISpanRecords..0.95
    41. input_context 70.98
    42. type1.00
    43. question1.00
    44. Execution1.00
    45. sandbox ≠ prod; deps, timeouts0.97
    46. G0.60
    47. AGENT1.00
    48. 0.08s1.00
    49. trace_ids Arra0.98
    50. 51.00
    51. Permission blocked, over-permitted, unsafe0.99
    52. Recovery1.00
    53. Loops1.00
    54. Model-harness fit0.98
    55. repeats errors, false success1.00
    56. Repeats the same step1.00
    57. differs across models1.00
    58. AGENT1.00
    59. AGENT1.00
    60. AGENT1.00
    61. AGENT1.00
    62. AGENT1.00
    63. AGENT1.00
    64. Orchestrator - Runlteration 30.99
    65. 3s, 31,525 tk, $0.0126240.99
    66. Orchestrator - Runlteration 20.99
    67. 2.69s, 31,116 tk, $0.0120540.99
    68. Orchestrator - Runlteration 40.99
    69. 0.01s0.95
    70. grep_ison0.90
    71. 0.02s1.00
    72. iq0.98
    73. • tracing_page0.92
    74. • span_ids Arra0.97
    75. highlighted_cc0.88
    76. • dataset 5 ite0.92
    77. [0]1.00
    78. [0]1.00
    79. page1.00
    80. project_id1.00
    81. view1.00
    82. G0.59
    83. AGENT1.00
    84. 1.89s, 31,700 tk, $0.0117391.00
    85. startTime1.00
    86. rld'sFai1.00
    87. AlEngineer0.95
    88. Λ arize0.88
    89. 81.00
    90. Trajectory1.00
    91. False Done claims success midway0.99
    92. ran, but not effectively1.00
    93. G0.52
    94. G0.53
    95. G0.56
    96. AGENT1.00
    97. AGENT1.00
    98. AGENT0.98
    99. AGENT1.00
    100. AGENT1.00
    101. AGENT1.00
    102. AGENT1.00
    103. 40.80
    104. Orchestrator - Runlteration 50.99
    105. 2.13s, 32,071 tk, $0.0121140.99
    106. 1.7s, 32,236 tk, $0.0116190.95
    107. Orchestrator - Runlteration 61.00
    108. Orchestrator - Runlteration 70.98
    109. 0.01s0.99
    110. 0.01s1.00
    111. 0.01s0.99
    112. iq0.97
    113. grep_jison0.93
    114. iq0.97
    115. • calumnVisib0.89
    116. endTime1.00
    117. subQuery1.00
    118. name1.00
    119. input_val1.00
    120. timeZone1.00
    121. queryFilte0.99
    122. attributes0.99
    123. output_v1.00
    124. otart tim0.91
    125. oget0.99
    126. porlze0.63
    127. Wol0.93
    128. Evals Track1.00
    129. AlEngi0.99
    130. rld'sF0.97
    131. Aparma Dhinakaran/ Co-founder & Chief Product Officer0.99
    132. arize1.00

  • 3:30 #10 skipped

    shot 10·duplicate of #9

  • 3:46 #11 done17 line(s)

    shot 11·sharpness 2444.3

    1. AlEngineer0.98
    2. World'sFair1.00
    3. The best way to evaluate an1.00
    4. agent is another agent1.00
    5. Engineer1.00
    6. Op1.00
    7. Id'sFair0.98
    8. Λ arize0.89
    9. AIE1.00
    10. gether0.94
    11. Worl1.00
    12. Evals Track1.00
    13. Engineer1.00
    14. Id'sFa0.97
    15. D/0.93
    16. Aparma Dhinakaran / Co-founder & Chief Product Officer0.96
    17. arize1.00

  • 4:25 #12 done47 line(s)

    shot 12·sharpness 3019.4

    1. AlEngineer1.00
    2. Different Evals for Different Failures0.99
    3. World's Fair0.97
    4. Deterministic code1.00
    5. Prompt hallucinations1.00
    6. Trajectory & harness failures1.00
    7. PRESENTED BY1.00
    8. Code Evaluators0.99
    9. Classic LLM-as-Judge1.00
    10. Agent-as-Judge1.00
    11. Microsoft1.00
    12. Unit tests, assertions, rules.1.00
    13. Score a known, well-defined failure.0.98
    14. Another agent reviews the whole trajectory.1.00
    15. input1.00
    16. output1.00
    17. criteria1.00
    18. traces0.97
    19. 0000.65
    20. flesystem1.00
    21. skills0.99
    22. valid_json (output)0.97
    23. / status == 2000.94
    24. / has_required_fields()0.97
    25. LLM1.00
    26. tools1.00
    27. Agent1.00
    28. subagents1.00
    29. no_pii_leaked(text)0.97
    30. AlEngine0.97
    31. PAsS 4 / 4 assertions - 0.02s0.90
    32. nAl0.98
    33. World's1.00
    34. label·score·explanation0.98
    35. insights1.00
    36. failure modes0.99
    37. evidence1.00
    38. Λarize0.92
    39. eer1.00
    40. sFai0.88
    41. Akal0.97
    42. Evals Track1.00
    43. AlEngine0.93
    44. ADOC0.94
    45. World's1.00
    46. Apama Dhinakaran / Co-founder & Chief Product Officer0.97
    47. arize1.00

  • 4:57 #13 done80 line(s)

    shot 13·sharpness 2502.0

    1. AlEngineer0.98
    2. Arize Signal1.00
    3. World's Fair0.96
    4. copilot-prod0.97
    5. Jun 19 - Jun 261.00
    6. 4 Live0.87
    7. Online Evaluators1.00
    8. Signal 271.00
    9. Evals & Metrics 50.97
    10. Traces1.00
    11. Spans1.00
    12. Sessions Agent Graph Agent Path1.00
    13. Signal Issues Active0.95
    14. arize-ai/arize0.98
    15. Viewing Signa's last 30 days of issues0.98
    16. ..0.78
    17. PRESENTED BY1.00
    18. Issues Found1.00
    19. Issues Over Time1.00
    20. Entity1.00
    21. Count1.00
    22. Microsoft1.00
    23. 271.00
    24. ROUTER-HOME_PAGE1.00
    25. run_mapping_fix_sub_agent0.98
    26. playground_agent1.00
    27. search_agent1.00
    28. Jun 90.98
    29. Jun 100.99
    30. Jun 140.99
    31. Jun 151.00
    32. Jun 161.00
    33. Jun 170.95
    34. Jun 190.99
    35. Jun 220.99
    36. Jun 240.99
    37. Jun 271.00
    38. Jun 281.00
    39. Jun 291.00
    40. Jun 301.00
    41. Detected Issues0.98
    42. todo_update retry loop on already-in-progress todo causes stream cancellation0.99
    43. Mute1.00
    44. Mark as fixed1.00
    45. #1 todo_update retry loop on already-in-0.99
    46. todo_update1.00
    47. 6/17|2026, 6:24 AM0.97
    48. progress todo causes stream cancellation0.99
    49. A'ROUTER-PROMPT_OPTIMIZATION' session0.95
    50. Next Steps Add to Dataset1.00
    51. + Create Evaluator0.99
    52. Open PR0.99
    53. ended with 'status_code=ERROR' and..0.96
    54. 6 matching traces1.00
    55. Overview1.00
    56. Al0.79
    57. World's1.00
    58. AlEngineer0.99
    59. #2 ROUTER-HOME_PAGE generates over-0.99
    60. verbose responses causing 20-75s latency0.99
    61. item cancels the parent stream.0.99
    62. A ROUTER-PROMPT_OPTIMIZATION session ended with status_code=ERROR and status_message=stream_cancelled . The export contains 100.99
    63. todo_update spans across the batch, and this is the same pattern previously identified where a todo_update retry loop on an already-in-progress0.99
    64. 83.57s end-to-end. The elevated latency in this run.0.98
    65. Reconfirmed: a ROUTER-HOME_PAGE session took0.99
    66. Evidence1.00
    67. • trace 1a9d4cd945573e097b4471e3dc7046cd — ROUTER-PROMPT_OPTIMIZATION, ERROR, stream_cancelled. User was working on prompt0.99
    68. arize1.00
    69. 33 matching traces1.00
    70. integration (partial message visible in attributes).0.99
    71. Proposed fix1.00
    72. Fa1.00
    73. arize0.81
    74. Akam1.00
    75. Evals Track1.00
    76. AlEngineer0.96
    77. DOG1.00
    78. Vorld's1.00
    79. Apara Dhinakaran/ Co-founder & Chief Product Officer0.98
    80. arize1.00

  • 5:31 #14 done18 line(s)

    shot 14·sharpness 2889.4

    1. AlEngineer1.00
    2. World's Fair0.98
    3. Λ arize0.92
    4. PRESENTED BY1.00
    5. Microsoft1.00
    6. Trace. Eval. Learn0.97
    7. -AlEngineer0.99
    8. rld's Fai0.96
    9. Aparna Dhinakaran1.00
    10. @aparnadhinak1.00
    11. tarlze0.69
    12. Oso0.68
    13. Evals Track1.00
    14. ite1.00
    15. gineer1.00
    16. I'sFai0.95
    17. Aparna Dhinakaran / Co-founder & Chief Product Officer0.97
    18. arize1.00

  • 5:45 #15 done10 line(s)

    shot 15·sharpness 362.6

    1. AlEngineer0.97
    2. OpenAl0.92
    3. rld's Fair0.96
    4. AlEngineer0.99
    5. osoft1.00
    6. orld's Fair0.98
    7. arize1.00
    8. ngineer1.00
    9. Buildkite1.00
    10. I'sFair0.94

  • 6:01 #16 skipped

    shot 16·duplicate of #0

Transcript

69 cues· 968 words· 5,196 chars

  1. 0:12 Awesome.
  2. 0:13 Well, hey, everyone.
  3. 0:14 My name is Aparna, one of the founders of Arise.
  4. 0:16 We work with some amazing teams to help them build evals.
  5. 0:21 And we have an incredible lineup of talks for you all today at the evals track.
  6. 0:26 It's happening in room 2005.
  7. 0:29 And there's going to be amazing speakers from Turnbench and Uber and Snorkel all happening after this.
  8. 0:35 But today, I'm here to talk to you about the future of evals.
  9. 0:38 evals have gone from the new skill that every PM and every AI engineer has to learn to the thing that every serious AI team is betting on.
  10. 0:50 We've been really fortunate to get to work with some of the best AI teams in the world.
  11. 0:54 So we get a front row seat into not just what's happening when they're building their actual agents and before they actually ship, but actually the evals the teams are running on their live production agent via their traces.
  12. 1:09 A little bit of some stats for you guys.
  13. 1:10 We run over 100 million evals every month.
  14. 1:13 The average team runs about 12 different eval jobs, with the top teams running over 3,800 different evaluators.
  15. 1:22 And offline evals, online evals, they each have their own place, but today what I'm actually gonna talk to you about is the teams that are running evals on their traces.
  16. 1:32 This is actually what's helping teams figure out what's working, catch their failures, and that's the type of data you need to fuel your continual learning loops.
  17. 1:43 And the industry kind of agrees.
  18. 1:45 I mean, all the CPOs of Anthropic, OpenAI, all, you know, GDB, you have Gary Tan saying, evals are everything you need.
  19. 1:53 And the whole industry kind of agrees.
  20. 1:55 So we added evals, they catch all the failures, right?
  21. 2:00 Here's the problem.
  22. 2:01 While we were building all of these first gen evals, the thing that we were actually evaluating has changed underneath us.
  23. 2:09 In 2023, it was about just answering a prompt.
  24. 2:12 In 2024, we started to see all the frontier models.
  25. 2:16 They've added tool calls, they've added reasoning, they've added deep research.
  26. 2:21 Now what we have is teams running loops on real world data with sub-agents kicked off on long horizon tasks.
  27. 2:30 Every one of these was actually a massive jump in complexity and we didn't just make the problem harder, we actually got a fundamentally different type of problem.
  28. 2:41 What that meant is that as these systems got more complex, so did the way that they actually fail.
  29. 2:46 We're really lucky because we have our own agent that we've built, Alex, that lives in our UI.
  30. 2:50 And we kind of get to feel this pain ourselves.
  31. 2:54 Every time the Frontier Labs added new functionality, we added it to our agent.
  32. 2:58 And now Alex has much longer memory.
  33. 3:03 It has the ability to create dynamic UIs.
  34. 3:05 It can go search across an enormous volume of traces.
  35. 3:09 But we also realized that it would forget context.
  36. 3:12 It wouldn't know when something was done.
  37. 3:14 Sometimes it would just get stuck in these loops.
  38. 3:17 And the key thing here is that the classical LLM as a judge evals that probably many of you have written in this room just weren't enough for us to be able to catch all the types of failures that we were experiencing.
  39. 3:32 I mean, it's just fundamentally different, right?
  40. 3:34 You have a deterministic flow and now what we have is literally every time a user interacted with Alex, it would create a new UI.
  41. 3:42 That's a fundamentally different trajectory.
  42. 3:45 So this led to our really big revelation.
  43. 3:48 What if the best way to evaluate an agent was actually with an agent?
  44. 3:54 Doesn't mean that all of the ways that we did evals with deterministic evals, with LLM as a judge, classic evals, doesn't matter anymore, but it just means that we have a different type of tool to solve a different type of problem.
  45. 4:07 Agent as a judge is about adaptive dynamic analysis.
  46. 4:12 LLM as a judge just gives you a fixed rubric with these fixed scores.
  47. 4:16 It's what everyone's doing, but when your agent's doing
  48. 4:19 completely different trajectories every time a user puts in data, it just means that you need a fundamentally different type of eval.
  49. 4:27 My take is that most teams today are doing the first two, but the future of evals is actually having all three.
  50. 4:35 And today I'm actually excited to share we've released Agent as a Judge to help our teams on their eval journey.

Chapters

  1. 0:00 Opening: the future of the Evals track
  2. 2:06 Why evals got harder as agents evolved
  3. 3:45 From deterministic checks to LLM as a judge
  4. 4:36 Agent as a judge, and where evals go next

Open at this second