vidtheque.

read-only demo

Videos 4VhbYlfC7Gs

Malleable Evals: Why Are We Evaluating Adaptive Systems with Static Tests? — Vincent Koc, OpenClaw

index_state ready data_status ok

AI Engineer· published 2026-05-12· 0:15:05· en-US· indexed 2026-08-10 22:30

Open on YouTube

Scene timeline

  1. Shot 0, 0:00 to 0:05, 1 of 1 keyframes kept
  2. Shot 1, 0:05 to 0:09, 1 of 1 keyframes kept
  3. Shot 2, 0:09 to 0:14, 1 of 1 keyframes kept
  4. Shot 3, 0:14 to 0:33, 1 of 1 keyframes kept
  5. Shot 4, 0:33 to 0:40, 1 of 1 keyframes kept
  6. Shot 5, 0:40 to 0:42, 1 of 1 keyframes kept
  7. Shot 6, 0:42 to 1:14, 1 of 1 keyframes kept
  8. Shot 7, 1:14 to 1:55, 1 of 1 keyframes kept
  9. Shot 8, 1:55 to 2:21, 1 of 1 keyframes kept
  10. Shot 9, 2:21 to 2:47, 1 of 1 keyframes kept
  11. Shot 10, 2:47 to 3:14, 0 of 1 keyframes kept
  12. Shot 11, 3:14 to 3:40, 1 of 1 keyframes kept
  13. Shot 12, 3:40 to 4:10, 1 of 1 keyframes kept
  14. Shot 13, 4:10 to 4:58, 1 of 1 keyframes kept
  15. Shot 14, 4:58 to 5:38, 1 of 1 keyframes kept
  16. Shot 15, 5:38 to 6:03, 1 of 1 keyframes kept
  17. Shot 16, 6:03 to 6:29, 1 of 1 keyframes kept
  18. Shot 17, 6:29 to 6:55, 1 of 1 keyframes kept
  19. Shot 18, 6:55 to 7:32, 1 of 1 keyframes kept
  20. Shot 19, 7:32 to 7:59, 1 of 1 keyframes kept
  21. Shot 20, 7:59 to 8:27, 1 of 1 keyframes kept
  22. Shot 21, 8:27 to 9:04, 1 of 1 keyframes kept
  23. Shot 22, 9:04 to 9:41, 0 of 1 keyframes kept
  24. Shot 23, 9:41 to 10:04, 1 of 1 keyframes kept
  25. Shot 24, 10:04 to 10:10, 0 of 1 keyframes kept
  26. Shot 25, 10:10 to 10:53, 0 of 1 keyframes kept
  27. Shot 26, 10:53 to 11:21, 1 of 1 keyframes kept
  28. Shot 27, 11:21 to 11:54, 1 of 1 keyframes kept
  29. Shot 28, 11:54 to 12:27, 0 of 1 keyframes kept
  30. Shot 29, 12:27 to 12:55, 0 of 1 keyframes kept
  31. Shot 30, 12:55 to 13:22, 0 of 1 keyframes kept
  32. Shot 31, 13:22 to 13:47, 1 of 1 keyframes kept
  33. Shot 32, 13:47 to 14:18, 1 of 1 keyframes kept
  34. Shot 33, 14:18 to 14:50, 1 of 1 keyframes kept
  35. Shot 34, 14:50 to 15:04, 1 of 1 keyframes kept

35 shot(s).

keyframes kept every frame deduplicated

What was stored

cues
156
whisperx 156
chunks
26
from 156 cues
keyframes
28
kept of 35 captured
frames with text
28
344 lines read
chapters
0
from the source metadata
keyframe bytes
3.0 MB
word timings on 156 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage state model started took
fetch done — 2026-08-10 22:27 1m 10s
stt done — 2026-08-10 22:29 17s
chunk done — 2026-08-10 22:29 0s
text_embed done — 2026-08-10 22:29 1s
keyframe done — 2026-08-10 22:29 1m 13s
ocr done — 2026-08-10 22:30 8s
frame_embed done — 2026-08-10 22:30 4s

Frames, and what the machine read

  • 0:03 #0 done2 line(s)

    shot 0·sharpness 658.4

    1. AlEngineer0.98
    2. EUROPE1.00

  • 0:08 #1 done2 line(s)

    shot 1·sharpness 829.1

    1. PRESENTINGSPONSOR1.00
    2. Google DeepMind1.00

  • 0:13 #2 done3 line(s)

    shot 2·sharpness 907.7

    1. PLATINUM SPONSORS0.98
    2. # Braintrust0.96
    3. WorkOS OpenAI0.95

  • 0:31 #3 done5 line(s)

    shot 3·sharpness 217.5

    1. comet1.00
    2. Malleable Evals: From Static AI to0.99
    3. asuring Adaptive Systems0.99
    4. Al0.91
    5. Improv0.89

  • 0:37 #4 done10 line(s)

    shot 4·sharpness 2155.9

    1. comet1.00
    2. ***0.62
    3. Malleable Evals: From Static AI to1.00
    4. AIE1.00
    5. ★1.00
    6. ★1.00
    7. Measuring Adaptive Systems1.00
    8. Vincent Koc (@vincent_koc)0.99
    9. Comet1.00
    10. Google DeepMind1.00

  • 0:41 #5 done10 line(s)

    shot 5·sharpness 1883.3

    1. Hi I'm Vincent Koc1.00
    2. Your Friendly Clanker1.00
    3. AIE1.00
    4. ★1.00
    5. ★1.00
    6. AIE Europe 20261.00
    7. Vincent Koc1.00
    8. comet1.00
    9. Google DeepMind1.00
    10. AlEngineei0.94

  • 1:10 #6 done13 line(s)

    shot 6·sharpness 2436.0

    1. Hi I'm Vincent Koc1.00
    2. Your Friendly Clanker1.00
    3. My love for testing new0.97
    4. technology (2013)0.99
    5. AIE1.00
    6. ★1.00
    7. ★1.00
    8. AIE Europe 20260.99
    9. Vincent Koc1.00
    10. comet1.00
    11. Braintrust1.00
    12. WorkOS OpenAI0.94
    13. AIE0.99

  • 1:22 #7 done25 line(s)

    shot 7·sharpness 2676.5

    1. Hamel Husain1.00
    2. @HamelHusain1.00
    3. Recap of tech ragebait I've seen in the last 24 hours0.99
    4. *★*0.66
    5. ★1.00
    6. - Serious work doesn't get done remotely0.98
    7. AIE1.00
    8. - Evals are dead0.99
    9. ★1.00
    10. - A/B testing is dead0.97
    11. ★1.00
    12. ★1.00
    13. ★1.00
    14. - If you aren't using an agent framework in favor of while loops it's just1.00
    15. hubris1.00
    16. - You can replace the PyData ecosystem only if there was pandas for1.00
    17. typescript1.00
    18. Did I miss anything0.98
    19. 7:14 AM · Sep 7, 2025·26KViews0.93
    20. AIE Europe 20260.99
    21. Vincent Koc0.98
    22. comet1.00
    23. AlEngineer0.97
    24. EUROPE1.00
    25. AIE0.99

  • 2:15 #8 done11 line(s)

    shot 8·sharpness 2120.4

    1. Example Focused Unit Tests1.00
    2. AIE1.00
    3. Manual Regression Suites1.00
    4. ★1.00
    5. ★1.00
    6. AIE Europe 20261.00
    7. Vincent Koc1.00
    8. comet1.00
    9. AlEngineer0.97
    10. EUROPE1.00
    11. AIEr.0.84

  • 2:42 #9 done11 line(s)

    shot 9·sharpness 2697.6

    1. Example Focused Unit Tests0.98
    2. AIE1.00
    3. Manual Regression Suites1.00
    4. ★1.00
    5. CI/CD Pipelines0.98
    6. Chaos Engineering and Observability1.00
    7. Vincent Koc1.00
    8. comet1.00
    9. AIE Europe 20261.00
    10. Engineering the future of Al0.99
    11. AIEn0.79

  • 3:10 #10 skipped

    shot 10·duplicate of #9

  • 3:37 #11 done13 line(s)

    shot 11·sharpness 2557.5

    1. StaticBenchmarks1.00
    2. AIE1.00
    3. Hand Curated Evaluations1.00
    4. ★1.00
    5. ★1.00
    6. Pre-Deployment Offline Evaluations0.99
    7. ??(gap)0.98
    8. AIE Europe 20261.00
    9. Vincent Koc1.00
    10. comet1.00
    11. AlEngineer0.98
    12. EUROPE1.00
    13. AIE1.00

  • 4:06 #12 done6 line(s)

    shot 12·sharpness 226.8

    1. Static Benchmarks0.99
    2. Hand Curated Evaluations1.00
    3. Pre-Deployment Offline Evaluations0.99
    4. ??(gap)1.00
    5. AIE0.94
    6. improy0.91

  • 4:34 #13 done12 line(s)

    shot 13·sharpness 2580.0

    1. StaticBenchmarks1.00
    2. AIE1.00
    3. Hand Curated Evaluations1.00
    4. ★1.00
    5. ★1.00
    6. Pre-Deployment Offline Evaluations1.00
    7. ??(gap)1.00
    8. AIE Europe 20260.99
    9. Vincent Koc1.00
    10. comet1.00
    11. Engineering the future of Al0.99
    12. AlEng.0.89

  • 5:22 #14 done35 line(s)

    shot 14·sharpness 4211.4

    1. ADAPTIVE TESTING FOR LLM EVALUATION:1.00
    2. A PSYCHOMETRIC ALTERNATIVE TO STATIC BENCHMARKS0.99
    3. A PREPRINT0.99
    4. ★1.00
    5. Si Chen, Ying Cheng, Ronald Metoyer, Ting Hua, Nitesh V. Chawla0.99
    6. Peiyu Li*, Xiuxiu Tang',0.93
    7. AIE1.00
    8. ★1.00
    9. Notre Dame, IN 46556 USA0.99
    10. University of Notre Dame0.98
    11. ★1.00
    12. {pli9, xtang8, schen34, ycheng4, rmetoyer, thua, nchawla}@nd.edu0.99
    13. ★1.00
    14. ★1.00
    15. ★1.00
    16. ABSTRACT1.00
    17. Evaluating large language models (LLMs) typically requires thousands of benchmark items, making0.99
    18. the process expensive, slow, and increasingly impractical at scale. Existing evaluation protocols rely0.99
    19. on average accuracy over fixed item sets, treating all items as equally informative despite substantial0.99
    20. variation in difficulty and discrimination. We introduce ATLAS, an adaptive testing framework based1.00
    21. on Item Response Theory (IRT) that estimates model ability using Fisher information-guided item0.99
    22. selection. ATLAS reduces the number of required items by up to 90% while maintaining measurement0.99
    23. precision. For instance, it matches whole-bank ability estimates using only 41 items (0.157 MAE)1.00
    24. on HellaSwag (5,600 items). We further reconstruct accuracy from ATLAS's ability estimates1.00
    25. and find that reconstructed accuracies closely match raw accuracies across all five benchmarks,0.99
    26. discrimination within accuracy-equivalent models: among more than 3,000 evaluated models, 23-1.00
    27. indicating that ability θ preserves the global performance structure. At the same time, θ provides finer0.99
    28. 31% shift by more than 10 rank positions, and models with identical accuracies receive meaningfully0.99
    29. different ability estimates. Code and calibrated item banks available at https://github. com/0.99
    30. AIE Europe 20261.00
    31. Vincent Koc1.00
    32. Peiyu-Georgia-Li/ATLAS.git.1.00
    33. comet1.00
    34. Engineering the future of Al0.98
    35. AIEr0.88

  • 5:56 #15 done20 line(s)

    shot 15·sharpness 2707.8

    1. The Emerging Dawn of Intentful AI Agents0.98
    2. An Evolution That's Leading to Self-Optimizing AI1.00
    3. AIE1.00
    4. ★1.00
    5. ★1.00
    6. ★1.00
    7. The sunsetting of0.97
    8. Prompt Engineering1.00
    9. (2022-23)1.00
    10. Door closes on the troubled1.00
    11. process of writing prompts1.00
    12. using trial and error "gut feel.0.95
    13. Killing the cycle of doom0.98
    14. wordsmithing instructions.1.00
    15. Vincent Koc1.00
    16. comet1.00
    17. AIE Europe 20261.00
    18. AlEngineer0.96
    19. EUROPE1.00
    20. AIEI0.89

  • 6:26 #16 done29 line(s)

    shot 16·sharpness 3212.2

    1. The Emerging Dawn of Intentful AI Agents0.99
    2. An Evolution That's Leading to Self-Optimizing AI1.00
    3. AIE1.00
    4. ★1.00
    5. ★1.00
    6. ★1.00
    7. ★1.00
    8. The sunsetting of0.97
    9. The evolution of0.98
    10. Prompt Engineering1.00
    11. Context Engineering1.00
    12. (2022-23)1.00
    13. (2024-25)1.00
    14. Door closes on the troubled1.00
    15. We can now somewhat steer agents,1.00
    16. process of writing prompts1.00
    17. a with RAG, reasoning models,0.99
    18. using trial and error "gut feel.0.95
    19. long context windows, MCP servers1.00
    20. Killing the cycle of doom0.98
    21. and smart long running models. It's1.00
    22. wordsmithing instructions.1.00
    23. about managing memory efficently.1.00
    24. Vincent Koc1.00
    25. comet1.00
    26. AIE Europe 20261.00
    27. AlEngineer0.96
    28. EUROPE1.00
    29. AIE1.00

  • 6:49 #17 done30 line(s)

    shot 17·sharpness 3243.2

    1. The Emerging Dawn of Intentful AI Agents0.99
    2. An Evolution That's Leading to Self-Optimizing AI1.00
    3. **0.89
    4. ★1.00
    5. AIE1.00
    6. ★1.00
    7. ★1.00
    8. ★1.00
    9. ★1.00
    10. The sunsetting of0.98
    11. The evolution of0.98
    12. Prompt Engineering1.00
    13. Context Engineering1.00
    14. (2022-23)1.00
    15. (2024-25)1.00
    16. Door closes on the troubled1.00
    17. We can now somewhat steer agents,1.00
    18. process of writing prompts0.98
    19. a with RAG, reasoning models,1.00
    20. using trial and error "gut feel.0.96
    21. long context windows, MCP servers1.00
    22. Killing the cycle of doom0.99
    23. and smart long running models. It's0.99
    24. wordsmithing instructions.1.00
    25. about managing memory efficently.1.00
    26. Vincent Koc1.00
    27. comet1.00
    28. AIE Europe 20260.96
    29. Engineering the future of Al0.99
    30. AlEngin0.91

  • 7:24 #18 done3 line(s)

    shot 18·sharpness 238.0

    1. The Emerging Dawn of Intentful AI Agents0.99
    2. AIEr0.78
    3. improy0.97

  • 7:56 #19 done9 line(s)

    shot 19·sharpness 1954.8

    1. AIE1.00
    2. Models Became Really Good0.98
    3. ★1.00
    4. ★1.00
    5. AIE Europe 20261.00
    6. Vincent Koc1.00
    7. comet1.00
    8. Engineering the future of Al1.00
    9. Al0.78

  • 8:08 #20 done10 line(s)

    shot 20·sharpness 2045.8

    1. AIE1.00
    2. Models Became Really Good0.99
    3. ★1.00
    4. ★1.00
    5. AIE Europe 20261.00
    6. Vincent Koc1.00
    7. comet1.00
    8. # Braintrust0.97
    9. WorkOS OpenAI0.96
    10. er1.00

  • 8:59 #21 done38 line(s)

    shot 21·sharpness 3698.7

    1. The Emerging Dawn of Intentful AI Agents0.99
    2. An Evolution That's Leading to Self-Optimizing AI1.00
    3. ★1.00
    4. ★1.00
    5. AIE1.00
    6. ★1.00
    7. ★1.00
    8. ★1.00
    9. ★1.00
    10. The sunsetting of0.98
    11. The evolution of0.97
    12. The dawn of1.00
    13. Prompt Engineering1.00
    14. Context Engineering1.00
    15. Intent Engineering1.00
    16. (2022-23)1.00
    17. (2024-25)1.00
    18. (2026)1.00
    19. Door closes on the troubled1.00
    20. We can now somewhat steer agents,1.00
    21. Machines can self-optimize based1.00
    22. process of writing prompts1.00
    23. a with RAG, reasoning models,1.00
    24. on the intent and needs of a user or0.99
    25. using trial and error “gut feel".0.97
    26. long context windows, MCP servers1.00
    27. system through approaches like0.98
    28. Killing the cycle of doom0.99
    29. and smart long running models. It's0.99
    30. online prompt optimization and1.00
    31. wordsmithing instructions.1.00
    32. about managing memory efficently.0.99
    33. self-optimizing models.1.00
    34. Vincent Koc1.00
    35. comet1.00
    36. AIE Europe 20261.00
    37. AlEngineer0.97
    38. EUROPE1.00

  • 9:33 #22 skipped

    shot 22·duplicate of #17

  • 9:52 #23 done11 line(s)

    shot 23·sharpness 2550.1

    1. StaticBenchmarks1.00
    2. AIE1.00
    3. Hand Curated Evaluations1.00
    4. ★0.99
    5. ★0.99
    6. Pre-Deployment Offline Evaluations1.00
    7. ??(gap)1.00
    8. AIE Europe 20261.00
    9. Vincent Koc1.00
    10. comet1.00
    11. Engineering the future of Al0.99

Transcript

156 cues· 2,876 words· 15,393 chars

  1. 0:15 cool hey everyone uh thanks for joining this session sorry if my sound's a little croaky i've done three talks back to back so one on wednesday one yesterday keynote and then uh workshop style session today so i'm vincent i'm going to be talking about malleable evals from static ai measuring to adaptive systems now
  2. 0:39 Let's jump into who I am, what I do.
  3. 0:41 I call myself the friendly canker, I use AI, I use technology, I'm always on the edge.
  4. 0:46 For those of you that haven't seen my keynote, I do, yeah, I just live on the edge and just do some fun stuff.
  5. 0:53 So this is me using VR goggles back in 2013 when people hadn't even heard of VR.
  6. 0:59 It came with a warning label, said only use it for five minutes, I used it for three hours, then I vomited for three hours after that.
  7. 1:05 So measurement, anything we do in technology, anything on the edge is going to be janky.
  8. 1:10 It's going to be weird.
  9. 1:12 And that's kind of fun, in my opinion.
  10. 1:14 Now, whenever we talk about evals to people and a little bit of pretext, like my role at Comet
  11. 1:21 I work in evals, I do in eval research, I work with universities, we benchmark and run evals for large set of companies and organizations, everything from like Uber to Netflix to banks even in the UK.
  12. 1:34 But the thing that's been going on right now is that, hey, like this kind of joke that like evals are a little bit dead.
  13. 1:40 And it's a little bit of a joke, but there's a little bit of truth to it as well.
  14. 1:44 And I'm going to hopefully like kind of walk you through the mindset shift and hopefully explain a little bit less about evals, but like what's actually happening in the sort of agentic AI space.
  15. 1:53 And then how do we then translate that back to evals?
  16. 1:56 So when we think about software engineering as a practice, when we're thinking about how do we measure things, we kind of look at it from the sense that we're going to start with this thing is meant to do something.
  17. 2:08 So we would start with a set of examples and write some unit tests.
  18. 2:12 We might do a manual regression suite, which is like, hey, when we do A and B, sometimes C happens, and C is unfavorable.
  19. 2:19 Let's not do that.
  20. 2:20 Let's not make people vomit when they put their VR goggles on.
  21. 2:24 We could do things like CI-CD pipelines to make sure that the thing ships out and works the way it's meant to and intended to.
  22. 2:32 But mostly we do things in engineering known as chaos engineering and observability.
  23. 2:37 For those of you that are unfamiliar with the term chaos engineering, it's basically where you're doing all kinds of random stuff and just breaking it and just having fun with the technology and just seeing where you can stretch it and where you can go.
  24. 2:49 Now, when we apply this to AI and data science space, as we traditionally know it in the last little while, 2025 included, we do things like static benchmarks.
  25. 3:00 We have these evaluations.
  26. 3:01 It's like, oh, I'll give an example.
  27. 3:04 It's like, how compliant is my AI in risk?
  28. 3:09 I'm gonna ask it a bunch of questions and make sure it doesn't talk about selling me some financial services, because that's a big no-no.
  29. 3:17 We will then handcraft a set of questions and examples and sit there and tune this thing up, make sure it's absolutely perfect.
  30. 3:26 Before we deploy the AI system or the models, we'll do some sort of offline evaluation where we're just cycling through those tests.
  31. 3:35 but we're missing that sort of chaos engineering space.
  32. 3:37 We're missing that, you know, like what comes next and how do we mess up with it?
  33. 3:42 And how do we know where we can stretch this thing?
  34. 3:43 And I think that's like an honest gap that we see in this space.
  35. 3:47 And that's why we're just so hyper fixated on benchmarks and evaluations.
  36. 3:51 If you go to any AI conference in the academic space, all people talk about is like benchmarks.
  37. 3:56 I created a benchmark for like,
  38. 3:59 adding numbers and what LLMs think about it.
  39. 4:01 It's like, well, great, but like, how is this actually helping me?
  40. 4:04 So then you end up with like this huge, humongous set of like data sets to try and somewhat explain what is happening with your agent until something goes wrong.
  41. 4:12 And it's a matter of time before something goes wrong and it will, and you're kind of back to the drawing board and trying to figure out what's going on.
  42. 4:20 And the reason for that is that our AI applications are not static, but we're treating them like they're static software.
  43. 4:27 Yes, when we ship software, we might change unit tests, they're a little bit quicker to do, but realistically speaking, even software is becoming malleable.
  44. 4:35 So,
  45. 4:36 Flip to my keynote I gave yesterday, where I'm one of the core contributors of something called OpenClaw, the harness changes itself, like the harness will shift, like you wanna create skills, you wanna do other things, like it will adapt, right?
  46. 4:48 So that adaption that we're seeing inside of things where software is being shipped at lightning speed, how does your benchmarks keep up with that?
  47. 4:55 Like how does your benchmarks adapt to that space?
  48. 4:58 This is one of many papers that are out there.
  49. 5:02 I don't remember when this one was published, but this concept of adaptive testing for LLM evals,
  50. 5:07 this concept is like somewhat revolutionary maybe, but like what happens if our benchmarks would change with our applications?

Open at this second