vidtheque.

read-only demo

Videos wEc9aG7cRQc

Why Your Agent Disagrees With Itself (And What To Do About It) - Diane Lin, Datadog

index_state ready data_status ok

AI Engineer· published 2026-07-20· 0:25:38· en-US· indexed 2026-08-10 19:46

Open on YouTube

Scene timeline

  1. Shot 0, 0:00 to 0:19, 1 of 1 keyframes kept
  2. Shot 1, 0:19 to 0:47, 1 of 1 keyframes kept
  3. Shot 2, 0:47 to 1:16, 0 of 1 keyframes kept
  4. Shot 3, 1:16 to 1:44, 0 of 1 keyframes kept
  5. Shot 4, 1:44 to 2:13, 0 of 1 keyframes kept
  6. Shot 5, 2:13 to 2:42, 1 of 1 keyframes kept
  7. Shot 6, 2:42 to 3:09, 1 of 1 keyframes kept
  8. Shot 7, 3:09 to 3:45, 1 of 1 keyframes kept
  9. Shot 8, 3:45 to 4:21, 0 of 1 keyframes kept
  10. Shot 9, 4:21 to 4:58, 0 of 1 keyframes kept
  11. Shot 10, 4:58 to 5:34, 0 of 1 keyframes kept
  12. Shot 11, 5:34 to 6:09, 0 of 1 keyframes kept
  13. Shot 12, 6:09 to 6:35, 1 of 1 keyframes kept
  14. Shot 13, 6:35 to 7:05, 0 of 1 keyframes kept
  15. Shot 14, 7:05 to 7:35, 0 of 1 keyframes kept
  16. Shot 15, 7:35 to 8:05, 0 of 1 keyframes kept
  17. Shot 16, 8:05 to 8:31, 1 of 1 keyframes kept
  18. Shot 17, 8:31 to 8:57, 0 of 1 keyframes kept
  19. Shot 18, 8:57 to 9:22, 0 of 1 keyframes kept
  20. Shot 19, 9:22 to 9:48, 0 of 1 keyframes kept
  21. Shot 20, 9:48 to 10:14, 1 of 1 keyframes kept
  22. Shot 21, 10:14 to 10:39, 0 of 1 keyframes kept
  23. Shot 22, 10:39 to 11:07, 1 of 1 keyframes kept
  24. Shot 23, 11:07 to 11:34, 0 of 1 keyframes kept
  25. Shot 24, 11:34 to 12:02, 0 of 1 keyframes kept
  26. Shot 25, 12:02 to 12:28, 1 of 1 keyframes kept
  27. Shot 26, 12:28 to 12:54, 0 of 1 keyframes kept
  28. Shot 27, 12:54 to 13:20, 0 of 1 keyframes kept
  29. Shot 28, 13:20 to 13:46, 0 of 1 keyframes kept
  30. Shot 29, 13:46 to 14:12, 0 of 1 keyframes kept
  31. Shot 30, 14:12 to 14:38, 0 of 1 keyframes kept
  32. Shot 31, 14:38 to 15:04, 1 of 1 keyframes kept
  33. Shot 32, 15:04 to 15:30, 0 of 1 keyframes kept
  34. Shot 33, 15:30 to 15:56, 0 of 1 keyframes kept
  35. Shot 34, 15:56 to 16:13, 1 of 1 keyframes kept
  36. Shot 35, 16:13 to 16:59, 1 of 1 keyframes kept
  37. Shot 36, 16:59 to 17:47, 1 of 1 keyframes kept
  38. Shot 37, 17:47 to 18:02, 0 of 1 keyframes kept
  39. Shot 38, 18:02 to 18:31, 1 of 1 keyframes kept
  40. Shot 39, 18:31 to 19:01, 0 of 1 keyframes kept
  41. Shot 40, 19:01 to 19:31, 1 of 1 keyframes kept
  42. Shot 41, 19:31 to 20:00, 0 of 1 keyframes kept
  43. Shot 42, 20:00 to 20:36, 1 of 1 keyframes kept
  44. Shot 43, 20:36 to 21:02, 1 of 1 keyframes kept
  45. Shot 44, 21:02 to 21:28, 0 of 1 keyframes kept
  46. Shot 45, 21:28 to 21:54, 0 of 1 keyframes kept
  47. Shot 46, 21:54 to 22:20, 0 of 1 keyframes kept
  48. Shot 47, 22:20 to 22:48, 1 of 1 keyframes kept
  49. Shot 48, 22:48 to 23:17, 0 of 1 keyframes kept
  50. Shot 49, 23:17 to 23:45, 0 of 1 keyframes kept
  51. Shot 50, 23:45 to 24:17, 1 of 1 keyframes kept
  52. Shot 51, 24:17 to 24:48, 0 of 1 keyframes kept
  53. Shot 52, 24:48 to 25:31, 1 of 1 keyframes kept
  54. Shot 53, 25:31 to 25:37, 1 of 1 keyframes kept

54 shot(s).

keyframes kept every frame deduplicated

What was stored

cues
185
whisperx 185
chunks
44
from 185 cues
keyframes
22
kept of 54 captured
frames with text
22
402 lines read
chapters
0
from the source metadata
keyframe bytes
6.3 MB
word timings on 185 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage state model started took
fetch done — 2026-08-10 03:43 1m 41s
stt done — 2026-08-10 03:45 25s
chunk done — 2026-08-10 03:45 0s
text_embed done — 2026-08-10 19:46 0s
keyframe done — 2026-08-10 03:45 38s
ocr done — 2026-08-10 03:46 9s
frame_embed done — 2026-08-10 19:46 4s

Frames, and what the machine read

  • 0:17 #0 done6 line(s)

    shot 0·sharpness 833.7

    1. Why Your Al Agent0.97
    2. Diane Lin0.99
    3. Disagrees With Itself0.98
    4. and What to Do About It0.99
    5. Dianhuan (Diane) Lin1.00
    6. DATADOG1.00

  • 0:39 #1 done23 line(s)

    shot 1·sharpness 1174.0

    1. Background1.00
    2. Education & Impact1.00
    3. PhD in Machine Learning, Imperial College London0.99
    4. Diane Lin1.00
    5. Research affiliations at MIT's CSAIL1.00
    6. Amazon (Alexa Team)0.99
    7. a0.99
    8. Founding applied scientist1.00
    9. Vicarious1.00
    10. (acquired by Gaogle DeepMind)0.98
    11. Senior Researcher1.00
    12. Zscaler1.00
    13. Director of1.00
    14. zscaler1.00
    15. Machine Learning1.00
    16. Culminate1.00
    17. (acquired by Datadog)1.00
    18. CTO & Co-founder0.97
    19. CULMINATE1.00
    20. Datadog1.00
    21. Tech Lead0.99
    22. Self-Evolving Al Agents0.98
    23. for the Modern SOC1.00

  • 1:04 #2 skipped

    shot 2·duplicate of #1

  • 1:35 #3 skipped

    shot 3·duplicate of #1

  • 2:06 #4 skipped

    shot 4·duplicate of #1

  • 2:39 #5 done7 line(s)

    shot 5·sharpness 1665.0

    1. Agenda1.00
    2. Diane Lin0.99
    3. The Problem: LLM inconsistency in practice0.99
    4. Where Does Inconsistency Come From?0.99
    5. Solutions1.00
    6. Experimental Results1.00
    7. DATADOG0.95

  • 2:56 #6 done4 line(s)

    shot 6·sharpness 386.7

    1. Diane Lin0.96
    2. The Problem1.00
    3. Same model. Same input. Different output.1.00
    4. DATADOG41.00

  • 3:41 #7 done14 line(s)

    shot 7·sharpness 4776.9

    1. LLM Inconsistency in Practice0.99
    2. Diane Lin0.95
    3. Sentiment analysis: classify hotel reviews as positive or negative0.99
    4. Review A0.98
    5. "Fantastic hotel in the perfect location. It's just a five-minute walk to all the major museums and restaurants, yet0.99
    6. tucked away on a quiet street so you don't hear any city noise at night. The concierge, David, gave us the best0.99
    7. recommendations for local dining that weren't tourist traps. Everything about this place is top-notch. Highly1.00
    8. recommend to anyone visiting the city."0.99
    9. Review B0.95
    10. "It's an okay hotel. It's clean and safe, which are the most important things, but it entirely lacks any charm or0.99
    11. personality. The check-in process was automated via a kiosk, so we never actually spoke to a human being during our0.99
    12. entire stay. The room had all the basics but felt very sterile, almost like a hospital room. If you just need a cheap,0.99
    13. clean place to crash before an early flight, it works fine. Just don't expect a memorable hospitality experience."1.00
    14. DATADOG 50.90

  • 4:13 #8 skipped

    shot 8·duplicate of #7

  • 4:43 #9 skipped

    shot 9·duplicate of #7

  • 5:13 #10 skipped

    shot 10·duplicate of #7

  • 5:41 #11 skipped

    shot 11·duplicate of #7

  • 6:29 #12 done5 line(s)

    shot 12·sharpness 613.0

    1. Where Does the1.00
    2. Diane Lin0.99
    3. Inconsistency Come From?1.00
    4. The gray zone: data that sit near the decision boundary1.00
    5. DATADOG70.99

  • 6:44 #13 skipped

    shot 13·duplicate of #7

  • 7:14 #14 skipped

    shot 14·duplicate of #7

  • 7:42 #15 skipped

    shot 15·duplicate of #7

  • 8:11 #16 done73 line(s)

    shot 16·sharpness 2595.3

    1. verdict_iter0 verdict_iter1 verdict_iter20.99
    2. Gray zone: data that sit near the de1.00
    3. suspicious1.00
    4. benign1.00
    5. benign1.00
    6. ry1.00
    7. Diane Lin1.00
    8. suspicious1.00
    9. suspicious1.00
    10. suspicious1.00
    11. suspicious1.00
    12. suspicious1.00
    13. suspicious1.00
    14. benign1.00
    15. suspicious1.00
    16. suspicious1.00
    17. Security Analysts triage alerts1.00
    18. benign1.00
    19. benign1.00
    20. benign1.00
    21. “Failed authentication attempts detected from suspicious IP"0.99
    22. benign1.00
    23. benign1.00
    24. benign1.00
    25. benign1.00
    26. benign1.00
    27. benign1.00
    28. suspicious1.00
    29. benign1.00
    30. benign1.00
    31. suspicious0.99
    32. benign1.00
    33. suspicious1.00
    34. benign1.00
    35. benign1.00
    36. benign1.00
    37. suspicious1.00
    38. suspicious1.00
    39. suspicious1.00
    40. benign1.00
    41. suspicious1.00
    42. suspicious1.00
    43. suspicious1.00
    44. suspicious1.00
    45. benign1.00
    46. benign1.00
    47. suspicious1.00
    48. benign1.00
    49. suspicious1.00
    50. benign1.00
    51. suspicious1.00
    52. suspicious1.00
    53. benign1.00
    54. benign1.00
    55. benign1.00
    56. benign1.00
    57. benign1.00
    58. suspicious1.00
    59. benign1.00
    60. benign1.00
    61. benign1.00
    62. benign1.00
    63. suspicious1.00
    64. benign1.00
    65. benign1.00
    66. benign1.00
    67. benign1.00
    68. benign1.00
    69. benign1.00
    70. DATADOG 90.96
    71. benign1.00
    72. benign1.00
    73. benign1.00

  • 8:41 #17 skipped

    shot 17·duplicate of #16

  • 9:12 #18 skipped

    shot 18·duplicate of #16

  • 9:45 #19 skipped

    shot 19·duplicate of #16

  • 10:11 #20 done4 line(s)

    shot 20·sharpness 1001.8

    1. Gray zone: data that sit near the decision boundary1.00
    2. Diane Lin0.99
    3. Gray Zone0.99
    4. DATADOG0.99

  • 10:22 #21 skipped

    shot 21·duplicate of #20

  • 10:45 #22 done5 line(s)

    shot 22·sharpness 596.2

    1. Identifying the Gray Zone:1.00
    2. Diane Lin1.00
    3. Active learning1.00
    4. Active learning tells us exactly where to look1.00
    5. DATADOG 110.95

  • 11:18 #23 skipped

    shot 23·duplicate of #22

Transcript

185 cues· 2,838 words· 16,372 chars

  1. 0:02 Hello.
  2. 0:04 This session is about why do your AI agents disagree with itself and what to do about it.
  3. 0:13 And this is a Dear Huang Lin, and you can call me Diane.
  4. 0:19 A little bit about my background.
  5. 0:22 My journey with AGI start from my PhD at Imperial College London, where I was working on continual learning.
  6. 0:31 Later, I worked with Professor Josh Tenenbaum at MIT on one-shot learning, meaning learning from one single example.
  7. 0:42 Afterwards, I was lucky that Alexa was launching.
  8. 0:46 I was the first three applying scientists on the question answering team at Alexa.
  9. 0:53 later attracted by a startup working on AGI at the Silicon Valley called Vicarious.
  10. 1:01 So today Vicarious is part of the Google demind.
  11. 1:05 There I got to work on the exciting frontier research on zero-shot transfer learning.
  12. 1:13 After working on AGI for a few years, I decided to pivot to the applied world at GSCADR, which is cybersecurity company.
  13. 1:25 There, I got to apply a variety of different machine learning model to address the cybersecurity challenge.
  14. 1:34 There for five years, I decided to co-found my own company because the pinpoint I have seen at GSCADR
  15. 1:44 So CloudMini is building AI agent to auto triage your security alerts and which make your SOG more efficient.
  16. 1:55 We are super proud that CloudMini has been acquired by Datadog earlier this year.
  17. 2:02 So now I'm part of the Datadog and leading the development of self-evolved agent
  18. 2:14 so today i'm going to share with you the problem you're probably seeing your day-to-day building ai agent the inconsistency and then we are discussed about where this inconsistency come from in order to figure out the solutions and we will discuss a few trade-offs among the different solutions and then eventually we'll show you the experimental results to show how feasible they are
  19. 2:46 You probably see this same model, same input, but different output.
  20. 2:52 Here, I don't mean the wording different.
  21. 2:55 I mean, semantically different output.
  22. 3:01 You might be thinking that's the stochastic nature of LLM.
  23. 3:07 Hold on to that thought.
  24. 3:09 Let's look at a few concrete examples.
  25. 3:13 So here, imagine you're giving a task for sentiment analysis, where we're supposed to label each review of the hotel either positive or negative.
  26. 3:26 This is a typical NLP task, and today's LLM is really good at this.
  27. 3:33 However, there's one problem.
  28. 3:36 When you pipe this input to the same LLM model,
  29. 3:42 Occasionally you will see different verdict coming out of it when you run multiple times.
  30. 3:49 That means a single evaluation run didn't tell you the whole story.
  31. 3:55 You need to repeat your evaluation multiple times in order to get a holistic picture.
  32. 4:05 By the average over the different runs results.
  33. 4:10 And this sounds like a technical inconvenience, but the reality is it's way more problematic than just inconvenience.
  34. 4:21 Let me show you an example in a different domain in cybersecurity.
  35. 4:26 Imagine you are a security analyst where your day-to-day job is triage the security alerts and decide whether it's malicious that you need to take certain actions to stop the attack or it's false alarm, meaning it's actually benign, which you could ignore.
  36. 4:47 So here, example alert you got is there's a failed logging attempt to Gmail account detected from suspicious IP.
  37. 4:58 And if we give such similar task to this one, to AI agent, and run it different times, you will see sometimes it stays consistently benign.
  38. 5:10 Sometimes it stays consistently suspicious across different lines.
  39. 5:15 However, there are cases where it flip-flops between benign and suspicious.
  40. 5:24 Now your customer will have a difficulty using such a product because they will wonder which one should I trust?
  41. 5:34 So the inconsistency is causing a trust issue in your product.
  42. 5:42 Imagine if you were in a POC bake-off, one vendor gave the consistent verdict all the time while the other
  43. 5:55 vendor have this flip-flop verdict you can imagine which one will win the deal now i'm pretty sure you want to solve this problem first we have to figure out where the inconsistency come from
  44. 6:21 The good news is the data points tend to flip-flop, actually concentrate around the decision boundary, the so-called gray zone.
  45. 6:34 Let me illustrate with the earlier example.
  46. 6:37 Here in the sentiment analysis, we have two reviews.
  47. 6:42 The first one
  48. 6:44 Actually, no matter how many times you run through the LL model, it will always give you the positive label.
  49. 6:54 In this case, it's pretty obvious the review is super positive.
  50. 7:00 However, the second case is the one tend to flip.

Open at this second