Videos imFedndyXYQ

Using LLMs to Secure Source Code — Eugene Yan, Anthropic

index_state ready data_status ok

AI Engineer· published 2026-07-17· 0:21:30· en-US· indexed 2026-08-11 01:05

Open on YouTube

50 shot(s).

keyframes kept every frame deduplicated

What was stored

cues: 306; whisperx 306
chunks: 37; from 306 cues
keyframes: 20; kept of 50 captured
frames with text: 20; 387 lines read
chapters: 21; from the source metadata
keyframe bytes: 6.1 MB; word timings on 306 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage	state	model	started	took
`fetch`	done	—	2026-08-11 01:01	1m 14s
`stt`	done	—	2026-08-11 01:02	25s
`chunk`	done	—	2026-08-11 01:03	0s
`text_embed`	done	—	2026-08-11 01:03	1s
`keyframe`	done	—	2026-08-11 01:03	2m 03s
`ocr`	done	—	2026-08-11 01:05	9s
`frame_embed`	done	—	2026-08-11 01:05	4s

Frames, and what the machine read

0:02 #0 done2 line(s)

shot 0·sharpness 449.7
1. AlEngineer0.96
2. World's Fair1.00
0:03 #1 done2 line(s)

shot 1·sharpness 662.0
1. AIEngineer0.95
2. World's Fair0.99
0:10 #2 done24 line(s)

shot 2·sharpness 2739.9
1. LAB & PLATINUM SPONSORS0.99
2. Amazon AGI Lab0.98
3. ANTHROP\C1.00
4. Google DeepMind1.00
5. MINIMAX0.97
6. OpenAI0.92
7. Akamai1.00
8. arize0.92
9. aws1.00
10. Braintrust bright data0.98
11. B1.00
12. Browserbase1.00
13. docker1.00
14. :neo4j0.93
15. ORACLE1.00
16. PayPal1.00
17. qodo1.00
18. reducto1.00
19. Sonar1.00
20. Makers of0.99
21. togetherai1.00
22. Unblocked1.00
23. WorkOS1.00
24. SonarQube1.00
0:17 #3 done2 line(s)

shot 3·sharpness 180.6
1. AlEngineer0.99
2. World's Fair0.99
0:36 #4 done11 line(s)

shot 4·sharpness 1526.9
1. AlEngineer0.99
2. World's Fair0.97
3. Working with Models1.00
4. PRESENTED BY1.00
5. to Secure Source Code0.99
6. Microsoft1.00
7. Eugene Yan·Anthropic1.00
8. AI Engineer World's Fair ·20260.98
9. ANTHROPIC1.00
10. World's Fair0.89
11. Engineering the future of Al0.99
0:49 #5 done11 line(s)

shot 5·sharpness 2326.5
1. AlEngineer0.97
2. What we'll cover0.97
3. World'sFair1.00
4. PRESENTED BY0.96
5. Why now: the trends we're seeing1.00
6. Microsoft1.00
7. How to work with models for security, and potential bottlenecks0.99
8. How to get started now + resources to help1.00
9. ANTHROPIC1.00
10. World'sFair1.00
11. Engineering the future of Al1.00
1:10 #6 done11 line(s)

shot 6·sharpness 947.9
1. AlEngineer0.98
2. World'sFair1.00
3. PRESENTED BY0.99
4. Sectionl0.93
5. Whynow1.00
6. Microsoft1.00
7. Three trends: Model capability, Vulns found, Shifting bottleneck1.00
8. ANTHROPIC1.00
9. World'sFair1.00
10. TRACK 5· JUNE 30, 20260.94
11. Security1.00
1:17 #7 done49 line(s)

shot 7·sharpness 1968.7
1. AlEngineer0.99
2. Model capability on cyber tasks is doubling every ~5 months0.99
3. World's Fair0.96
4. AISI Cybersecurity Time Horizons0.99
5. AISI IANSECUITY0.75
6. AISI CTF suite, 80% rellability, 2.5M token cap, reasoning models only0.98
7. INSTITUTE1.00
8. Previous AISI estimate (Nov 2025): -8mo doubling0.98
9. Frontier post-reasoning trend (pre-Mythos): -4.7mo doubiing0.98
10. Caveats:1.00
11. B% l le)0.65
12. 4d:0.92
13. - Models evaluated under a 2.5M token cap; uncapped time horizons would be0.98
14. • Narrow cyber tasks only; no claim about generalization to other capability0.97
15. significantly higher1.00
16. PRESENTED BY1.00
17. - Mythos Preview (new) and GPT-5.5 saturate the task suite, resulting in highly0.98
18. uncertain time horizons0.98
19. domains1.00
20. Microsoft1.00
21. 10.0h1.00
22. Mythos Preview (new)1.00
23. GPT-5.51.00
24. 1.0h1.00
25. Claude Opus 4.6*0.95
26. GPT-5.3-Codex1.00
27. Claude 4 Opus1.00
28. GPT-51.00
29. Claude Sonnet 4.51.00
30. -Claude Opus 4.50.97
31. 6m1.00
32. Claude 3.7 Sonnet1.00
33. Provider1.00
34. Anthropic1.00
35. OpenAl0.94
36. 2025-011.00
37. 2025-041.00
38. 2025-071.00
39. 2025-101.00
40. 2026-011.00
41. 2026-041.00
42. 2026-071.00
43. 2026-101.00
44. Model Release Date1.00
45. ANTHROPIC1.00
46. UK AISI — How fast is autonomous Al cyber capability advancing?0.97
47. World's Fair0.92
48. TRACK 5·JUNE 30,20260.98
49. Security1.00
2:06 #8 done58 line(s)

shot 8·sharpness 2291.6
1. AlEngineer0.97
2. Mozilla found more vulns in April 2026 than all of 20250.98
3. World'sFair1.00
4. Firefox Security Bug Fixes by Month1.00
5. All Sources • All Severities0.96
6. 4231.00
7. 611.00
8. 761.00
9. 211.00
10. 201.00
11. 261.00
12. 311.00
13. 171.00
14. 211.00
15. 221.00
16. 171.00
17. 181.00
18. 261.00
19. 191.00
20. 201.00
21. 251.00
22. 20251.00
23. JAN1.00
24. 20251.00
25. FEB1.00
26. 20251.00
27. MAR1.00
28. 20251.00
29. APR1.00
30. 20251.00
31. MAY1.00
32. 20251.00
33. JUN1.00
34. 20251.00
35. JUL1.00
36. 20251.00
37. AUG1.00
38. 20251.00
39. SEP1.00
40. 20251.00
41. OCT1.00
42. 20251.00
43. NOV1.00
44. 20251.00
45. DEC1.00
46. 20261.00
47. JAN1.00
48. 20261.00
49. FEB1.00
50. 20261.00
51. MAR1.00
52. 20261.00
53. APR1.00
54. ANTHROPIC1.00
55. Mozilla Hacks — Behind the Scenes: Hardening Firefox0.99
56. World'sFair0.97
57. TRACK 5· JUNE 30, 20260.94
58. Security1.00
2:58 #9 done28 line(s)

shot 9·sharpness 3331.1
1. AlEngineer0.98
2. What are some notable public vulns and exploits?0.99
3. World'sFair1.00
4. Log4Shell1.00
5. Heartbleed1.00
6. CVE-2021-44228· Dec 20210.99
7. CVE-2014-0160· Apr 20140.99
8. A bug in a Java logging library let attackers1.00
9. A bug in OpenSSL — the encryption library0.99
10. run code on your server just by getting it to0.98
11. behind most of the internet's HTTPS traffic1.00
12. log a string. Minecraft, iCloud, Steam, AWS0.99
13. — let anyone read server memory, including0.99
14. —all affected.0.97
15. passwords and private keys.1.00
16. 93% of enterprise cloud environments0.98
17. Half a million HTTPS servers0.99
18. were vulnerable. Belgium's Defence0.99
19. vulnerable; 4.5M patient records0.97
20. Ministry was breached within days; one0.97
21. stolen in one breach. Sat undetected0.99
22. fintech platform leaked 2M users' data.0.99
23. in open-source code for two years.0.98
24. ANTHROPIC1.00
25. Log4Shell - Heartbleed - Wikipedia0.94
26. World'sFair0.93
27. TRACK 5· JUNE 30, 20260.96
28. Security1.00
3:36 #10 skipped

shot 10·duplicate of #7
3:48 #11 done9 line(s)

shot 11·sharpness 891.0
1. AlEngineer0.98
2. World'sFair1.00
3. Section Il0.89
4. Six simple steps1.00
5. To building your own agentic harness for security1.00
6. ANTHROPIC1.00
7. World'sFair1.00
8. TRACK 5• JUNE 30, 20260.94
9. Security1.00
4:09 #12 skipped

shot 12·duplicate of #7
4:46 #13 done27 line(s)

shot 13·sharpness 1923.0
1. AlEngineer1.00
2. Distilling what we learned into six steps1.00
3. World's Fair0.99
4. Setup—invest once0.98
5. Cycle — repeat until clean0.97
6. Threat model0.98
7. Sandbox1.00
8. Discovery1.00
9. Verification1.00
10. Triage0.98
11. Patching1.00
12. Decide what counts1.00
13. Build a runnable target0.99
14. Surface candidates1.00
15. Confirm what's real1.00
16. Rank what matters0.97
17. Fix and find varlants0.99
18. Setup feeds every step0.97
19. re-scan periodically, or with each new feature or commit1.00
20. The threat model scopes the scan1.00
21. and calibrates severity. The sandbox1.00
22. is where every proof of concept fires.0.98
23. ANTHROPIC1.00
24. Using LLMs to secure source code — Anthropic0.97
25. World's Fair0.98
26. TRACK 5· JUNE 30, 20260.94
27. Security1.00
5:22 #14 skipped

shot 14·duplicate of #13
5:40 #15 done22 line(s)

shot 15·sharpness 1729.2
1. AlEngineer0.99
2. RUNNING EXAMPLE1.00
3. World's Fair0.98
4. Meetorder-service1.00
5. 0000.72
6. shop.example.com/orders?id=ORD-24120.99
7. Find your order1.00
8. PRESENTED BY0.97
9. Microsoft1.00
10. 0RD-24120.97
11. Look up0.96
12. Order #ORD-2412 Dellvered0.97
13. [email protected]·2items·$148.001.00
14. 1234 Market St, San Francisco CA· Delivered Tue, Jun 230.97
15. Track package0.98
16. View invoice0.99
17. Start a return0.99
18. Report an issue0.98
19. ANTHROPIC1.00
20. World's Fair0.92
21. TRACK 5· JUNE 30, 20260.94
22. Security1.00
5:54 #16 skipped

shot 16·duplicate of #7
6:30 #17 skipped

shot 17·duplicate of #7
7:07 #18 skipped

shot 18·duplicate of #7
7:39 #19 skipped

shot 19·duplicate of #7
8:12 #20 done31 line(s)

shot 20·sharpness 3540.6
1. AlEngineer0.99
2. Sandbox: Isolate the agent, make runs reproducible0.99
3. World's Fair0.97
4. Why1.00
5. What and how1.00
6. One offensive-security team built a0.99
7. Isolate1.00
8. harness with a simple rule: it's only a0.99
9. Target and PoCs in a microVM with egress locked down0.99
10. true positive if the agent can build a0.99
11. proof of concept and run it on the test0.97
12. Never mount credentials, tokens, or prod secrets into the agent's context1.00
13. bed. After six weeks, "the biggest0.96
14. Enforce constraints via configuration, not in the prompt1.00
15. efficacy lever has been giving the0.98
16. model test beds, live systems, and0.98
17. Reproducibility1.00
18. running the PoCs."1.00
19. Pin image tags, commit SHAs, and dependency versions1.00
20. Snapshot the environment so every run sees the same code and deps0.99
21. A reproducible sandbox lets a separate agent re-verify each vuln0.98
22. 1· Threat model0.95
23. 2·Sandbox0.99
24. 3 · Discovery0.88
25. 4·Verification0.99
26. 5 · Triage0.91
27. 6·Patching0.99
28. ANTHROPIC1.00
29. World's Fair0.98
30. TRACK 5·JUNE 30,20260.98
31. Security1.00
8:48 #21 skipped

shot 21·duplicate of #20
8:58 #22 skipped

shot 22·duplicate of #20
9:22 #23 skipped

shot 23·duplicate of #9

Transcript

306 cues· 3,765 words· 20,921 chars

chunk 0 · 0:12–0:53 · 151 words · 816 chars
0:12 Hi, I'm Eugene, member of Technical Staff at Anthropic.
0:16 I've spent the last several months working with security teams and Claude to find and fix vulnerabilities in codes and systems.
0:21 So I'll share with you what we've learned as much as we can.
0:24 Before I get started, quick show of hands.
0:26 How many of you here are security engineers in your day job?
0:31 OK, OK. How many of you here are engineers in your day job?
0:34 Oh, sweet.
0:35 OK, great.
0:36 I have a better understanding of how to give this talk now.
0:39 Security engineers in the room, please don't boo me if I oversimplify things.
0:42 I want to get this message out to as many people as I can.
chunk 1 · 0:45–1:27 · 139 words · 763 chars
0:45 So first I want to share with you three high-level trends that we are seeing in terms of model capabilities, right?
0:50 Number of vulnerabilities found and where the bottleneck is shifting to.
0:54 Then I want to distill the lessons we've learned from working with dozens of organizations to improve their security posture, as well as the bottleneck that you might encounter and how you can try to address it and prep for it.
1:03 And finally, I want to share how you can get started this week with links to resources.
1:09 So why do we care so much about cybersecurity now?
1:12 Time horizon benchmarks track the length of time an AI model can complete a task measured against an actual human doing the task.
chunk 2 · 1:23–2:07 · 121 words · 728 chars
1:23 So we all know about the meta eval task, how much knowledge one agent can do.
1:28 The UK AI Security Institute has a cybersecurity version of this.
1:32 These tasks require identifying exploiting security weaknesses in target systems and testing skills like reverse engineering and web exploitation.
1:40 So in this chart, we see that models are able to increasingly do longer cybersecurity tasks.
1:46 But what's also interesting is that you kind of see a step jump against the previous regression line.
1:51 So these models are a step improvement in capability.
chunk 3 · 1:55–2:35 · 101 words · 562 chars
1:55 And what does this step improvement in capability buy us?
1:58 Recently, Mozilla Firefox published a number of security bug fixes they made each month.
2:04 The average in 2025 was about 20, 20-ish.
2:08 And then you can see in February and March, it kind of 3x'd to about 60 and 70.
2:13 And then in April, it 7x'd to 400.
2:18 So what this means is that what's happened in April is 20x of last year's average.
2:25 They attributed about two-thirds of this to Mito's preview, about 271, which shows that frontier models can help defenders like yourself find and fix vulnerabilities at scale.
chunk 4 · 2:37–3:20 · 116 words · 655 chars
2:37 We also, one of some notable vulnerabilities, hands up, who here remembers log4shell?
2:43 Anyone?
2:43 Anyone was here over the December holidays like scrambling to patch this stuff too?
2:47 So log4shell,
2:50 It's a bug in the Java logging library.
2:51 If an attacker sends a string, you log it, the attacker can run code on your system.
2:57 The Belgium Defense Ministry was breached within days, and a fintech platform leaked 2 million users' data.
3:03 And then before that, we had Heartbleed, which is a bug in OpenSSL.
3:07 So much of the internet runs on OpenSSL, and we all know what impact it had.
chunk 5 · 3:13–3:58 · 117 words · 677 chars
3:13 This is entropic sharing of our own work, scanning more than 1,000 open source repos.
3:20 From 23,000 candidates, 6,200 of them were rated as high or critical.
3:25 And at the time of the update, 1,600 of them were reported to maintainers and about 100 patched upstream.
3:32 We shared our observation that finding vulnerabilities now is quite straightforward.
3:39 The bottleneck has now shifted to verification triage and patching.
3:43 And I want to share with you everything we've learned about that so you can get ahead of it.
chunk 6 · 3:48–4:32 · 111 words · 660 chars
3:48 So how do we do this in our own systems and code bases?
3:52 Two words, agentic harnesses.
3:54 I'm citing Mozilla again.
3:55 And this was even before they worked with Mito's preview.
3:58 I'm going to read off the screen here.

Chapters

0:00 Working with security teams to find and fix vulnerabilities
0:49 Three trends in model security capability
1:16 Cybersecurity benchmarks and the step jump in capability
1:54 Mozilla's 20x jump in monthly security fixes
2:44 Log4Shell, Heartbleed, and why this matters
3:22 Anthropic's scan of a thousand open source repos
3:35 The bottleneck shifts to verify, triage, and patch
3:48 Why agentic harnesses changed the game
4:29 The six step workflow
5:31 A running example: the order service
5:45 Step 1: the threat model and 90% true positives
7:42 Step 2: the sandbox for isolation and reproducibility
9:24 Step 3: discovery and the five line SQL injection
11:44 Step 4: independent adversarial verification
13:36 Step 5: triage and the scarcity of engineer attention
15:52 Step 6: patching and closing the loop
17:19 It all looks like a machine learning pipeline
17:43 The non technical bottlenecks are harder
18:47 Organizational bottlenecks: routing, severity, bandwidth
20:05 Three takeaways and how to start this week
20:38 Scanning was never the bottleneck.