Videos jRCpXUjz4CI

Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute

index_state ready data_status ok

AI Engineer· published 2026-07-24· 0:21:11· en-US· indexed 2026-08-10 19:40

Open on YouTube

74 shot(s).

keyframes kept every frame deduplicated

What was stored

cues: 217; whisperx 217
chunks: 37; from 217 cues
keyframes: 48; kept of 74 captured
frames with text: 48; 1,032 lines read
chapters: 0; from the source metadata
keyframe bytes: 8.5 MB; word timings on 217 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage	state	model	started	took
`fetch`	done	—	2026-08-09 22:55	0s
`stt`	done	—	2026-08-09 07:22	23s
`chunk`	done	—	2026-08-09 07:22	0s
`text_embed`	done	—	2026-08-10 19:40	0s
`keyframe`	done	—	2026-08-09 07:22	2m 51s
`ocr`	done	—	2026-08-09 07:25	25s
`frame_embed`	done	—	2026-08-10 19:40	8s

Frames, and what the machine read

0:02 #0 done2 line(s)

shot 0·sharpness 455.2
1. AlEngineer0.95
2. World's Fair0.97
0:03 #1 done2 line(s)

shot 1·sharpness 670.0
1. AlEngineer0.96
2. World's Fair0.99
0:11 #2 done24 line(s)

shot 2·sharpness 2742.1
1. LAB & PLATINUM SPONSORS0.97
2. Amazon AGI Lab0.99
3. ANTHROP\C1.00
4. Google DeepMind1.00
5. MINIMAX0.98
6. OpenAl0.92
7. Akamai1.00
8. arize1.00
9. aws1.00
10. Braintrust bright data0.99
11. B1.00
12. Browserbase1.00
13. docker1.00
14. :neo4j0.91
15. ORACLE1.00
16. PayPal1.00
17. qodo0.99
18. reducto1.00
19. Sonar1.00
20. Makers of1.00
21. together.ai0.98
22. Unblocked1.00
23. WorkOS1.00
24. SonarQube1.00
0:15 #3 done2 line(s)

shot 3·sharpness 220.5
1. AlEngineer0.98
2. World's Fair0.96
0:55 #4 done12 line(s)

shot 4·sharpness 1571.6
1. Ask Gemini1.00
2. Relaunch to update i0.97
3. AlEngineer0.98
4. World's Fair0.97
5. 40.78
6. PRESENTED BY0.99
7. You're going to see Avengers0.98
8. Microsoft1.00
9. Infinity War later today1.00
10. World's Fair0.97
11. Engineering tasf0.87
12. füture of Al0.93
1:13 #5 done13 line(s)

shot 5·sharpness 1327.3
1. 中D0.54
2. Ask Gemini1.00
3. Relaunch to update i0.98
4. AlEngineer0.97
5. World'sFair0.96
6. 40.80
7. PRESENTED BY1.00
8. Musical.ly was about to rebrand0.98
9. Microsoft1.00
10. to“TikTok"0.97
11. World's Fair0.95
12. TRACK 5· JULY 1, 20260.95
13. Evals1.00
1:50 #6 done16 line(s)

shot 6·sharpness 3504.4
1. 00.55
2. Ask Gemini0.99
3. Relaunch to update0.98
4. AlEngineer0.97
5. World'sFair1.00
6. staysaasy1.00
7. Follow1.00
8. @staysaasy1.00
9. It's 2018 and your coworker just sent you a 400 line pull request.0.99
10. You get a cup of coffee and sit down to review it.1.00
11. It's beautiful. Elegant micro-refactors. Crispy method names.1.00
12. You catch a few things, but that's ok. It's part of the dance. They didn't1.00
13. consider extensibility on part of their APl. Here's a comment buddy.0.98
14. World's Fair0.95
15. TRACK 5· JULY 1, 20260.95
16. Evals1.00
2:13 #7 done11 line(s)

shot 7·sharpness 1169.9
1. 中p0.51
2. Ask Gemini1.00
3. Relaunch to update i0.95
4. AlEngineer0.98
5. World'sFair0.97
6. 40.88
7. Musical.ly was about to rebrand1.00
8. to“TikTok"0.97
9. World's Fair0.96
10. TRACK 5· JULY 1, 20260.95
11. Evals1.00
2:24 #8 skipped

shot 8·duplicate of #6
2:35 #9 done10 line(s)

shot 9·sharpness 1471.4
1. Ask Gemini1.00
2. Relaunch to update i0.96
3. AlEngineer0.98
4. World'sFair1.00
5. 40.87
6. What will the history books say0.99
7. about software engineering?1.00
8. World'sFair0.98
9. TRACK 5· JULY 1, 20260.93
10. Evals1.00
2:50 #10 done11 line(s)

shot 10·sharpness 1744.0
1. Ask Gemini1.00
2. Relaunch to update i0.95
3. AlEngineer0.97
4. World'sFair1.00
5. Q0.60
6. "Software engineering was when1.00
7. you knew what the code would do1.00
8. before you ran it."0.99
9. World's Fair0.95
10. TRACK 5· JULY 1, 20260.95
11. Evals1.00
2:57 #11 done14 line(s)

shot 11·sharpness 2664.2
1. Ask Gemini0.98
2. Relaunch to update0.94
3. AlEngineer0.96
4. World'sFair1.00
5. Q0.61
6. François Chollet0.98
7. Subscribe1.00
8. @fchollet1.00
9. Agentic coding is a form of machine learning. Generated code is best0.99
10. treated as a blackbox artifact whose behavior and generalization should1.00
11. be managed via empirical evaluation, like with any ML model.1.00
12. World'sFair1.00
13. TRACK 5· JULY 1,20260.97
14. Evals1.00
3:31 #12 done14 line(s)

shot 12·sharpness 2282.4
1. Ask Gemini1.00
2. Relaunch to update i0.99
3. AlEngineer0.97
4. World'sFair1.00
5. 40.58
6. def extract_phone_number(text):1.00
7. match = re.search(r"\d{3}-\d{3}-\d{4}", text)0.97
8. return match.group()1.00
9. def main():0.98
10. text = "My number is 555-123-4567"0.98
11. print(extract_phone_number(text))1.00
12. World'sFair0.99
13. TRACK 5· JULY 1, 20260.96
14. Evals1.00
4:08 #13 done21 line(s)

shot 13·sharpness 2801.9
1. CD0.74
2. 00.87
3. 80.80
4. Ask Gemini1.00
5. Relaunch to update0.98
6. AlEngineer0.98
7. World'sFair1.00
8. def extract_phone_number(text):1.00
9. 40.99
10. match = re.search(r"\d{3}-\d{3}-\d{4}", text)0.98
11. return match.group()0.97
12. response = OpenAI().responses.create(1.00
13. model="gpt-5.5",0.99
14. input=f"Extract the phone number:\n\ntext}",0.98
15. return response.output_text0.99
16. def main():1.00
17. text = "My number is 555-123-4567"0.98
18. print(extract_phone_number(text))1.00
19. World's Fair0.99
20. TRACK 5 • JULY 1, 20260.95
21. Evals1.00
4:43 #14 skipped

shot 14·duplicate of #13
5:16 #15 skipped

shot 15·duplicate of #11
5:37 #16 done15 line(s)

shot 16·sharpness 1163.2
1. CD0.57
2. Ask Gemini0.96
3. Relaunch to update0.95
4. AlEngineer0.99
5. World's Fair0.99
6. 40.87
7. Machine Learning1.00
8. → Agent Building0.99
9. Training data0.99
10. PRESENTED BY0.99
11. →0.81
12. Microsoft1.00
13. World's Fair0.97
14. TRACK 5 · JULY 1, 20260.92
15. Evals1.00
6:11 #17 done20 line(s)

shot 17·sharpness 2047.1
1. D0.90
2. 白0.55
3. Ask Gemini0.99
4. Relaunch to update i0.96
5. AlEngineer0.99
6. World's Fair0.99
7. 40.90
8. Machine Learning1.00
9. → Agent Building0.97
10. PRESENTED BY1.00
11. Training data0.99
12. → Environments0.99
13. Test / validation set0.98
14. → Evals (environments)0.98
15. Microsoft1.00
16. Weights1.00
17. → Skills, prompts, tools, model0.99
18. World's Fair0.97
19. TRACK 5· JULY 1, 20260.96
20. Evals1.00
6:38 #18 done25 line(s)

shot 18·sharpness 3585.6
1. 中D0.61
2. Ask Gemini1.00
3. Relaunch to update i0.95
4. AlEngineer0.99
5. World's Fair0.99
6. 40.99
7. Machine Learning1.00
8. → Agent Building0.96
9. Training data1.00
10. → Environments0.99
11. Test / validation set0.98
12. → Evals (environments)0.98
13. Weights1.00
14. → Skills, prompts, tools, model0.99
15. Loss function1.00
16. → Environment rewards & feedback0.96
17. Backprop / optimizer0.97
18. → GEPA or text-based optimization0.98
19. Gradient descent step1.00
20. → Pull request0.98
21. Overfitting1.00
22. → Reward hacking (or overfitting)0.97
23. World's Fair0.99
24. TRACK 5· JULY 1, 20260.95
25. Evals1.00
7:02 #19 skipped

shot 19·duplicate of #18
7:21 #20 done10 line(s)

shot 20·sharpness 741.9
1. 中D0.59
2. Ask Gemini1.00
3. Relaunch to update i0.98
4. AlEngineer0.99
5. World's Fair0.96
6. 40.77
7. Agents >> ML0.99
8. World's Fair0.98
9. TRACK 5 · JULY 1, 20260.92
10. Evals1.00
7:42 #21 done17 line(s)

shot 21·sharpness 2369.1
1. CD0.78
2. Ask Gemini1.00
3. Relaunch to update i0.94
4. AlEngineer1.00
5. World's Fair0.99
6. 40.92
7. François Chollet1.00
8. Subscribe1.00
9. @fchollet1.00
10. agents are1.00
11. agent performance0.99
12. Agentic coding is a form of machine learning. Generated code is best0.98
13. treated as a blackbox artifact whose behavior and generalization should1.00
14. be managed via empirical evaluation, like with any ML model.1.00
15. World's Fair0.97
16. TRACK 5· JULY 1,20260.96
17. Evals1.00
8:16 #22 done12 line(s)

shot 22·sharpness 1133.0
1. 中p0.59
2. Ask Gemini0.97
3. Relaunch to update0.98
4. AlEngineer0.98
5. World'sFair1.00
6. Environments1.00
7. 40.77
8. Instruction1.00
9. Sandbox1.00
10. World's Fair0.96
11. TRACK 5· JULY 1, 20260.96
12. Evals1.00
8:32 #23 done14 line(s)

shot 23·sharpness 1300.3
1. CD0.67
2. G0.63
3. Ask Gemini0.99
4. Relaunch to update0.93
5. AlEngineer0.99
6. World's Fair0.99
7. Environments1.00
8. 40.81
9. Instruction1.00
10. Sandbox1.00
11. Verifier1.00
12. World's Fair0.97
13. TRACK 5 · JULY 1, 20260.92
14. Evals1.00

Transcript

217 cues· 3,491 words· 19,050 chars

chunk 0 · 0:16–1:00 · 118 words · 612 chars
0:16 Awesome, thank you so much.
0:19 So yeah, like you said, my name's Alex Shaw.
0:22 I work at Lott Institute.
0:24 And I'll be speaking today about Harbor, which is an agent evaluation and RL environment framework.
0:31 And the title of my talk is Everything is a Rollout.
0:34 And I think you'll see as I get into it.
0:37 why we titled the talk that way.
0:39 But first, I want everybody to come travel back in time with me to the year 2018, so eight years ago.
chunk 1 · 0:49–1:28 · 95 words · 560 chars
0:49 And we're going to talk about some of the things that were going on in 2018.
0:53 So probably you're going to see Avengers Infinity War later today.
0:58 The second one is just getting released.
1:01 GPT-1 was just released, and you probably didn't even notice, although maybe some people did.
1:10 Musically, this random startup was about to rebrand to a product called TikTok.
1:17 And you might have just learned about AirPods when you saw somebody walking around with headphones that had no cord.
chunk 2 · 1:23–2:05 · 122 words · 668 chars
1:23 So what about software engineering?
1:25 What did software engineering look like in 2018?
1:30 We're gonna read this tweet from Stay Sassy Sassy with two A's about what it was like to write code in 2018.
1:38 So it says, it's 2018 and your coworker just sent you a 400 line pull request.
1:43 You get a cup of coffee and sit down to review it.
1:46 It's beautiful, elegant micro-refactors, crispy method names.
1:50 You catch a few things, but that's OK.
1:51 It's part of the dance.
1:53 They didn't consider extensibility on part of their API.
chunk 3 · 1:56–2:38 · 119 words · 638 chars
1:56 Here's a comment, buddy.
1:57 And this is actually just part of the tweet, so it keeps going.
2:00 You should look it up if you want.
2:02 And it's obviously written humorously.
2:06 But the thing is, it does feel a little bit nostalgic, just like some of these other things that were going on in 2018.
2:15 Now the thing is, this only changed maybe six or 12 or 18 if you're a very early adopter.
2:22 months ago, but it already feels like the distant past in some ways.
chunk 4 · 2:27–3:11 · 122 words · 692 chars
2:27 And I think it's time to start talking about, well, what will the history books say about software engineering?
2:34 And when I say software engineering, I mean that style of 2018 software engineering.
2:40 It will probably say, the history books will probably say a lot of things, but I think one thing for sure that they'll say is software engineering was when you knew what the code would do before you ran it.
2:53 So, and that brings me then to a different tweet from Francois Chalet, where he says, agentic coding is a form of machine learning.
chunk 5 · 3:02–3:45 · 115 words · 651 chars
3:02 Generated code is best treated as a black box artifact whose behavior and generalization should be managed via empirical evaluation, like with any ML model.
3:12 And that kind of brings us to the next part in this talk, which is to compare and contrast what agent development looks like versus what more traditional software engineering development looks like and why it demands a new set of tools to really understand what's going on and have confidence and trust.
3:31 So here's a 2018 program right here.
chunk 6 · 3:34–4:16 · 103 words · 528 chars
3:34 So you can tell already the purpose of the program is to extract phone numbers from text.
3:41 And we have a regex right here that looks for the phone number.
3:45 And I can say with 100% confidence what will happen if I run this program 1 million times in a row.
3:55 So now let's update it to the 2026 version.
3:59 So I swap out my regex.
4:02 And instead, obviously, I throw in my model call instead.
chunk 7 · 4:07–4:47 · 112 words · 616 chars
4:07 And I say, extract this phone number.
4:10 So in some ways, this is actually a more powerful program because the regex was actually a little bit brittle.
4:16 It would have missed any phone number that wasn't formatted exactly like how it was specified.
4:22 Whereas I'm pretty confident that this program with GPT 5.5 will catch a lot of the phone numbers that are formatted weirdly.
4:30 However, if I ran this exact program 1 million times, I'm not 100% confident that it will print the same thing every single time or that I know exactly what it will print.
chunk 8 · 4:44–5:24 · 115 words · 624 chars
4:44 It probably gets it right almost every time.
4:46 This is a pretty simple task.

Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute

Scene timeline

What was stored

Provenance

Frames, and what the machine read

Transcript