Videos Cz4v1WHVyZc

HTML Is All Agents Need — James Russo, HeyGen

index_state ready data_status ok

AI Engineer· published 2026-07-21· 0:15:13· en-US· indexed 2026-08-10 22:44

Open on YouTube

58 shot(s).

keyframes kept every frame deduplicated

What was stored

cues: 146; whisperx 146
chunks: 26; from 146 cues
keyframes: 44; kept of 58 captured
frames with text: 44; 587 lines read
chapters: 14; from the source metadata
keyframe bytes: 6.6 MB; word timings on 146 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage	state	model	started	took
`fetch`	done	—	2026-08-10 22:40	1m 06s
`stt`	done	—	2026-08-10 22:41	16s
`chunk`	done	—	2026-08-10 22:42	0s
`text_embed`	done	—	2026-08-10 22:42	0s
`keyframe`	done	—	2026-08-10 22:42	1m 47s
`ocr`	done	—	2026-08-10 22:44	14s
`frame_embed`	done	—	2026-08-10 22:44	7s

Frames, and what the machine read

0:02 #0 done2 line(s)

shot 0·sharpness 453.8
1. AlEngineer0.96
2. World's Fair0.97
0:03 #1 done2 line(s)

shot 1·sharpness 660.7
1. AlEngineer0.95
2. World's Fair0.99
0:10 #2 done24 line(s)

shot 2·sharpness 2728.5
1. LAB & PLATINUM SPONSORS0.99
2. Amazon AGI Lab0.95
3. ANTHROP\C1.00
4. Google DeepMind1.00
5. MINIMAX0.93
6. OpenAI0.92
7. Akamai1.00
8. arize1.00
9. aws1.00
10. Braintrust bright data0.98
11. B1.00
12. Browserbase1.00
13. docker1.00
14. :neo4j0.91
15. ORACLE1.00
16. PayPal1.00
17. qodo1.00
18. reducto1.00
19. Sonar1.00
20. Makers of0.99
21. togetherai1.00
22. Unblocked1.00
23. WorkOS1.00
24. SonarQube1.00
0:19 #3 done2 line(s)

shot 3·sharpness 97.8
1. HF1.00
2. HYPERFRAMES1.00
0:20 #4 done1 line(s)

shot 4·sharpness 116.6
1. SEEKABLE1.00
0:28 #5 done1 line(s)

shot 5·sharpness 2392.1
1. RENBER0.70
0:35 #6 done2 line(s)

shot 6·sharpness 327.8
1. HEYGEN·HYPERFRAMES1.00
2. CODE IS THE CANVAS0.95
0:54 #7 done2 line(s)

shot 7·sharpness 158.6
1. AlEngineer0.98
2. World's Fair0.99
1:25 #8 done10 line(s)

shot 8·sharpness 1563.1
1. World's Fair0.92
2. AlEngineer0.98
3. World's Fair0.96
4. Codingagents1.00
5. PRESENTED BY1.00
6. democratizedbuilding,1.00
7. Microsoft1.00
8. notlaunching.1.00
9. World's Fair0.96
10. Engineering the future of Al0.99
1:28 #9 done15 line(s)

shot 9·sharpness 2032.8
1. World's Fair0.97
2. AlEngineer0.99
3. World'sFair1.00
4. WHERE WE START0.96
5. Mostpeople knowHeyGenfor Alavatars.0.98
6. That'stheA-roll.1.00
7. PRESENTED BY1.00
8. Microsoft1.00
9. SARA HAHN0.95
10. A-ROLL1.00
11. B-ROLL + POLISH1.00
12. Just the avatar.0.99
13. The full video.0.97
14. World'sFair1.00
15. Engineering the future of Al0.98
1:35 #10 skipped

shot 10·duplicate of #9
1:40 #11 skipped

shot 11·duplicate of #9
1:48 #12 skipped

shot 12·duplicate of #9
1:51 #13 skipped

shot 13·duplicate of #9
1:59 #14 done15 line(s)

shot 14·sharpness 1973.2
1. World'sFair1.00
2. AlEngineer0.99
3. World'sFair1.00
4. WHERE WE START1.00
5. Mostpeople knowHeyGenfor Alavatars.0.97
6. That'stheA-roll.0.97
7. PRESENTED BY1.00
8. Microsoft1.00
9. A-ROLL1.00
10. B-ROLL + POLISH0.97
11. Just the avatar.0.99
12. The full video.1.00
13. World'sFair0.97
14. TRACK 1· JULY 2, 20260.93
15. Generative Media1.00
2:04 #15 done16 line(s)

shot 15·sharpness 2037.4
1. World'sFair0.99
2. AlEngineer0.99
3. World'sFair1.00
4. WHERE WE START0.99
5. Mostpeople knowHeyGenfor Alavatars.0.98
6. That'stheA-roll.1.00
7. PRESENTED BY1.00
8. Microsoft1.00
9. SARA HAHN0.95
10. A-ROLL1.00
11. B-ROLL + POLISH1.00
12. Just the avatar.1.00
13. The full video.0.99
14. World'sFair1.00
15. TRACK 1· JULY 2, 20260.95
16. Generative Media1.00
2:18 #16 done11 line(s)

shot 16·sharpness 1535.8
1. World's Fair0.94
2. AlEngineer0.98
3. World'sFair0.97
4. PRESENTED BY1.00
5. THE THESIS0.97
6. HTML, CSS, andJavaScriptare the0.98
7. Microsoft1.00
8. nativelanguage ofLLMs.0.98
9. World'sFair1.00
10. TRACK 1· JULY 2, 20260.94
11. Generative Media1.00
2:47 #17 done9 line(s)

shot 17·sharpness 1410.5
1. World's Fair0.96
2. AlEngineer0.98
3. World'sFair1.00
4. THE THESIS0.96
5. HTML, CSS, andJavaScriptare the0.98
6. nativelanguage ofLLMs.0.99
7. World'sFair0.99
8. TRACK 1· JULY 2, 20260.94
9. Generative Media1.00
3:18 #18 done37 line(s)

shot 18·sharpness 2395.1
1. World's Fair0.93
2. AlEngineer1.00
3. World'sFair1.00
4. THE FIELD IS CONVERGING ON HTML0.99
5. Andrej Karpathy1.00
6. x0.78
7. Thariq1.00
8. x0.77
9. @karpathy·Follow0.98
10. @trq212·Follow1.00
11. This works really well btw, at the end of your query ask1.00
12. HTML is the new markdown.1.00
13. your LLM to "structure your response as HTML", then0.98
14. view the generated file in your browser. I've also had1.00
15. I've stopped writing markdown files for almost everything0.98
16. some success asking the LLM to present its output as0.99
17. and switched to using Claude Code to generate HTML for1.00
18. slideshows, etc.1.00
19. me. This is why.1.00
20. More generally, imo audio is the Show more0.99
21. Thariq@trq2121.00
22. x.com/i/article/2052...1.00
23. Thariq@trq2121.00
24. x.com/i/article/2052...0.98
25. 6:03 PM ·May 8,20260.94
26. 4:20 PM · May 11, 20260.94
27. 12.1K1.00
28. Reply1.00
29. Copy link0.99
30. 19.3K1.00
31. Reply1.00
32. Copy link1.00
33. Read 900 replies1.00
34. Read 1K replies1.00
35. World's Fair0.99
36. TRACK 1· JULY 2, 20260.93
37. Generative Media1.00
3:41 #19 done16 line(s)

shot 19·sharpness 1291.1
1. World's Fair0.94
2. AlEngineer0.98
3. World'sFair1.00
4. Nothingwas bothgreatandagent-friendly.0.99
5. After Effects1.00
6. HTML1.00
7. DU ULTS0.75
8. Remotion1.00
9. Premiere1.00
10. ● Rive0.93
11. Lottie1.00
12. AGENT-FRIENDLY →0.97
13. Don't fight the model.0.98
14. World'sFair1.00
15. TRACK 1· JULY 2, 20260.95
16. Generative Media1.00
4:24 #20 skipped

shot 20·duplicate of #19
4:31 #21 skipped

shot 21·duplicate of #19
5:14 #22 skipped

shot 22·duplicate of #19
5:46 #23 done13 line(s)

shot 23·sharpness 1648.3
1. World's Fair0.94
2. AlEngineer0.98
3. World'sFair1.00
4. — THE TELL0.91
5. Weletasmallmodeldesignit.1.00
6. Thesimplestthingwon.1.00
7. <div class="clip" data-start="0"0.99
8. data-duration="5">1.00
9. The entire framework: plain HTML plus a little metadata.0.98
10. authored by Gemini 3 Flash1.00
11. World'sFair1.00
12. TRACK 1· JULY 2, 20260.95
13. Generative Media1.00

Transcript

146 cues· 2,488 words· 13,364 chars

chunk 0 · 0:36–1:21 · 133 words · 759 chars
0:36 Since we're talking about generative media, I thought starting with a video might be good.
0:40 That video was made completely by an agent in a single shot, all utilizing HTML.
0:46 My name is James Russo.
0:47 I am the co-creator and tech lead of HyperFrames at HeyGen.
0:51 Today, we're going to be talking about why we think HTML is all your agents need to create great videos.
0:58 Show of hands here who finds creating that launch post or that launch video harder than actually building now with coding agents?
1:06 All right, some hands up there.
chunk 1 · 1:09–1:53 · 136 words · 752 chars
1:09 Coding agents have democratized building, made it incredibly easy for anyone to create anything that comes to their minds.
1:16 However, we think that launching your product or your feature, getting it out into the world is still quite hard.
1:22 And we at HeyGen have been trying to close that gap.
1:27 If you're not familiar with HeyGen, our mission is to solve communication through video.
1:31 We started by creating the best AI avatar on the market.
1:35 You can think of this as the A-roll, the footage you see here on the left side of the screen.
chunk 2 · 1:40–2:22 · 119 words · 648 chars
1:40 It's the narrator, the character, the main subject of the video.
1:43 However, it's kind of plain, and great videos have a lot more pieces to them.
1:48 They have B-roll, which is the images and the other pieces of media and assets.
1:53 They have animations.
1:54 They have captions.
1:55 They have music.
1:56 All of this is needed to create a great video.
2:00 And for us, it's important that we nail every layer, not just the avatar, to solve communication through video and give agents the right canvas to create great videos.
chunk 3 · 2:13–2:56 · 141 words · 732 chars
2:13 So how do you give agents the ability to generate all of these layers and build up this composition needed to create a great video?
2:21 Our bet is on HTML.
2:23 HTML, CSS, and JavaScript are the native languages of LLMs.
2:27 Most of their training data, every web page that gets scraped at the end of the day, is essentially just HTML, CSS, and JavaScript under the hood.
2:35 When you try to teach a model a new DSL or even your own custom JSON structure, it's forcing it to speak another language.
2:42 We like to think of this as trying to ask Shakespeare to write a poem in Japanese or Chinese.
chunk 4 · 2:49–3:31 · 137 words · 720 chars
2:49 Even if you give them a bunch of examples and teach them a bunch of things, you will not get the best output from them because it's not their native tongue.
2:57 So why not let the LLMs and agents talk in their native tongue when creating videos?
3:04 We're not the only ones saying this.
3:06 Over the last few months, Tarek, Andre Kaparthi have been talking a lot about how HTML is a new markdown.
3:12 It's a great output for LLMs to give you visual representation.
3:16 For what's worth, I submitted this talk before both of these tweets came out.
chunk 5 · 3:20–4:02 · 118 words · 642 chars
3:20 I can show you proof later if you're curious.
3:22 But we all kind of came to the same conclusion here, which is that let LLMs talk in their native language, and you can get better output from them.
3:34 It wasn't necessarily an easy thing.
3:36 We've been working on this for over a year now and tried a bunch of things along the way.
3:42 There's a bunch of different frameworks and tools to help create great videos.
3:47 And we like to think about them on this axis of quality of results and agent friendly.
chunk 6 · 3:54–4:35 · 114 words · 673 chars
3:54 For us, After Effects, Premiere Pro, these are kind of like the gold standards of creatives.
3:58 They create great output.
4:00 but they're not very agent-friendly.
4:02 And even with the more recent connectors into it, it's more of a co-pilot or something that can help you facilitate things you already know how to do.
4:10 It doesn't give it creative output.
4:12 We tried things like Lottie and Rive, which are coding languages in JSON or custom XML formats.
4:20 And they can get you pretty good output, but agent-friendly isn't necessarily true for those because they're not their native tongue.
chunk 7 · 4:31–5:07 · 114 words · 611 chars
4:31 They aren't as editable or controllable, which is a big thing for us, is that controllability layer.
4:36 We played around with promotion quite a bit and honestly thought it was a great example of what LLMs and agents could do with coding.
4:44 However, we noticed that we had to teach them the framework, we had to teach them the language, and give them a bunch of examples on how to write code properly, which ultimately took away a lot of the creativity of it.
4:58 Then we have HTML, which around November of last year when Gemini 3 and the latest models came out, we saw a step function improvement in what LLMs could do.

Chapters

0:00 Introduction and the HeyGen mission
0:58 The challenge of creating launch videos
1:27 The importance of A-roll, B-roll, and composition
2:13 Why HTML, CSS, and JavaScript are the native languages of LLMs
3:06 Comparing HTML to other frameworks like Remotion
5:24 Designing the Hyperframes framework with Gemini Flash
6:54 How Hyperframes works in the browser
8:56 Leveraging browser-native technologies like Three.js and WebGL
9:25 Using Skills to teach agents video taste
10:56 Crafting videos: the human-in-the-loop workflow
12:09 Keyframes integration
12:35 Scaling and performance metrics
13:28 Future goals: Code-to-Video benchmarking
14:31 Agents have made building incredibly easy. Launching is still quite hard.