Videos Bc6Ojl2XS1w

From Transcription to Live Music: Gemini's Audio Stack — Thor Schaeff, Google DeepMind

index_state ready data_status ok

AI Engineer· published 2026-06-09· 0:19:33· en-US· indexed 2026-08-11 11:01

Open on YouTube

106 shot(s).

keyframes kept every frame deduplicated

What was stored

cues: 240; whisperx 240
chunks: 34; from 240 cues
keyframes: 60; kept of 106 captured
frames with text: 60; 1,571 lines read
chapters: 0; from the source metadata
keyframe bytes: 11.2 MB; word timings on 240 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage	state	model	started	took
`fetch`	done	—	2026-08-11 10:56	1m 20s
`stt`	done	—	2026-08-11 10:58	20s
`chunk`	done	—	2026-08-11 10:58	0s
`text_embed`	done	—	2026-08-11 10:58	0s
`keyframe`	done	—	2026-08-11 10:58	2m 14s
`ocr`	done	—	2026-08-11 11:00	33s
`frame_embed`	done	—	2026-08-11 11:01	10s

Frames, and what the machine read

0:03 #0 done2 line(s)

shot 0·sharpness 658.4
1. AlEngineer0.98
2. EUROPE1.00
0:08 #1 done2 line(s)

shot 1·sharpness 829.1
1. PRESENTINGSPONSOR1.00
2. Google DeepMind1.00
0:13 #2 done3 line(s)

shot 2·sharpness 907.6
1. PLATINUM SPONSORS0.98
2. # Braintrust0.96
3. WorkOS OpenAI0.95
0:27 #3 done5 line(s)

shot 3·sharpness 675.6
1. WHAT'S NEW IN AI AUDIO?1.00
2. at1.00
3. Google DeepMind1.00
4. Thor Schaeff I @thorwebdev0.96
5. ALFngineer0.93
0:41 #4 done6 line(s)

shot 4·sharpness 2225.6
1. AIE1.00
2. Thor Schaeff0.95
3. Developer Relations Engineer,0.98
4. @thorwebdev1.00
5. Google DeepMind0.99
6. AlEngineer0.99
0:47 #5 skipped

shot 5·duplicate of #4
0:50 #6 skipped

shot 6·duplicate of #4
0:53 #7 done6 line(s)

shot 7·sharpness 2225.3
1. AIE1.00
2. Thor Schaeff0.97
3. Developer Relations Engineer,1.00
4. @thorwebdev1.00
5. Engineering the future of Al1.00
6. AlEngineer1.00
0:56 #8 skipped

shot 8·duplicate of #4
0:58 #9 skipped

shot 9·duplicate of #7
1:02 #10 skipped

shot 10·duplicate of #7
1:04 #11 skipped

shot 11·duplicate of #7
1:07 #12 skipped

shot 12·duplicate of #7
1:10 #13 done7 line(s)

shot 13·sharpness 2289.6
1. AIE1.00
2. Thor.Schaeff:0.93
3. Developer Relations Engineer,0.98
4. @thorwebdev1.00
5. Braintrust1.00
6. WorkOS OpenAI0.95
7. AIE1.00
1:12 #14 skipped

shot 14·duplicate of #13
1:14 #15 done31 line(s)

shot 15·sharpness 1706.3
1. EchoScript | Google Al S0.92
2. Gemini Voice Library0.98
3. Ooogle AI Studilo0.87
4. [Public] Live Jukebax | Ooog0.87
5. +Ask Oemini0.89
6. alstudio.google.com/apps/bundled/echoscript?showPreview=true&showAssistant=true&fullscreenApplet=true0.98
7. 白0.59
8. Ca0.67
9. EchoScript1.00
10. Remix0.92
11. L Device0.86
12. 650.71
13. EchoScript Al0.99
14. Powered by Gemini 3 Flash Preview0.99
15. Turn your audio into accurate text1.00
16. Upload a file or record directly to get speaker-identified transcripts with1.00
17. timestamps and language detection instantly.1.00
18. ★1.00
19. AIE1.00
20. Record Audio1.00
21. Upload File0.99
22. ★1.00
23. ★1.00
24. ★1.00
25. ★1.00
26. Recording...1.00
27. 01:011.00
28. Stop Recording0.99
29. Braintrust1.00
30. WorkOS OpenAI0.93
31. Al0.90
1:22 #16 skipped

shot 16·duplicate of #7
1:34 #17 done47 line(s)

shot 17·sharpness 1833.6
1. Shipping at relentless pace1.00
2. Frontier Models0.97
3. Gemini 2.5 Pro0.99
4. Gemini 2.5 Pro1.00
5. Gemini 2.5 Pro0.98
6. Gemini 2.5 Flash-Lite0.99
7. ★1.00
8. Geminl 2.5 Flash0.98
9. AIE1.00
10. ★1.00
11. Geminl1.5 Pro0.95
12. Gemini 2.00.99
13. Flash1.00
14. Gemini 2.00.97
15. Flash-Lite1.00
16. Geminl 3.1 Flash-Lite0.97
17. ★1.00
18. ★1.00
19. ★1.00
20. ★1.00
21. Gemini 1.5 Flash0.97
22. Flash Thinking0.98
23. Gemini 2.00.98
24. Gemini 2.0 Pro0.99
25. Geminl 3.0 Pro0.96
26. Gemini 3.1 Pro0.99
27. (Preview)1.00
28. 2024May-Jun-Jul-Aug-Sep-Oct-Nov-Dec2025Jan-Feb-Mar-Apr-May-Jun-Jul-Aug-Sep-Oct-Nov-Dec1.00
29. 2026Jan - Feb -Mar - Apr0.95
30. Gemma 20.99
31. Gemma 30.94
32. Gemma 40.99
33. 98.2780.93
34. 18. 48. 128, 2780.89
35. E28, E480.268 A48, 3180.82
36. Gemma 21.00
37. Gemma 3n0.96
38. E28. E480.87
39. Gemma 2 for Japan0.97
40. Gemma 30.99
41. 270M1.00
42. Open Models0.96
43. 20240.91
44. AlEngineer0.97
45. AlEngineer0.99
46. EUROPE1.00
47. 20260.96
2:40 #18 done47 line(s)

shot 18·sharpness 1964.6
1. genMedia Models0.98
2. Imagen 4 Standard/UItra0.98
3. Imagen 4 Standard/UItra0.99
4. (Preview)0.93
5. Veo 3 & Veo 3 Fast0.98
6. Veo 3.1 Lite1.00
7. (Prevlew)1.00
8. Lyria RealTime0.99
9. Imagen 4 Fast1.00
10. dard. Ultra GA)0.93
11. Lyria 3 Clip & Pro0.97
12. *★*0.52
13. AIE1.00
14. ★1.00
15. ★1.00
16. ★1.00
17. Imagen 31.00
18. Veo1.00
19. Genie 21.00
20. (Preview)0.92
21. Veo 20.92
22. Imagen 3 0020.99
23. Gemini 2.0 Flash Image0.98
24. Veo 20.96
25. (GA)1.00
26. Nano Banana1.00
27. (Preview)0.94
28. Veo 3.11.00
29. Veo 3.1 Fast1.00
30. Nano Banana Pro0.99
31. Nano-Banana 21.00
32. ★1.00
33. ★1.00
34. ★1.00
35. 2024May-Jun-Jul-Aug-Sep-Oct-Nov-Dec2025Jan-Feb-Mar-Apr-May- Jun-Jul-Aug-Sep-Oct-Nov-Dec 20260.99
36. Jan - Feb - Mar - Apr0.91
37. Gemini 3.1 Flash Live0.98
38. Chirp 30.99
39. Gemini 2.5 Flash/Pro TTS0.98
40. Gemini 2.51.00
41. Flash/Pro TTS1.00
42. Gemini 2.5 Flash Native Audio0.99
43. Gemini 2.5 Flash Native Audio1.00
44. Gemini 2.0 Live0.96
45. Audio models1.00
46. Engineering the future of Al0.99
47. AlEng0.96
3:02 #19 done5 line(s)

shot 19·sharpness 1492.6
1. Audio1.00
2. AIE1.00
3. Understanding1.00
4. Engineering the future of Al0.99
5. AlEng0.95
3:07 #20 skipped

shot 20·duplicate of #19
3:13 #21 skipped

shot 21·duplicate of #19
3:19 #22 skipped

shot 22·duplicate of #19
3:25 #23 skipped

shot 23·duplicate of #19

Transcript

240 cues· 2,734 words· 14,590 chars

chunk 0 · 0:15–0:59 · 105 words · 570 chars
0:15 What's new in AI audio?
0:17 I'm sorry, it's a little bit misleading because the title leaves out the Google DeepMind.
0:24 So we're just kind of looking at what we've been working on at DeepMind.
0:28 If we were to look at everything in AI audio, we'd be spending a lot of time here.
0:33 But I'd love to show you what we're working on at DeepMind.
0:40 Yeah, this is me.
0:41 Hi, everyone.
0:42 I'm Thor.
0:43 I work on the developer experience at Google DeepMind, working on the Gemini API and Google AI Studio.
chunk 1 · 0:51–1:31 · 82 words · 429 chars
0:51 Hello, everyone.
0:53 Welcome.
0:53 My name is Thorsten.
0:55 Bonjour.
0:56 Je m'appelle Thor.
0:58 Je suis très désolé.
0:59 Mon français, c'est très mauvais.
1:02 Konnichiwa.
1:02 Au revoir.
1:03 Writing da.
1:07 Okay, that was for the demo and now I just need to make sure last time I did this demo, I recorded over it and then it was all gone.
1:21 That was very sad, but we'll come back to that in a bit.
chunk 2 · 1:26–2:10 · 116 words · 622 chars
1:26 Yeah, what have we been up to at DeepMind?
1:29 There's been a couple releases.
1:32 I actually joined the team in November, literally the day before Gemini 3 was released.
1:38 So I joined, and they told me, tomorrow we're releasing Gemini 3.
1:42 And I was like, yay!
1:43 Didn't do anything, but it was great.
1:46 It was a great time.
1:48 Most recently, on the open model side, we released Gemma 4, I think literally last week.
1:54 And yeah, pretty incredible.
chunk 3 · 1:57–2:36 · 95 words · 518 chars
1:57 Some cool stuff you can do there.
1:58 Multi modality as well, baked into Gemma 4.
2:01 So there's audio understanding in the Gemma 4 models, and you can do that on device, on kind of edge devices as well.
2:10 So that is some very exciting stuff.
2:14 In terms of gen media and audio, you're probably very familiar with our image generation models, video generation models.
2:23 Obviously, Veo has audio generation in there as well.
chunk 4 · 2:28–3:10 · 84 words · 477 chars
2:28 So this is kind of where
2:30 The progression is there most recently with VO 3.1 Lite on the Gen Media model site.
2:36 And then on the audio models, we recently launched Gemini 3.1 Flash Life, which is our kind of full duplex, you know, sound-to-sound, real-time conversational model.
2:50 Also multimodal, so you can ingest real-time text, voice, vision, which we'll look at in a bit.
chunk 5 · 3:01–3:39 · 78 words · 446 chars
3:01 So on audio, very, very broad topic, but the baseline of everything we do are the frontier Gemini models.
3:10 And so Gemini 3 is incredibly good at understanding audio.
3:16 And that's not just transcribing it, but really understanding all the nuances that are in there.
3:23 So that might be obviously speech, but also the context of the speech, the emotion,
3:29 your pacing, your sort of, yeah, anything that sort of swings within the audio that's not just text.
chunk 6 · 3:40–4:24 · 95 words · 596 chars
3:40 So on the audio understanding, our goal is to build models that deeply comprehend, richly transcribe, and robustly reason through audio, seamlessly handling a large mix of different languages, dialect, accents, and modalities.
3:56 And sort of anywhere and always.
3:59 Gemini is really good at transcribing even people that are talking over each other, which is pretty incredible, seamlessly switching between different languages.
chunk 7 · 4:10–4:39 · 78 words · 377 chars
4:10 That was sort of the demo we're looking at now.
4:14 So EchoScript is kind of Gemini 3 Flash preview to sort of analyze audio recordings and extract sort of information out of it.

From Transcription to Live Music: Gemini's Audio Stack — Thor Schaeff, Google DeepMind

Scene timeline

What was stored

Provenance

Frames, and what the machine read

Transcript