Videos hMlLw1LeIK8

Voice Agents That Handle Interrupts - Chintan Agrawal and Daniel Wirjo, AWS

index_state ready data_status ok

AI Engineer· published 2026-07-20· 0:32:56· en-US· indexed 2026-08-10 19:44

Open on YouTube

80 shot(s).

keyframes kept every frame deduplicated

What was stored

cues: 266; whisperx 266
chunks: 56; from 266 cues
keyframes: 32; kept of 80 captured
frames with text: 32; 1,730 lines read
chapters: 0; from the source metadata
keyframe bytes: 10.8 MB; word timings on 266 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage	state	model	started	took
`fetch`	done	—	2026-08-10 00:28	1m 55s
`stt`	done	—	2026-08-10 00:30	30s
`chunk`	done	—	2026-08-10 00:31	0s
`text_embed`	done	—	2026-08-10 19:43	1s
`keyframe`	done	—	2026-08-10 00:31	1m 35s
`ocr`	done	—	2026-08-10 00:33	34s
`frame_embed`	done	—	2026-08-10 19:43	5s

Frames, and what the machine read

0:08 #0 done8 line(s)

shot 0·sharpness 1283.9
1. Chintan Agrawal1.00
2. AI ENGINEER WORLD'S FAIR 20260.99
3. Voice agents0.97
4. that handle1.00
5. interrupts1.00
6. The real-time audio engineering nobody talks about1.00
7. Chintan Agrawal & Daniel Wirjo· AWS Solutions Architects0.99
8. 2026-06-23 23:58:520.98
0:50 #1 skipped

shot 1·duplicate of #0
0:58 #2 done24 line(s)

shot 2·sharpness 2534.0
1. amazon0.96
2. aws1.00
3. Chintan Agrawal1.00
4. WHAT WE'LL COVER1.00
5. Eight stops1.00
6. 011.00
7. What does a natural-sounding Al voice agent feel1.00
8. 051.00
9. Level 2 — built-in turn detection0.99
10. like?1.00
11. 021.00
12. The200ms constraint1.00
13. 061.00
14. Level 3 — Smart Turn, own your stack0.98
15. 031.00
16. Architecture — the pipeline0.98
17. 071.00
18. Latency budget1.00
19. 041.00
20. Level 1 — Silero VAD0.99
21. 081.00
22. Production lessons & open problems0.99
23. 02 / 230.97
24. 2026-06-23 23:59:520.98
1:42 #3 done21 line(s)

shot 3·sharpness 2356.8
1. aws1.00
2. 70.94
3. Chintan Agrawal1.00
4. 011.00
5. INTRODUCTION1.00
6. Same user. Same sentence. Different result.0.98
7. The difference between a voice agent that feels broken and one that feels natural comes down to how well it1.00
8. handles interrupts and turn-taking.0.98
9. BROKEN1.00
10. WORKING1.00
11. User: "I want to fly to—"1.00
12. User: "I want to fly to—"0.97
13. Agent keeps talking — ignores interruption for 1.8 s0.99
14. STOP — interruption detected0.96
15. User: "no wait, a HOTEL—" (ignored)0.98
16. User: "actually, a hotel in Sydney"1.00
17. 1.8 s delay0.97
18. Agent: "Hotel in Sydney — got it!"0.97
19. 180 ms1.00
20. 03 / 230.93
21. 2026-06-2400:00:451.00
2:33 #4 skipped

shot 4·duplicate of #3
2:38 #5 done10 line(s)

shot 5·sharpness 1208.5
1. aws1.00
2. Chintan Agrawal1.00
3. 021.00
4. . THE CONSTRAINT0.94
5. The 200 ms0.99
6. constraint1.00
7. 04 / 230.93
8. 4/230.91
9. ResetR1.00
10. 2026-06-24 00:01:521.00
3:01 #6 done25 line(s)

shot 6·sharpness 2145.1
1. aws1.00
2. Chintan Agrawal1.00
3. THE 200 MS TARGET0.99
4. Human turn-gap: measured across 10 languages1.00
5. 200 ms is the time from when the user stops speaking to when the agent starts responding. Below that,0.99
6. conversation feels natural. Above it, users pause, repeat themselves, or assume the call dropped. Chat agents0.99
7. get seconds — voice agents don't get a second chance.0.98
8. 0 ms0.97
9. 200 ms1.00
10. 400 ms1.00
11. 800 ms0.99
12. 1500 ms+0.98
13. NATURAL1.00
14. ACCEPTABLE1.00
15. NOTICEABLE1.00
16. BROKEN1.00
17. CALL DROPPED?1.00
18. 200 ms1.00
19. 755 ms0.99
20. Best measured human TTFA1.00
21. Best measured TTFA — cascaded pipeline0.99
22. Stivers et al., PNAS 2009 — 10 languages, 10K+ conversations0.99
23. Qiu et al., Salesforce 20260.99
24. 05 / 230.97
25. 2026-06-24 00:02:200.99
3:28 #7 skipped

shot 7·duplicate of #6
3:51 #8 done10 line(s)

shot 8·sharpness 1229.5
1. amazon1.00
2. aws1.00
3. Chintan Agrawal1.00
4. 031.00
5. ARCHITECTURE1.00
6. The voice0.95
7. agent pipeline1.00
8. 031.00
9. 06 / 230.88
10. 2026-06-24 00:03:200.98
3:58 #9 done36 line(s)

shot 9·sharpness 2523.3
1. amazon0.99
2. aws1.00
3. Chintan Agrawal0.99
4. FRAME-BASED· ASYNC· BACKPRESSURE-AWARE0.98
5. Voice agents need a pipeline0.99
6. Speech-to-Text (STT), a Language Model (LLM), and Text-to-Speech (TTS) are all critical components of a voice agent. But0.99
7. VAD — voice activity detection — is an often-overlooked layer. It decides when the user has finished speaking. Get it wrong0.98
8. and everything downstream breaks.1.00
9. WebRTC1.00
10. VAD1.00
11. STT0.84
12. LLM1.00
13. TTS1.00
14. WebRTC1.00
15. DAILY1.00
16. SILERO V51.00
17. CARTESIA INK0.96
18. CLAUDE / GPT0.98
19. →1.00
20. CARTESIA1.00
21. DAILY1.00
22. 20 MS FRAMES0.95
23. 32 MS FRAMES1.00
24. STREAMING1.00
25. TOKEN STREAM1.00
26. PCM 24 KHZ1.00
27. OUT1.00
28. SMART TURN DETECTION1.00
29. INTERRUPTION HANDLER1.00
30. Listens to VAD + audio features. Decides: respond or wait.0.99
31. Propagates InterruptionFrame — flushes TTS + LLM buffers.0.99
32. WHAT IS PIPECAT?1.00
33. 10k+0.97
34. Open-source framework for real-time voice agents — wires STT, LLM, TTS, and VAD into a frame-based pipeline with backpressure, interruption handling, and streaming built in.1.00
35. GITHUB STARS1.00
36. 2026-06-24 00:03:290.99
4:21 #10 skipped

shot 10·duplicate of #9
4:47 #11 skipped

shot 11·duplicate of #9
5:17 #12 done9 line(s)

shot 12·sharpness 1384.0
1. aws1.00
2. Chintan Agrawal1.00
3. 04. LEVEL 10.91
4. Silero VAD:1.00
5. take control1.00
6. The simplest component you fully own. This is where most production1.00
7. systems start.1.00
8. 08/231.00
9. 2026-06-24 00:05:030.99
5:54 #13 done32 line(s)

shot 13·sharpness 1662.7
1. amazon1.00
2. aws1.00
3. 70.91
4. Chintan Agrawal0.98
5. 041.00
6. LEVEL11.00
7. Silero VAD — 309K parameters, <1 ms0.95
8. STFT1.00
9. 4×Conv1d1.00
10. LSTM (128)0.97
11. Linear1.00
12. 0→11.00
13. 16 KHZ PCM0.98
14. SPECTRAL1.00
15. FEATURES1.00
16. + RELU0.98
17. TEMPORAL0.99
18. CONTEXT1.00
19. P(SPEECH)1.00
20. SIGMOID0.95
21. SPEECH PROB0.99
22. 309K1.00
23. 2 MB0.96
24. <1 ms0.98
25. MCC 0.720.96
26. PARAMETERS1.00
27. MODEL SIZE0.99
28. PER 32 MS FRAME1.00
29. VS WEBRTC 0.411.00
30. vad = SileroVADAnalyzer(params=VADParams(stop_secs=0.3)) # confirmed: Qiu et al. (2026) same config0.99
31. 09 / 230.97
32. 2026-06-24 00:05:480.99
6:04 #14 skipped

shot 14·duplicate of #13
6:50 #15 done31 line(s)

shot 15·sharpness 1514.2
1. amazor0.99
2. aws1.00
3. Chintan Agrawal1.00
4. 041.00
5. LEVEL 11.00
6. Tuning stop_secs — the critical parameter0.99
7. SALES / 0.2 50.97
8. GENERAL / 0.5 50.96
9. HEALTH / 0.8 S0.95
10. DATA ENTRY / 1.2 S0.97
11. Snappy. May cut off1.00
12. Balanced. Most users1.00
13. Patient. Respects1.00
14. Never interrupts. Dead-air1.00
15. thinking.1.00
16. happy.1.00
17. pauses.1.00
18. risk.1.00
19. # The critical parameter1.00
20. class VADParams:0.99
21. threshold1.00
22. = 0.50.92
23. min_speech_ms1.00
24. = 2500.93
25. min_silence_ms1.00
26. = 3000.94
27. # THIS ONE0.96
28. speech_pad_ms1.00
29. = 1000.95
30. 10 / 230.97
31. 2026-06-2400:06:551.00
7:12 #16 skipped

shot 16·duplicate of #9
7:36 #17 skipped

shot 17·duplicate of #9
8:12 #18 done24 line(s)

shot 18·sharpness 2406.4
1. amazon0.99
2. aws1.00
3. Chintan Agrawal1.00
4. THREE LEVELS OF SOLVING THIS - INCREASING CONTROL, INCREASING COMPLEXITY0.99
5. What VAD alone can't solve0.97
6. 011.00
7. SILENCE DETECTION0.99
8. 021.00
9. BARGE-IN1.00
10. 031.00
11. TURN DETECTION1.00
12. When did they stop talking?0.99
13. They started talking. Stop, fade,0.99
14. Same silence means four things1.00
15. or finish?1.00
16. Is the user done, or just thinking? 300 ms and 1200 ms0.99
17. Complete sentence. Incomplete thought. Thinking1.00
18. silence look identical to raw VAD.0.99
19. The decision depends on context — correction,0.99
20. pause. Backchannel "yeah". VAD fires on all of them.0.99
21. affirmation, or ambient noise. Most agents don't0.99
22. distinguish.1.00
23. 11 / 230.93
24. 2026-06-24 00:08:330.98
8:28 #19 done13 line(s)

shot 19·sharpness 1399.6
1. amagon0.99
2. aws0.99
3. Chintan Agrawal1.00
4. 05LEVEL 21.00
5. Built-in turn1.00
6. detection1.00
7. Smarter than raw VAD. State-of-the-art STT models are now starting to incorporate1.00
8. turn detection.1.00
9. 050.99
10. 12/231.00
11. 12/231.00
12. ResetR1.00
13. 2026-06-2400:08:521.00
8:51 #20 done22 line(s)

shot 20·sharpness 2507.9
1. amazon1.00
2. aws1.00
3. 70.91
4. Chintan Agrawal1.00
5. 051.00
6. LEVEL 20.97
7. Cartesia Ink-2 and Deepgram Flux1.00
8. CARTESIA INK-21.00
9. DEEPGRAM FLUX1.00
10. Turn detection built into the STT WebSocket protocol — emits turn. start0.99
11. Endpointing built into the Deepgram streaming API — single-service turn0.99
12. / turn.end events0.99
13. detection1.00
14. No local VAD or turn analyzer needed — the server drives boundaries1.00
15. Configurable endpointing parameter controls silence threshold0.99
16. P50 STT latency: 299 ms · P95: 328 ms0.97
17. P50 STT latency: 247 ms · P95: 298 ms (nova-3)0.98
18. Pipecat: CartesiaTurnsSTTService0.99
19. Trade-off: no visibility into model decisions, vendor-specific behaviour0.98
20. Both are Level 2 — smarter than raw VAD, no local model required. Best for prototypes and latency-sensitive deployments where control is less critical.0.98
21. 13 / 230.89
22. 2026-06-24 00:09:200.99
9:16 #21 skipped

shot 21·duplicate of #3
9:41 #22 skipped

shot 22·duplicate of #20
10:06 #23 done10 line(s)

shot 23·sharpness 1284.5
1. amazon0.98
2. aws1.00
3. Chintan Agrawal1.00
4. 06. LEVEL 30.93
5. Silero VAD +1.00
6. Smart Turn0.99
7. Full control. Transparent. Tunable. Portable.1.00
8. 061.00
9. 14/231.00
10. 2026-06-2400:10:501.00

Transcript

266 cues· 4,590 words· 24,790 chars

chunk 0 · 0:03–0:43 · 119 words · 665 chars
0:03 Hey, everyone.
0:04 I'm Chintan, and I have my colleague Daniel with me.
0:07 We are solutions architect on the AWS APG startup team.
0:11 So today, we are going to talk about something we've been working on for a while, which is turn-taking and voice agents.
0:18 What I mean by that is, how does the system actually know that you have finished speaking and it's time for the agent to respond?
0:25 These are all audio engineering problems.
0:27 They are not LLM problems, because you can have the perfect model, perfect track,
chunk 1 · 0:33–1:11 · 85 words · 479 chars
0:33 But the experience still might feel broken if the turn taking is off.
0:38 So we are going to walk you through the concepts and three different approaches to solving this.
0:44 And Daniel is going to make it more concrete by running all the three demo for you.
0:56 So to kick things off, we'll start with the 200 millisecond constraint, which is basically the physics of why voice is hard.
chunk 2 · 1:06–1:44 · 98 words · 552 chars
1:06 Then we look into the pipeline architecture, what are the components, where turn taking actually lives.
1:12 We'll then look at three levels of solving this problem, going from simple silence detection all the way up to running your own turn detection model.
1:21 and then we'll also talk about the latency the budget and production issues that we've seen so but before any of that i just want to show you the problem statement more clearly because you have the same user you have the same sentence on the both side but the user outcome is very different because
chunk 3 · 1:45–2:28 · 121 words · 659 chars
1:45 On the left, when the user says, I want to fly, and midway they try to correct themselves, the agent just does not notice because it keeps going on for almost two seconds while the person is sitting there trying to get a word in.
2:00 On the right side, the same exact interaction is happening, but the agent is able to capture the interruption in under 200 milliseconds and immediately backs out so the user cannot speak.
2:13 Both the scenarios, the LLM was identical.
chunk 4 · 2:15–2:46 · 81 words · 446 chars
2:15 It was the same model.
2:16 It was the same problem.
2:17 But the difference in the user experience is so different.
2:21 And the difference is purely because of the audio pipeline.
2:26 And how fast does it notice us?
2:28 Like someone else is talking and it knows when to shut up.
2:31 Like that's the problem we are trying to solve today.
2:37 So we cover this 200 millisecond constraint because 200 millisecond is how fast humans switch turns with each other in a conversation.
chunk 5 · 2:47–3:29 · 110 words · 618 chars
2:47 And the implications are pretty brutal because at 800 milliseconds, things start to feel off while at 1.5 second, your user would just hang up on you because you have like your HR chat agents might get five seconds to respond and nobody will care.
3:05 But with the voice agents, you don't get that luxury.
3:08 We've seen recently like Salesforce was able to pay for the pipeline and they published their results in March 26.
3:15 But even their best measure response time was 755 milliseconds.
chunk 6 · 3:19–3:52 · 94 words · 505 chars
3:19 So that's like almost 4x more than how humans naturally take turns while talking.
3:26 So the question here becomes, how do we close this gap?
3:29 And where we cannot close it, at least how do we get the turn taking right?
3:34 So the user experience does not feel as bad as the raw numbers might suggest here.
3:40 So if you're working with a budget of 755 millisecond and every millisecond count, you also need to understand what's actually in the pipeline.
3:49 Like where does the time go and where does this turn taking fit?
chunk 7 · 3:54–4:32 · 102 words · 560 chars
3:54 so the pipeline itself you probably might know the main components the std the llm the tts you have the audio in text out text in audio but the piece that's often missing like from people while people are thinking about this is this voice activity detection commonly called as bad it's a tiny component that sits right at the front and it is the one that controls term thinking
4:21 Its job is to basically detect whether the user has stopped talking or not.
chunk 8 · 4:26–5:03 · 107 words · 600 chars
4:26 And the two other things that you said alongside the main pipeline are also very critical for turn-taking.
4:33 So you have the smart turn detection, which watches the VRE signal, plus the audio features, and then it has to make that decision that whether should it respond now or should it wait because the user might still be talking.
4:47 And the interruption handler is triggered when someone barges in.
4:51 So what it'll do is it'll propagate a flush of downstream, like the TTS would stop and the LLM generation need to be canceled so that the pipeline is ready for the new input within about 15 millisecond.
chunk 9 · 5:05–5:45 · 93 words · 508 chars
5:05 So if you remember, I was talking about how there are different levels to building this turn taking in voice agent.
5:15 So we'll start with the level one.
5:17 Level one is your cellular VAD.
5:19 So this is a simple component which you fully own.
5:22 And like a lot of production system, even today, we see are running agents just by using this.
5:30 So coming to cellular VAD, it's a small 300,000 parameter model.
chunk 10 · 5:37–6:18 · 108 words · 597 chars
5:37 It takes in a short-term Fourier transform, and it takes raw audio, and it will convert it into spectral features.
5:46 It has four convolutional layers to pick up the pattern, an LSTM that gives it memory across the frame, so it's just not looking at one chunk in isolation.
5:57 And then it has a sigmoid that gives a probability of speech, and it's like a very small 2-megabyte model.

Voice Agents That Handle Interrupts - Chintan Agrawal and Daniel Wirjo, AWS

Scene timeline

What was stored

Provenance

Frames, and what the machine read

Transcript