Text Diffusion — Brendan O’Donoghue, Google DeepMind

1. chunk 0 · 0:15–0:50 · 108 words · 632 chars
2. 0:15 So people are still filtering into the room.
3. 0:18 It's mostly intro stuff for the first couple of slides, so they won't miss anything.
4. 0:21 OK, welcome, everybody.
5. 0:22 My name's Brendan.
6. 0:23 I'm a research scientist at DeepMind.
7. 0:25 I'm talking today about text diffusion, which is kind of a more forward-looking research area at DeepMind.
8. 0:32 So you're probably familiar with image and video diffusion, which is kind of state of the art for these modalities right now, where you take ground truth, say image, you add noise to it in training, and then you train a neural network to remove that noise gradually.
9. chunk 1 · 0:48–1:31 · 149 words · 821 chars
10. 0:48 And then at inference time, you just initialize
11. 0:51 the picture with pure noise, and then you iteratively refine out the noise to recover back to whatever image or video or audio or whatever you're looking for.
12. 1:00 And the principle is essentially the same for text, for text diffusion, where you start with a clean sequence of tokens, so a clean sentence or something like that.
13. 1:10 And then you gradually add noise.
14. 1:12 You corrupt it somehow.
15. 1:13 There's lots of different ways to do that.
16. 1:15 You can do it in a continuous or discrete way.
17. 1:17 But let's just say discrete for now, which would, in this case, just mean adding random tokens or replacing tokens with other random tokens.
18. chunk 2 · 1:23–2:05 · 135 words · 725 chars
19. 1:23 And you do that for a bunch of different noise levels.
20. 1:25 And you train the neural network to try to fill in, to try to correct
21. 1:30 the mistakes basically in the text.
22. 1:32 And then at inference time, you initialize the sequence of tokens to just pure noise, like pure random discrete tokens from the vocabulary.
23. 1:40 And then you iteratively refine through that to fill in the information in the order that the neural network wants to do to recover back to, say, a clean sentence.
24. 1:49 And then in practice, I showed you some GIFs here of what it looks like for images.
25. chunk 3 · 1:54–2:22 · 83 words · 453 chars
26. 1:54 You get very similar looking outputs for text, where it kind of starts off all noisy and then gradually fills in the text.
27. 2:01 And then you get relatively clean outputs at the end.
28. 2:06 OK, so the team I'm on, we had a research demo release one year ago now called Gemini Diffusion, which was a variant of a Gemini model with text diffusion instead of autoregressive next token generation.
29. 2:19 And that was like a research preview that was open to about 100k people.
30. chunk 4 · 2:23–3:06 · 139 words · 780 chars
31. 2:23 uh and you know we're still you know keep keep posted for new new developments in that direction uh upcoming soon um and we did have some good numbers at the time but again it's a year ago which is like you know prehistoric times in this field uh our kind of main comparator model was gemini 2.0 flashlight at the time because that was the architecture we were branching from the text fusion model and we basically had very similar quality across the board there
32. 2:50 Mostly a little bit of advantage in code, a little bit of disadvantage in some other areas, but kind of relatively similar performance at much better latencies.
33. chunk 5 · 3:00–3:42 · 144 words · 814 chars
34. 3:00 But again, this is a year ago, so I wouldn't fixate too much on these numbers.
35. 3:04 OK, so what's the difference between autoregressive generation and diffusion?
36. 3:06 So in the standard vanilla Gemini, Gemma, GPT, whatever, generation of text, you do this.
37. 3:15 You have some context that comes in, and you want to generate some response to that.
38. 3:18 And the model does it one token at a time.
39. 3:20 So it generates the first token, and then conditional on that generates the next token, and so on.
40. 3:24 Whereas in diffusion, a context will come in, whatever that is, and it'll initialize, like I mentioned, like a long sequence of tokens, could be hundreds, could be thousands, could be shorter, depends on the model, to be random noise.
41. chunk 6 · 3:37–4:22 · 150 words · 812 chars
42. 3:37 And then it iteratively refines that canvas to remove the noise over the course of a few denoising steps.
43. 3:43 So rather than one token at a time, it does the entire block together.
44. 3:47 but over a couple of iterations.
45. 3:49 So it's not just one pass, it has multiple passes, but it gets to attend to the future tokens and so on.
46. 3:55 So it's kind of a different way of generating text.
47. 4:00 So that obviously has some pros and cons.
48. 4:02 So the main pro that people really like, and probably is the biggest advantage that text diffusion models have, is that it's just faster inference, it just generates faster tokens per second, because it makes much better use of the hardware, the TPU and the GPU.
49. chunk 7 · 4:15–4:55 · 130 words · 747 chars
50. 4:15 And I have some slides on that to explain why.
51. 4:17 But some other advantages are it can do bidirectional attention within this canvas of tokens.
52. 4:22 So, you know, autoregressive models can only attend to the past.
53. 4:26 They have causal attention within their, you know, their transformer.
54. 4:30 Whereas a text fusion model is not restricted, though.
55. 4:32 They can attend to the future.
56. 4:34 And that has some interesting properties, like it can do self-corrected generation based on future tokens.
57. 4:39 So we could do some reasoning, see that I got the answer incorrect, and then go back and fix the reasoning and do it again.
58. chunk 8 · 4:45–5:24 · 122 words · 679 chars
59. 4:45 I have a demo of that.

loading