Videos 418t26CVz-w

Local Agentic Theory For Mobile Games — Shafik Quoraishee & Joanne Song, The New York Times

index_state ready data_status ok

AI Engineer· published 2026-07-23· 0:18:04· en-US· indexed 2026-08-10 19:43

Open on YouTube

46 shot(s).

keyframes kept every frame deduplicated

What was stored

cues: 168; whisperx 168
chunks: 30; from 168 cues
keyframes: 26; kept of 46 captured
frames with text: 26; 814 lines read
chapters: 10; from the source metadata
keyframe bytes: 5.5 MB; word timings on 168 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage	state	model	started	took
`fetch`	done	—	2026-08-09 23:46	0s
`stt`	done	—	2026-08-09 20:34	31s
`chunk`	done	—	2026-08-09 20:34	0s
`text_embed`	done	—	2026-08-10 19:42	1s
`keyframe`	done	—	2026-08-09 20:35	2m 20s
`ocr`	done	—	2026-08-09 20:37	17s
`frame_embed`	done	—	2026-08-10 19:42	4s

Frames, and what the machine read

0:02 #0 done2 line(s)

shot 0·sharpness 453.4
1. AlEngineer0.96
2. World's Fair0.97
0:03 #1 done2 line(s)

shot 1·sharpness 665.7
1. AlEngineer0.95
2. World's Fair0.99
0:10 #2 done24 line(s)

shot 2·sharpness 2743.2
1. LAB & PLATINUM SPONSORS0.98
2. Amazon AGI Lab0.98
3. ANTHROP\C1.00
4. Google DeepMind1.00
5. MINIMAX0.94
6. OpenAI0.92
7. Akamai1.00
8. arize1.00
9. aws1.00
10. Braintrust bright data0.99
11. B1.00
12. Browserbase1.00
13. docker1.00
14. :neo4j0.93
15. ORACLE1.00
16. PayPal1.00
17. qodo1.00
18. reducto1.00
19. Sonar1.00
20. Makers of0.99
21. togetherai1.00
22. Unblocked1.00
23. WorkOS1.00
24. SonarQube1.00
0:15 #3 done1 line(s)

shot 3·sharpness 265.3
1. World's Fair0.99
0:25 #4 done22 line(s)

shot 4·sharpness 1629.1
1. 05:00 - Google Search0.97
2. Ask Gemin1.00
3. Work0.98
4. AlEngineer0.98
5. ViewFinder Chaperizer WorldCup xoxW SlopDetection AndroidAIClub IntMaterials CUNY Mentorship0.91
6. World's Fair0.99
7. ON-DEVICE1.00
8. EDGE AGENTS0.96
9. GAME THEORY0.93
10. ACCESSIBILITY0.97
11. PRESENTED BY1.00
12. 40.92
13. Microsoft1.00
14. Local Agentic Theory for0.99
15. Accessible Mobile Games1.00
16. Small Al agents that run on the phone and change how easy or hard a0.98
17. game feels, per player. No cloud, no fixed difficulty settings.0.99
18. Shafik Quoraishee0.96
19. Joanne Song1.00
20. 回回0.89
21. World's Fair0.94
22. Engineering the future of Al1.00
1:11 #5 done28 line(s)

shot 5·sharpness 980.8
1. 05:00 - Google Search0.96
2. Ask Gemin1.00
3. ☆0.66
4. Work0.88
5. AlEngineer0.99
6. C NYT Bookmarks0.90
7. 品0.94
8. xxW-Board0.90
9. TOOO0.88
10. ViewFinder Chaperizer WorldCup xoXW SlopDetection AndroidAIClub IntMaterials CUNY Mentorship0.91
11. World'sFair1.00
12. Notes0.99
13. BEFORE WE START0.99
14. 40.57
15. PRESENTED BY1.00
16. Disclaimers1.00
17. Microsoft1.00
18. A0.94
19. Puzzles are made by people1.00
20. No Al in the Games0.97
21. Experimental work1.00
22. The New York Times does not use Al features in its Games.0.99
23. This is experimental work we are doing on our own. It is nesearch, not a product.0.98
24. editors.1.00
25. This is personal, exploratory work. The experiments and views here are our own.0.98
26. World's Fair0.98
27. TRACK 5·JULY 2,20260.97
28. Graphs1.00
1:47 #6 done27 line(s)

shot 6·sharpness 1892.0
1. 05:00 - Google Search0.95
2. Ask Gemin1.00
3. .com/projects-prototype/aiewl/presentation2.html0.98
4. Work0.95
5. AlEngineer1.00
6. Co NYT Bookmarks0.87
7. 品0.96
8. ViewFinder Chapterizer WorlaCup xxWSlopDetection AndroidAIClub IntMaterials CUNY Mentorship0.92
9. World'sFair1.00
10. Notes0.92
11. A deeper dive: my Al Engineer World's Fair0.99
12. NTRO1.00
13. BACKGROUND1.00
14. 2025 talk0.99
15. A longer version of where these ideas come from, from my talk at the Al Engineer World's Fair, 2025.1.00
16. AlEngineer1.00
17. World's Fair0.99
18. Connections May 8.20250.97
19. System 10.94
20. System 21.00
21. TheNewYlorkTimes0.96
22. Al&Game Theory:0.96
23. logical0.98
24. A CASE STUDY ON NYT'S CONNECTIONS0.97
25. World's Fair0.97
26. TRACK 5·JULY 2,20260.97
27. Graphs1.00
1:56 #7 done29 line(s)

shot 7·sharpness 1145.0
1. 05:00 - Google Search0.96
2. Ask Gemini0.94
3. -prototype/aiewl/presentation2.html0.99
4. Work0.94
5. AlEngineer0.99
6. C NYT Bookmarks0.95
7. 品0.95
8. xxw-Board0.91
9. TOOO0.88
10. ViewFinder0.96
11. ChaperizerWorldCup0.96
12. xoW SlopDetection AndroidAIClub InoMaterials CUNY Mentorship0.90
13. World'sFair0.97
14. Notes0.99
15. Al has always been part of games1.00
16. INTRO0.96
17. Game Al isn't new. Pac-Man's ghosts ran a hand-coded finite-state machine in 1980, switching between chasing you,0.99
18. scattering to their corners, and fleeing when you grab a power pelet. Deep Blue beat Kasparov at chess in 1997. AlphaGo0.99
19. beat the world's best at Go in 2016. Today's systems learn whole games from scratch. Each was more capable than the last,0.99
20. and all of them were built to win.1.00
21. GHOST STATE RACHINE0.97
22. SCATTER1.00
23. CHASE0.95
24. FROGHT0.76
25. BATEN0.95
26. red goost FRIGNTENED0.91
27. World's Fair0.97
28. TRACK 5· JULY 2,20260.95
29. Graphs1.00
2:26 #8 skipped

shot 8·duplicate of #7
3:00 #9 done28 line(s)

shot 9·sharpness 1058.5
1. 05:00 - Google Search0.93
2. Ask Gemin1.00
3. 00.68
4. Wark0.90
5. AlEngineer1.00
6. NYT Bookmarks0.96
7. 品0.95
8. xXW-Board0.90
9. ViewFinder0.98
10. Chapterizer1.00
11. WerlaCup0.88
12. xxWSlopDetection0.94
13. AndroidAIClub0.95
14. InoMaterialsCUNYMentorship0.96
15. World's Fair0.99
16. Notes0.92
17. INTRO . WHY ON-DEVICE0.95
18. 40.89
19. Run the model on the0.99
20. device, not the cloud1.00
21. A local model runs entirely on the phone's own chip, with nothing sent1.00
22. to a server. For an agent meant to tune the game to each player, it is the0.99
23. only approach that scales: you cannot run an always-on agent for1.00
24. billions of people in a data center, but you can run it on the device they0.99
25. already own. Doing it locally also changes what the agent can do:1.00
26. World's Fair0.96
27. TRACK 5· JULY 2,20260.95
28. Graphs1.00
3:28 #10 skipped

shot 10·duplicate of #9
3:46 #11 skipped

shot 11·duplicate of #9
4:07 #12 skipped

shot 12·duplicate of #9
4:43 #13 skipped

shot 13·duplicate of #9
5:11 #14 skipped

shot 14·duplicate of #7
5:24 #15 skipped

shot 15·duplicate of #5
5:55 #16 done49 line(s)

shot 16·sharpness 1558.7
1. 05:00 - Google Search0.95
2. Ask Gemin1.00
3. 00.59
4. Work0.91
5. AlEngineer0.98
6. NYT Bookmarks0.98
7. 品0.96
8. xxW-Board0.91
9. TOOO0.86
10. ViewFinder0.97
11. Chapterizer WorlaCup xxWSlopDetection AndroidAiClub0.93
12. IntMaterialsCUNYMentorship0.97
13. World's Fair0.98
14. Notes0.98
15. Reinforcement learning: trained by0.99
16. REINFORCEMENT LEARNING1.00
17. trial and error0.99
18. Reinforcement learning is how most game-playing Al is trained. The system plays, earns a reward when it does well,0.99
19. and nudges its own weights over millions of attempts until the play is good. It work, but it is data-hungry: DQN1.00
20. needed about 200 million frames to reach human-level Atari. EfficientZero is the efficiency leap, above-human Atari0.99
21. from roughly two hours of play, about 500× less data, with V2 pushing the scores higher in 2024. That drop in cost is0.99
22. PRESENTED BY1.00
23. what puts this kind of learner within reach of a phone.1.00
24. Microsoft1.00
25. game experience to reach human-level Atari play1.00
26. ≈500× less data0.97
27. EfficientZero1.00
28. DQN·20151.00
29. log scale0.99
30. 100K0.98
31. 1M0.98
32. 18M0.85
33. 100M0.91
34. 180.88
35. ~0.4M frames. ~2 hrs - superhuman0.95
36. ~200M frames - weeks of play0.97
37. AlphaGo1.00
38. AlphaZero1.00
39. MuZero1.00
40. EfficientZero1.00
41. EZ-V21.00
42. O0.78
43. O0.80
44. still advancing1.00
45. 171.00
46. 191.00
47. World's Fair0.97
48. TRACK 5·JULY 2,20260.97
49. Graphs1.00
6:16 #17 skipped

shot 17·duplicate of #16
6:54 #18 done36 line(s)

shot 18·sharpness 1144.7
1. 05:00 - Google Search0.97
2. Ask Gemin1.00
3. Wark0.90
4. AlEngineer0.99
5. NYT Bookmarks0.97
6. 品0.98
7. xw-Board0.95
8. TODO0.79
9. ViewFinder0.98
10. ChaperizerWorldCup xoxW SlopDetection AndroidAlClub0.91
11. IntMaterialsCUNYMentorship0.94
12. World'sFair1.00
13. Notes0.95
14. REINFORCEMENT LEARNING VS AGENTIC AI1.00
15. Trained for the game, or reasoning about it0.99
16. Both can learn to play, and they borrow heavily from each other. The difference is how each gets good, and how it0.99
17. behaves once it's running.1.00
18. 00.90
19. TRADITIONAL RL1.00
20. AGENTIC AI1.00
21. 3.00M1.00
22. agent1.00
23. situation1.00
24. episodes0.96
25. sk0.78
26. vs0.83
27. episodes1.00
28. no training· reasons each step0.98
29. Learns by trial and error over millions of episodes, chasing a reward.0.98
30. •Reasons over a model that is already trained, with no reward to grind out.0.99
31. Every loop nudges the model's weights, slowly, until the play gets good.0.99
32. The loop is watch, decide, act, and the weights are left untouched.1.00
33. Strong once trained, but the policy is mostly locked in after that.0.98
34. World's Fair0.96
35. TRACK 5· JULY 2, 20260.94
36. Graphs1.00
7:03 #19 done39 line(s)

shot 19·sharpness 1273.3
1. 05:00 - Google Search0.96
2. Ask Gemin1.00
3. ☆0.57
4. AlEngineer0.99
5. C NYT Bookmarks0.94
6. 品0.94
7. xxw-Board0.91
8. TOOO0.83
9. ViewFinder0.91
10. Chaperizer WorldCup0.91
11. xoW SlopDetection0.88
12. AndroidAIClub0.93
13. InoMaterialsCUNYMentorship0.95
14. World's Fair0.97
15. Notes0.92
16. AGENTIC AI. THE AGENTIC MODEL0.93
17. 40.97
18. GAME · arcade shooter0.94
19. THE AGENT - on-device0.97
20. Inside the agent that1.00
21. PERCEIVE1.00
22. plays the shooter0.97
23. read the board into a state0.98
24. act on the game0.99
25. One loop, repeated every frame. The same four jobs run in order, all on1.00
26. PREDICT1.00
27. where each bullet is heading0.99
28. the device, nothing sent to a server.0.99
29. always beats a shot, so the agent breaks off aiming the instant a bullet1.00
30. The policy's priority is clear: stay alive first, then attack. A dodge0.99
31. DECIDE1.00
32. survive first, then aim1.00
33. lines up.1.00
34. ACT1.00
35. fire - dodge0.93
36. one loop, every frame - -4 ms each - on-device0.97
37. World's Fair0.98
38. TRACK 5· JULY 2,20260.96
39. Graphs1.00
7:43 #20 skipped

shot 20·duplicate of #19
8:01 #21 skipped

shot 21·duplicate of #19
8:30 #22 done33 line(s)

shot 22·sharpness 1061.8
1. 05:00 - Google Search0.96
2. Ask Gemin1.00
3. à0.81
4. AlEngineer0.99
5. C NYT Bookmarks0.95
6. 品0.95
7. xxw-Board0.97
8. TODO0.85
9. ViewFinder Chaperizer WorldCup xoxW SlopDetection AndroidAIClub IntMaterials CUNY Mentorshp0.93
10. World'sFair0.99
11. Notes0.91
12. Three budgets, all on the phone at once0.99
13. CHALLENGES1.00
14. THE DEVICE BUDGET0.99
15. Everything runs on the device, sharing it with the game. That puts the agent inside three hard limits at the same time: the0.99
16. space in memory, the time between frames, and the energy in the battery.1.00
17. 40.92
18. Space1.00
19. nenory In kan0.77
20. and the Live working set all share one timited0.98
21. DEVICE RAM BUDGET1.00
22. model weights1.00
23. 42%0.99
24. compressed state history0.97
25. 20%0.94
26. working set / planning0.98
27. 24%0.85
28. render headroon0.99
29. 14%0.99
30. state history1.00
31. World's Fair0.96
32. TRACK 5· JULY 2,20260.97
33. Graphs1.00
8:53 #23 done38 line(s)

shot 23·sharpness 1136.7
1. 05:00 - Google Search0.97
2. Ask Gemin0.99
3. 台0.67
4. à0.88
5. Work0.99
6. AlEngineer0.98
7. NYT Bookmarks0.94
8. 品0.99
9. ViewFinder0.96
10. ChapterizerWorldCup0.97
11. xoW SlopDetection0.91
12. AndroidAIClub0.94
13. IntMaterials0.93
14. CUNY Mentorship0.98
15. World's Fair0.98
16. Notes0.93
17. Three budgets, all on the phone at once0.99
18. CHALLENGES1.00
19. THE1.00
20. DEVICE BUDGET1.00
21. Everything runs on the device, sharing it with the game. That puts the agent inside three hard limits at the same time: the0.99
22. space in memory, the time between frames, and the energy in the battery.1.00
23. Space1.00
24. senory In Ran0.82
25. Time1.00
26. DEVICE RAM BUDGET1.00
27. model weights0.97
28. 42%0.99
29. compressed state history0.97
30. 20%1.00
31. working set / planning0.99
32. 24%0.98
33. render headroon0.98
34. 14%0.92
35. state history1.00
36. World's Fair0.95
37. TRACK 5·JULY 2,20260.97
38. Graphs1.00

Transcript

168 cues· 2,966 words· 16,601 chars

chunk 0 · 0:12–0:57 · 134 words · 707 chars
0:12 Awesome.
0:13 So yes, as a great introduction took us to, we are going to be talking about local agentic theory for accessible mobile games.
0:21 And we realize this is the graph track, but there are graphs here in this presentation.
0:25 So fret not if you're missing them.
0:28 So yeah, so a few disclaimers before we continue.
0:33 Since we do work at the New York Times, and there's a few things we have to say about our puzzles, AI, et cetera, our puzzles are made by people.
0:40 They're not made by AI, so that's just a thing that is true, and it'll always be true.
chunk 1 · 0:47–1:26 · 109 words · 601 chars
0:47 There's no AI in the games themselves, so there's no AI features.
0:51 You might have seen Wordlebot.
0:52 That's not an AI feature.
0:54 So all our games are pretty much AI-free.
0:58 And the work that we're going to be talking about now is experimental work.
1:03 So it's stuff that we look at in terms of solvability and other capabilities of agents in the space that we can potentially take advantage of on a local scale.
1:12 device to do other things and actually generate games, more so things about playability and other kinds of features we can utilize agents for in an intelligent way.
chunk 2 · 1:23–2:07 · 136 words · 667 chars
1:23 So a little bit of an introduction.
1:26 So I also, if anyone has seen the last year's World Fair talk that I did on connections, there's a similar vibe to this.
1:36 I did a deeper dive into how a solver can be built for that game.
1:40 And if you're interested, you can take a look.
1:42 It's on YouTube, and it has a lot of interesting deeper dive specifics for RL solvers, which I'll go into in a bit as well.
1:50 So a little bit of a history.
1:52 Everybody here, and just in case you aren't familiar with AI in gaming, which for, you know, it's not the most common topic here at the World Fair right now, but if you are familiar with the history of it, AI has been in games for a long time.
chunk 3 · 2:08–2:43 · 96 words · 514 chars
2:08 And since the 80s, it's been established as a core pattern in games like Pac-Man.
2:16 This is a simple version of an AI called a finite state machine.
2:19 And if anybody, everyone's familiar with a finite state machine here, probably.
2:23 or I would imagine enough people are.
2:25 It's a basic AI system which tells the ghosts in the game what to do when Pac-Man gets a dot or not.
2:33 But basically it is a conditional kind of symbolic AI.
chunk 4 · 2:38–3:16 · 101 words · 584 chars
2:38 And then moving forward, we'll talk more about RL and the advancements of where that went to.
2:43 But firstly, the core thesis of what we're going to really focus on is the model of running the devices, running AI on our local devices.
2:54 So most AI infrastructure today runs really on the cloud.
2:59 And if you're running a mobile application,
3:02 Most practical scenarios involve cloud architecture, and that essentially is expensive because you have AI calls and inference calls that you have to make upstream, and then they have to come to your device.
chunk 5 · 3:13–3:53 · 118 words · 636 chars
3:13 So there's things like latency, et cetera.
3:16 So in an ideal world, when we evolve to that place, we can offload a lot of the intelligence onto a device.
3:24 What that does is make
3:27 make a lot of the calls faster when we get to that world because you're not doing a round trip to the cloud to get that information.
3:33 So you can actually compress that time to the time within the device frame computation.
3:40 It's private, too, so your AI computation is within the device itself, and if it's set up properly, it won't leave the device because there's some things that, for the gameplay experience, that you don't actually need it to leave the device.
chunk 6 · 3:53–4:37 · 125 words · 686 chars
3:53 Local computation is the way to go, and it stays within your confined security zone locally, and it's not needed upstream for telemetry or anything like that.
4:03 And then it should work anywhere.
4:04 So one thing is that games, a lot of games rely on the internet, and they will for a live connection point, but at least in many cases where you need advanced AI compute, you can develop games that can work locally on your machine, on your mobile device, rather.
4:23 And that's great because if you're in a subway tunnel or something like that, you can still have a game with a very smart AI that is building, doing something without being disrupted due to HTTP calls that are not getting through.
chunk 7 · 4:39–5:19 · 100 words · 585 chars
4:39 And then it's, again, personalization is a big thing in general with on-device models because personalization is important.
4:46 And a lot of people want their game experience to be tailored to them.
4:49 And in a world where we can do things on-device, we have that capability more so because the features are local.
4:57 So now we're going to talk about agentic AI for games.
5:01 And as I'm sure everybody's familiar with AlphaGo, AlphaZero, et cetera here, the Alpha series models.
chunk 8 · 5:09–5:44 · 89 words · 480 chars
5:09 So basically reinforcement learning was the way that games were in the past
5:16 trained in order to do intelligence, right?
5:20 So basically what you would do is that you would take a model, have many iterations, and then that model would then be really tuned to a particular game.

Chapters

0:00 Disclaimers and the experimental frame
1:55 A short history of AI in games
2:49 Why run the AI on the device, not the cloud
4:56 From reinforcement learning to agentic play
7:00 Demo: an agent playing Space Invaders
8:19 The on device budget: space, time, and energy
11:15 Demo: solving the mini crossword by backtracking
12:33 Accessibility: from toggles to graded dials
14:40 The agent tuning the game to you in real time
16:28 What is still needed, and billions of local brains