Videos ObTPqBGsEbA

The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks

index_state ready data_status ok

AI Engineer· published 2026-06-18· 0:37:05· en-US· indexed 2026-08-10 19:52

Open on YouTube

85 shot(s).

keyframes kept every frame deduplicated

What was stored

cues: 403; whisperx 403
chunks: 64; from 403 cues
keyframes: 58; kept of 85 captured
frames with text: 58; 1,391 lines read
chapters: 0; from the source metadata
keyframe bytes: 10.2 MB; word timings on 403 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage	state	model	started	took
`fetch`	done	—	2026-08-10 17:01	1m 46s
`stt`	done	—	2026-08-10 17:03	40s
`chunk`	done	—	2026-08-10 17:04	0s
`text_embed`	done	—	2026-08-10 19:52	1s
`keyframe`	done	—	2026-08-10 17:04	3m 06s
`ocr`	done	—	2026-08-10 17:07	29s
`frame_embed`	done	—	2026-08-10 19:52	10s

Frames, and what the machine read

0:03 #0 done2 line(s)

shot 0·sharpness 658.4
1. AlEngineer0.98
2. EUROPE1.00
0:08 #1 done2 line(s)

shot 1·sharpness 829.1
1. PRESENTINGSPONSOR1.00
2. Google DeepMind1.00
0:13 #2 done3 line(s)

shot 2·sharpness 907.7
1. PLATINUM SPONSORS0.98
2. # Braintrust0.96
3. WorkOS OpenAI0.95
0:24 #3 done11 line(s)

shot 3·sharpness 426.9
1. The Producti1.00
2. AlEngineer0.99
3. EUROPE1.00
4. AI Playbook1.00
5. Learnings from deploying agents in enterprises0.98
6. SWIACC0.66
7. Sandipan Bhaumik1.00
8. Data & Al Tech Lead0.99
9. Databricks1.00
10. AlEngineer0.99
11. EUROPE1.00
0:54 #4 done10 line(s)

shot 4·sharpness 431.8
1. The Producti0.97
2. AlEngineer0.99
3. EUROPE1.00
4. AI Playbook1.00
5. Learnings from deploying agents in enterprises1.00
6. Sandipan Bhaumik0.98
7. Data & Al Tech Lead0.96
8. Databricks1.00
9. AlEngineer0.99
10. EUROPE1.00
1:07 #5 skipped

shot 5·duplicate of #3
1:44 #6 done11 line(s)

shot 6·sharpness 429.1
1. The Producti0.97
2. AlEngineer0.99
3. EUROPE1.00
4. AI Playbook1.00
5. Learnings from deploying agents in enterprises0.99
6. EUIACC0.80
7. Sandipan Bhaumik1.00
8. Data & Al Tech Lead0.96
9. Databricks1.00
10. AlEngineer0.99
11. EUROPE1.00
2:09 #7 done29 line(s)

shot 7·sharpness 1809.8
1. THEPROBLEM1.00
2. The pattern you already know0.99
3. *★★0.63
4. AIE1.00
5. ★1.00
6. Weeks 1-40.96
7. Weeks 4–80.95
8. Weeks 8-120.99
9. Week 141.00
10. Month 61.00
11. ★1.00
12. ★1.00
13. ★1.00
14. ★1.00
15. Pick models.0.98
16. Build features.1.00
17. Demo to leaders.0.99
18. "Why is Al1.00
19. $$in projects0.99
20. Looks great.0.96
21. Sign-off.Ship.1.00
22. b*s*-ing us?"0.99
23. failed last year.0.97
24. Sound familiar? You're not here because you haven't seen this. You're here because you want to stop it.1.00
25. AIEngin0.95
26. EUROPE1.00
27. Engineering the future of Al0.99
28. AlEngineer1.00
29. 20261.00
2:36 #8 skipped

shot 8·duplicate of #7
3:15 #9 done22 line(s)

shot 9·sharpness 2644.7
1. THE INSIGHT1.00
2. The AI is the easy part1.00
3. You can't debug what you can't see.1.00
4. 011.00
5. *★*0.57
6. Observability gap1.00
7. AIE1.00
8. ★1.00
9. ★1.00
10. ★1.00
11. You can't improve what you can't measure.1.00
12. 021.00
13. Evaluation gap1.00
14. You can't trust what you can't explain.1.00
15. 031.00
16. Governance gap1.00
17. AIEngin0.93
18. EUROPE1.00
19. AlEngineer0.96
20. AlEngineer0.97
21. 20260.99
22. EUROPE1.00
3:37 #10 skipped

shot 10·duplicate of #9
4:05 #11 done20 line(s)

shot 11·sharpness 2713.2
1. THE INSIGHT1.00
2. The AI is the easy part0.98
3. You can't debug what you can't see.1.00
4. 011.00
5. Observability gap1.00
6. AIE1.00
7. ★1.00
8. ★1.00
9. ★1.00
10. You can't improve what you can't measure.1.00
11. 021.00
12. Evaluation gap1.00
13. You can't trust what you can't explain.0.99
14. 031.00
15. Governance gap1.00
16. AlEngin0.95
17. EUROPE1.00
18. Engineering the future of Al1.00
19. AlEngineer1.00
20. 20261.00
4:36 #12 skipped

shot 12·duplicate of #11
4:48 #13 done33 line(s)

shot 13·sharpness 2691.4
1. THE FRAMEWORK1.00
2. The Five Pillars of Production AI0.98
3. 011.00
4. 021.00
5. 031.00
6. 041.00
7. 051.00
8. *★★0.61
9. AIE1.00
10. ★1.00
11. Evaluation1.00
12. Observability1.00
13. Data Foundation1.00
14. Orchestration1.00
15. Governance1.00
16. ★1.00
17. ★1.00
18. ★1.00
19. ★1.00
20. Define success before1.00
21. See everything, always1.00
22. Question + Tracking1.00
23. Patterns that scale1.00
24. What keeps you in0.96
25. code1.00
26. data1.00
27. production1.00
28. Each pillar enables the next. Skip one and the whole thing breaks in production.1.00
29. AlEngin0.93
30. EUROPE1.00
31. Engineering the future of Al0.99
32. AlEngineer0.98
33. 20260.99
5:18 #14 done18 line(s)

shot 14·sharpness 451.3
1. THE FR0.94
2. The Five Pillars of Pr0.98
3. AlEngineer0.99
4. 011.00
5. 021.00
6. 031.00
7. EUROPE1.00
8. Evaluation1.00
9. Observability1.00
10. Data Foundatic0.97
11. Define success before1.00
12. See everything, always1.00
13. Question + Trackir0.99
14. code1.00
15. data1.00
16. Each pillar enables the next.0.98
17. AlEngineer0.99
18. EUROPE1.00
5:47 #15 done17 line(s)

shot 15·sharpness 457.0
1. THE FR0.92
2. The Five Pillars of Pr0.99
3. AlEngineer0.98
4. 011.00
5. 021.00
6. 031.00
7. EUROPE1.00
8. Evaluation1.00
9. Observability1.00
10. Data Foundatic0.98
11. Define success before1.00
12. See everything, always1.00
13. Question + Trackir0.99
14. data1.00
15. Each pillar enables the next. Skip one and the0.97
16. AlEngineer0.99
17. EUROPE1.00
6:08 #16 skipped

shot 16·duplicate of #13
6:33 #17 skipped

shot 17·duplicate of #13
7:14 #18 done34 line(s)

shot 18·sharpness 2681.5
1. THE FRAMEWORK1.00
2. The Five Pillars of Production AI0.99
3. 011.00
4. 020.99
5. 031.00
6. 041.00
7. 051.00
8. ★1.00
9. AIE1.00
10. ★1.00
11. Evaluation1.00
12. Observability1.00
13. Data Foundation1.00
14. Orchestration1.00
15. Governance1.00
16. ★1.00
17. ★1.00
18. ★1.00
19. ★1.00
20. Define success before0.98
21. See everything, always0.99
22. Question + Tracking1.00
23. Patterns that scale1.00
24. What keeps you in1.00
25. code1.00
26. data1.00
27. production1.00
28. Each pillar enables the next. Skip one and the whole thing breaks in production.0.99
29. AIEngin0.94
30. EUROPE1.00
31. AlEngineer0.97
32. AlEngineer1.00
33. 20261.00
34. EUROPE1.00
7:41 #19 done32 line(s)

shot 19·sharpness 2998.3
1. PILLAR 010.96
2. Define success with numbers1.00
3. 11.00
4. Not 'accurate' — but 87% on fee disputes, <2% false positives, 60% deflection0.99
5. Evaluation1.00
6. rate. Numbers make model selection obvious.0.99
7. First1.00
8. AIE1.00
9. ★1.00
10. ★0.97
11. Build test cases from real data1.00
12. 21.00
13. ★1.00
14. ★1.00
15. ★1.00
16. ★1.00
17. Your eval suite is0.99
18. Pull 200+ real conversations from production logs. Anonymise them. Every case1.00
19. needs pass/fail criteria — not 'looks good'.0.99
20. your specification.1.00
21. Without it, you're1.00
22. guessing.1.00
23. Wire automated grading1.00
24. 31.00
25. Evaluation pipeline that runs on every code change. Dashboard showing metrics.1.00
26. You measure success before you have anything to measure.1.00
27. AlEngin0.95
28. EUROPE1.00
29. AlEngineer0.97
30. AlEngineer0.96
31. 20261.00
32. EUROPE1.00
8:07 #20 done30 line(s)

shot 20·sharpness 3017.9
1. PILLAR 011.00
2. Define success with numbers1.00
3. 11.00
4. Not 'accurate' — but 87% on fee disputes, <2% false positives, 60% deflection0.99
5. Evaluation1.00
6. rate. Numbers make model selection obvious.0.99
7. First1.00
8. *★★0.58
9. AIE1.00
10. ★1.00
11. ★1.00
12. Build test cases from real data1.00
13. 21.00
14. ★1.00
15. ★1.00
16. Your eval suite is1.00
17. Pull 200+ real conversations from production logs. Anonymise them. Every case0.99
18. needs pass/fail criteria — not 'looks good'.0.98
19. your specification.1.00
20. Without it, you're1.00
21. guessing.1.00
22. Wire automated grading1.00
23. 31.00
24. Evaluation pipeline that runs on every code change. Dashboard showing metrics.1.00
25. You measure success before you have anything to measure.1.00
26. AlEngin0.95
27. EUROPE1.00
28. Engineering the future of Al1.00
29. AlEngineer1.00
30. 20261.00
8:13 #21 skipped

shot 21·duplicate of #20
8:39 #22 skipped

shot 22·duplicate of #20
9:18 #23 done18 line(s)

shot 23·sharpness 495.9
1. PILLAR O0.98
2. Three ray ers of evaluat0.99
3. AlEngineer0.99
4. Layer 1– Deterministic0.96
5. EUROPE1.00
6. validation, Response length bounds0.98
7. PIl detection (NER + regex), Output format0.98
8. Layer 2-Semantic0.96
9. Correctness & groundedness, LLM-as-a-Judge, Non-0.97
10. determinism fix: run each test 3x - flag variance0.96
11. above threshold1.00
12. EMIACC0.82
13. Layer 3-Behavioural0.98
14. Did it call the right tools, in the right order? Did it0.98
15. escalate when confidence was low? Did it stay1.00
16. within scope?1.00
17. AlEngineer0.99
18. EUROPE1.00

Transcript

403 cues· 6,084 words· 33,835 chars

chunk 0 · 0:15–0:51 · 90 words · 518 chars
0:15 All right, thank you for joining my session.
0:20 Thank you, mate.
0:22 I'm Sandy.
0:23 I'm a technical lead for data and AI at Databricks.
0:28 Prior to working in Databricks, I worked in Amazon Web Services for five years as a principal architect for data and AI.
0:35 In the past few years, I've worked extensively building and scaling data and AI platforms using distributed systems and technology.
0:43 And in the past couple of years, specifically, I've been working with customers trying to figure out what we do with this new AI technology.
chunk 1 · 0:52–1:35 · 106 words · 592 chars
0:52 When I say new AI, AI has been here for a long time, but we all started experimenting quite exponentially in the past couple of years, right?
1:03 I have learned a great deal of lessons from building demos and how to take those demos to production, working with different customers in B2B software industries and then regulated industries like financial services.
1:18 So in this session, I want to share a playbook, a framework that I put together from lessons that I have learned working in the trenches.
chunk 2 · 1:29–2:12 · 138 words · 778 chars
1:29 that you can take and apply on when you think about how to put your AI systems into production.
1:36 And I think this session is nicely placed in the afternoon because what you can do now is in this framework, you can feed the different knowledge, the knowledge that you have gathered attending these different sessions throughout the day and see where they fit in each of these elements in the framework.
1:52 So when I started two years ago, this is the pattern I noticed in every customer conversation, right?
1:58 So everyone wanted to do something with AI.
chunk 3 · 2:01–2:40 · 111 words · 659 chars
2:01 There was immense pressure from the top to do something, to build a demo.
2:06 And every conversation started with, let's choose the model, right?
2:09 And it was nobody's fault because the market was like that.
2:11 We were talking about models.
2:12 The models were new technology for us, right?
2:15 And every conversation started, shall we use GPT?
2:18 Shall we use cloud?
2:20 You know, there was huge debate within organizations.
2:23 Then you would choose a model, you'll build some features, offsets over what features to build for that application.
2:31 You would build that in a controlled environment, so predictable data sets, you know, limited scenarios, and then it looked great as a demo.
chunk 4 · 2:40–3:25 · 112 words · 641 chars
2:40 And then leadership would get happy, they would sign it off, and they'll put it into an environment, in a production environment.
2:48 Then after a few weeks, people would start asking questions that what the hell is AI doing, right?
2:56 Why is it not answering the questions the way we expected it to answer when we were doing the demos?
3:02 It would result in not only no realization in return on investment, but also loss of money and effort in building these demos that can never scale to production
chunk 5 · 3:15–3:48 · 91 words · 515 chars
3:15 Throughout these meetings, I gathered three insights that connect to everything that we are talking about when thinking about taking AI to production.
3:26 The first one is the observability gap.
3:28 When we use AI and put it into production, if we can't see what it is actually doing, if we can't trace every decision that it's making, it's no use in production.
3:39 Second is the evaluation gap.
3:41 A lot of these conversations that we were doing, we were not actually thinking about what is that one thing that we are measuring.
chunk 6 · 3:48–4:29 · 116 words · 640 chars
3:48 Yes, we talk about accuracy, we talk about latency, we talk about groundedness, but we were not defining what is that exact thing that matters to the business and how can we build a system that can continuously measure that.
4:03 Whether it's improving, whether it's not improving, what is that system that we need to build?
4:08 And that was that evaluation gap that I noticed.
4:10 And the third is the governance gap.
4:12 We were not actually thinking, what happens when AI fails in production?
4:16 Who's accountable?
4:17 Who do I go to when something happens at 3 AM in the morning?
chunk 7 · 4:22–5:01 · 99 words · 559 chars
4:22 Who needs to own the data assets that feed some AI responses?
4:27 What happens if AI
4:30 you know, talks nonsense to a customer, right?
4:35 What happens, right?
4:36 So there is no accountability, no governance around it.
4:39 And these three insights led me to build a framework on how
4:46 I think AI should be taken to production, and this has been implemented across multiple customer organizations, and I think this is something that you can pick up from here.
chunk 8 · 4:55–5:36 · 125 words · 701 chars
4:55 These are the five pillars, and these are absolutely what you need to think about even before starting a project, right?
5:01 Then you start, build them gradually, preferably in sequence, but in real life, I know that this sequence don't work, but these are the pillars that you have to know about and you have to think about when start building.
5:13 First one is evaluation.

The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks

Scene timeline

What was stored

Provenance

Frames, and what the machine read

Transcript