vidtheque.

read-only demo

Videos maTp79FD9gI

Agent Output Is Not UX: Rendering Layer Your LLM Pipeline Is Missing - Bala Ramdoss, Amazon Lens

index_state ready data_status ok

AI Engineer· published 2026-07-20· 0:14:12· en-US· indexed 2026-08-10 19:48

Open on YouTube

Scene timeline

  1. Shot 0, 0:00 to 0:38, 1 of 1 keyframes kept
  2. Shot 1, 0:38 to 0:59, 1 of 1 keyframes kept
  3. Shot 2, 0:59 to 1:08, 1 of 1 keyframes kept
  4. Shot 3, 1:08 to 1:21, 1 of 1 keyframes kept
  5. Shot 4, 1:21 to 1:54, 1 of 1 keyframes kept
  6. Shot 5, 1:54 to 2:11, 1 of 1 keyframes kept
  7. Shot 6, 2:11 to 2:54, 1 of 1 keyframes kept
  8. Shot 7, 2:54 to 3:22, 1 of 1 keyframes kept
  9. Shot 8, 3:22 to 3:50, 0 of 1 keyframes kept
  10. Shot 9, 3:50 to 4:28, 1 of 1 keyframes kept
  11. Shot 10, 4:28 to 5:05, 0 of 1 keyframes kept
  12. Shot 11, 5:05 to 5:52, 1 of 1 keyframes kept
  13. Shot 12, 5:52 to 6:19, 1 of 1 keyframes kept
  14. Shot 13, 6:19 to 6:47, 0 of 1 keyframes kept
  15. Shot 14, 6:47 to 7:27, 1 of 1 keyframes kept
  16. Shot 15, 7:27 to 8:04, 1 of 1 keyframes kept
  17. Shot 16, 8:04 to 8:41, 0 of 1 keyframes kept
  18. Shot 17, 8:41 to 8:51, 1 of 1 keyframes kept
  19. Shot 18, 8:51 to 9:30, 1 of 1 keyframes kept
  20. Shot 19, 9:30 to 9:48, 1 of 1 keyframes kept
  21. Shot 20, 9:48 to 10:12, 1 of 1 keyframes kept
  22. Shot 21, 10:12 to 10:35, 1 of 1 keyframes kept
  23. Shot 22, 10:35 to 10:59, 1 of 1 keyframes kept
  24. Shot 23, 10:59 to 11:16, 1 of 1 keyframes kept
  25. Shot 24, 11:16 to 11:38, 1 of 1 keyframes kept
  26. Shot 25, 11:38 to 12:07, 1 of 1 keyframes kept
  27. Shot 26, 12:07 to 12:39, 1 of 1 keyframes kept
  28. Shot 27, 12:39 to 13:12, 0 of 1 keyframes kept
  29. Shot 28, 13:12 to 13:55, 1 of 1 keyframes kept
  30. Shot 29, 13:55 to 14:12, 0 of 1 keyframes kept

30 shot(s).

keyframes kept every frame deduplicated

What was stored

cues
171
whisperx 171
chunks
26
from 171 cues
keyframes
24
kept of 30 captured
frames with text
24
301 lines read
chapters
0
from the source metadata
keyframe bytes
2.2 MB
word timings on 171 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage state model started took
fetch done — 2026-08-10 04:45 1m 57s
stt done — 2026-08-10 04:47 13s
chunk done — 2026-08-10 04:47 0s
text_embed done — 2026-08-10 19:48 0s
keyframe done — 2026-08-10 04:47 39s
ocr done — 2026-08-10 04:47 7s
frame_embed done — 2026-08-10 19:48 4s

Frames, and what the machine read

  • 0:33 #0 done26 line(s)

    shot 0·sharpness 3245.7

    1. help me make a reservation at zuni cafe0.99
    2. I can help with this. A couple of things first:1.00
    3. What date and time are you thinking, and how many people?1.00
    4. Also, just so you know how this will actually work: I can't book the0.99
    5. reservation myself since that requires logging into a system on your1.00
    6. behalf and confirming details with the restaurant. What I can do is look1.00
    7. up Zuni Cafe's actual reservation options (phone number,0.98
    8. OpenTable/Resy link, hours) so you have exactly what you need to book0.99
    9. it in one step.0.99
    10. Searched the web>0.96
    11. Worth noting, since it'll shape which option makes sense for you: I found0.99
    12. a couple of conflicting reservation systems in the results, so here's0.99
    13. what's actually true rather than just one link.1.00
    14. Reservation options for Zuni Cafe (1658 Market St, SF):0.99
    15. • Phone: 415-552-2522, Tuesday through Thursday 2:00-9:30pm and0.99
    16. Friday through Sunday 10:30am-9:30pm Zunicafe0.99
    17. • Online: OpenTable shows current availability, and Tock also lists0.98
    18. the restaurant, so it's worth checking both since restaurants0.99
    19. sometimes split inventory between platforms OpenTable0.99
    20. Reservations are taken up to 60 days in advance, and groups of 5 to0.99
    21. 12 are asked to call directly rather than book online OpenTable0.99
    22. If you can't get a slot, the zinc bar at the front is walk-in only, no0.99
    23. reservation needed, and it's a p↓ ar spot to eat oysters and have0.99
    24. wine while waiting for a table Reacker0.99
    25. Source: Claude0.98
    26. Write a message...0.98

  • 0:40 #1 done15 line(s)

    shot 1·sharpness 205.4

    1. help me make a reservation at zuni cafe1.00
    2. Zuni Cafe1.00
    3. 1658 Market St · San Francisco0.98
    4. RESERVE A TABLE0.99
    5. Fri, Jun 260.99
    6. Party of 21.00
    7. Available tonight0.97
    8. 6:000.99
    9. 6:301.00
    10. 7:151.00
    11. 8:000.95
    12. Book table for 7:150.98
    13. Write a message...0.97
    14. +0.98
    15. Fictional, not real Claude1.00

  • 1:00 #2 done4 line(s)

    shot 2·sharpness 1615.9

    1. Agent Output Is Not UX0.97
    2. Building the generative-UI rendering layer your LLM pipeline is0.99
    3. missing0.99
    4. Bala Ramdoss |Al Engineer World's Fair0.99

  • 1:16 #3 done7 line(s)

    shot 3·sharpness 1827.7

    1. Seeit.Shopit.1.00
    2. Take a photo of anything you see, and instantly find0.99
    3. Bala Ramdoss1.00
    4. the same or similar items in the Amazon shopping app.0.99
    5. Sr SDE @ Amazon Search0.99
    6. Amazon Lens0.96
    7. lens ai0.92

  • 1:28 #4 done6 line(s)

    shot 4·sharpness 1242.8

    1. Seeit.Shopit.1.00
    2. Take a photo of anything you see, and instantly find0.99
    3. the same or similar items in the Amazon shopping app.0.99
    4. Amazon Lens0.98
    5. lens ai0.93
    6. Take a photo to search products1.00

  • 1:57 #5 done99 line(s)

    shot 5·sharpness 2535.1

    1. 9:411.00
    2. 9:411.00
    3. 9:411.00
    4. 9:411.00
    5. ChatGPT51.00
    6. e0.54
    7. ChatGPT 5>0.92
    8. ChatGPT 50.95
    9. @0.70
    10. ChatGPT 50.94
    11. @0.62
    12. …0.77
    13. can you send me a list of pizza0.99
    14. Show me more about Brick &0.99
    15. What's good there?0.98
    16. What's good around here1.00
    17. restaurants in San Francisco?1.00
    18. Basil please0.97
    19. Pizzazz1.00
    20. Pizzazz1.00
    21. Pizzazz1.00
    22. Pizzazz1.00
    23. UITTLE NONA'S0.93
    24. Pepperoni1.00
    25. $21.501.00
    26. tay Street0.95
    27. Mason Street0.98
    28. 20.53
    29. Dot1.00
    30. BRICK&BASIL0.96
    31. Cupped pepperoni that curls and pools just the1.00
    32. of oregano and a drizzle of extra-virgin olive oil.0.99
    33. right amount of spicy oil. Finished with a sprinkle0.99
    34. PI20.90
    35. Mushroom & Taleggio1.00
    36. $20.001.00
    37. treet0.99
    38. Fil: Street0.90
    39. ORTH BE0.97
    40. This one's quieter and more layered - earthy0.98
    41. Union Street0.98
    42. that melts into the sauce, and a finish of thyme0.99
    43. mushrooms roasted until golden, creamy taleggio0.99
    44. Green Street0.99
    45. Val1.00
    46. 1427 Via Campania0.99
    47. A tiny, brick-walled trattoria tucked down0.99
    48. a side street near Washington Square P...0.98
    49. Here are a few standout pizzerias to keep on1.00
    50. Little Nona's1.00
    51. your radar before you decide. Each one brings1.00
    52. 6:00pm1.00
    53. d tn thn t~hln0.67
    54. 6:30pm1.00
    55. 9.21.00
    56. nn thn hant0.81
    57. A music-theme0.97
    58. converted reco1.00
    59. Dough-Re-M1.00
    60. 512 Harmony A1.00
    61. 6:00pm1.00
    62. If you're craving something comforting and1.00
    63. classic, Brick & Basil is a great pick. It's the1.00
    64. kind of place that balances warmth and0.98
    65. tradition — handmade nastas elow-simmered0.96
    66. Opened in 2009 in San Francisco's North Beach0.98
    67. neighborhood by world-champion pizzaiolo Tony...1.00
    68. Brick & Basil0.97
    69. 1432 Shattuck Ave, Berkeley, CA1.00
    70. 9.21.00
    71. nersonalitv: the neoneroni leans classic and0.98
    72. Each of these pies offers a distinct0.99
    73. and olive oil. It's a more "thinky" pie.0.96
    74. If you want something you haven't had a hundred0.98
    75. pie with Calabrian chiles and lemon zest. I's0.96
    76. times, go for the punchy, umami-packed anchovy1.00
    77. bright, salty, and surprising.0.99
    78. Anchovy & Calabrian Chili1.00
    79. $21.501.00
    80. Here's a quick look at some of the best spots0.99
    81. in the area — from beloved neighborhood0.98
    82. kitchens. You could easily make a night of it0.99
    83. by picking one here, or even plan a little pizza0.98
    84. roadway1.00
    85. institutions to a few newer, more creative0.99
    86. crawl and sample a few styles back-to-back.1.00
    87. Vallejo Street0.98
    88. Darific Avenue0.94
    89. Broadway1.00
    90. +0.83
    91. Ask anything0.98
    92. +0.68
    93. Ask anything1.00
    94. Q0.57
    95. +0.75
    96. Ask anything1.00
    97. +0.82
    98. Ask anything1.00
    99. Image source: OpenAl blog0.96

  • 2:49 #6 done4 line(s)

    shot 6·sharpness 1287.5

    1. The wall0.98
    2. • Latency feels broken0.98
    3. • Not sure how to render LLM text output to the desired CX0.99
    4. • Frontend platform fragmentation0.99

  • 2:58 #7 done4 line(s)

    shot 7·sharpness 1020.5

    1. The layer has a name0.99
    2. generative UI0.97
    3. A2UI (Google): defines the components (the payload)0.99
    4. A2UI is an open standard, declarative and built for streaming.0.99

  • 3:44 #8 skipped

    shot 8·duplicate of #7

  • 3:55 #9 done5 line(s)

    shot 9·sharpness 2186.6

    1. From data APls to Ul-shaping APls0.97
    2. • Controlled: model picks a prebuilt component (eg: product_card)0.98
    3. Declarative: model composes from a catalog (A2UI)0.99
    4. • Open-ended: model generates novel UI (MCP Apps)0.97
    5. Spectrum: https://www.copilotkit.ai/generative-ui-spectrum0.99

  • 4:35 #10 skipped

    shot 10·duplicate of #9

  • 5:10 #11 done13 line(s)

    shot 11·sharpness 746.6

    1. Mobile adds complexity1.00
    2. Web1.00
    3. Deploy fix0.97
    4. Live in minutes1.00
    5. 100% on fixed version0.99
    6. Mobile0.93
    7. Build, review1.00
    8. Staged rollout0.97
    9. 5% to 100%, over days to weeks0.99
    10. unaffected, still on the old version0.99
    11. Long tail: weeks+0.99
    12. You control the timeline0.97
    13. Users control the timeline1.00

  • 5:55 #12 done11 line(s)

    shot 12·sharpness 693.5

    1. The rendering layer, mapped1.00
    2. Context Phase1.00
    3. Version aware surfacing1.00
    4. Model output0.96
    5. Streamed1.00
    6. BFF1.00
    7. Streamed1.00
    8. Client1.00
    9. Typed Ul intent0.97
    10. format * context hydration actions0.96
    11. rendering• fallback0.96

  • 6:30 #13 skipped

    shot 13·duplicate of #12

  • 7:23 #14 done4 line(s)

    shot 14·sharpness 2102.2

    1. The rendering contract1.00
    2. • Typed: Prevent client from trying to use patterns from raw token output1.00
    3. • Versioned: model knows what the client is capable of1.00
    4. Takeaway: Model capabilities includes picking the right UI1.00

  • 7:32 #15 done12 line(s)

    shot 15·sharpness 2285.2

    1. The rendering contract1.00
    2. Model streams: conversation_block + ui_block1.00
    3. Rule: 1-3 results -> flight_carousel · 4+ -> flight_list0.98
    4. "flight_carousel": {0.97
    5. "layout": "horizontal_swipeable",0.99
    6. "max_items":3,0.98
    7. "data_spec": { "airline": "string", "price": "number", "depart": "time" }0.99
    8. },0.84
    9. "flight_list": {0.97
    10. "layout": "vertical_scrollable"0.98
    11. "data_spec": { "airline": "string", "price": "number", "depart": "time" }0.99
    12. }0.99

  • 8:12 #16 skipped

    shot 16·duplicate of #15

  • 8:42 #17 done11 line(s)

    shot 17·sharpness 606.1

    1. The rendering contract1.00
    2. Context Phase1.00
    3. Version aware surfacing1.00
    4. Model output0.97
    5. Streamed1.00
    6. BFF1.00
    7. Streamed1.00
    8. Client1.00
    9. Typed Ul intent0.97
    10. format * context • hydration actions0.92
    11. rendering* fallback0.94

  • 9:21 #18 done4 line(s)

    shot 18·sharpness 1101.7

    1. Streaming into typed UI1.00
    2. Chunks over SSE*1.00
    3. request1.00
    4. response stream1.00

  • 9:34 #19 done7 line(s)

    shot 19·sharpness 377.4

    1. Streaming into typed UI0.99
    2. • Chunks over SSE*0.96
    3. JetBridge Air1.00
    4. $2141.00
    5. JetBridge Air1.00
    6. $2141.00
    7. 8:40 AM → 11:55 AM0.97

  • 9:53 #20 done5 line(s)

    shot 20·sharpness 1408.0

    1. Streaming into typed UI0.99
    2. • Chunks over SSE*0.99
    3. • Optimize time to first chunk, not total latency · TTFT / TTFC0.98
    4. • A spinner reads as broken after ~2 seconds0.97
    5. *A2UI standardizes this.1.00

  • 10:19 #21 done13 line(s)

    shot 21·sharpness 859.8

    1. Latency as a product decision1.00
    2. 9:411.00
    3. When you can't beat latency,0.99
    4. lens aí0.90
    5. 30.56
    6. design around it1.00
    7. Top matches:1.00
    8. White Ceramic Planter1.00
    9. Cylindrical Ribbed Pot1.00
    10. 4.7★★★★★3.7K0.96
    11. 1590 Lit Pic: 69990.57
    12. Porous ceramic regulates moisture indoors or out1.00
    13. Does it come with a drainage hole?0.99

  • 10:40 #22 done5 line(s)

    shot 22·sharpness 1181.3

    1. Latency as a product decision1.00
    2. Do not use thinking CX for1.00
    3. OH! YOU ARE THINKING?0.98
    4. everything1.00
    5. I SHOULD WAIT FOR A RESPONSE THEN1.00

  • 11:04 #23 done5 line(s)

    shot 23·sharpness 609.9

    1. Latency as a product decision1.00
    2. Do not use thinking CX for1.00
    3. everything1.00
    4. • Listing Custom Sunshades0.99
    5. Comparing Sunshade Features1.00

Transcript

171 cues· 1,968 words· 10,495 chars

  1. 0:01 I asked my AI assistant to help me reserve a table at a popular restaurant.
  2. 0:06 Here's what it gave me.
  3. 0:09 Now, it's not wrong.
  4. 0:12 The phone number is there.
  5. 0:13 The hours are right.
  6. 0:15 It even knows the walking oyster bar at the front.
  7. 0:19 The model did the real work.
  8. 0:21 But look at the outcome.
  9. 0:24 I have to do my research and work towards actually booking the table.
  10. 0:29 We have been here for a while.
  11. 0:30 Now, imagine if you were to build something like this for your customers.
  12. 0:35 What would you want it to do?
  13. 0:39 Same answer rendered like this instead.
  14. 0:42 A date, a time, a couple of taps, and then you're done.
  15. 0:47 The agentic product you're building already has all the tools to support your customers' needs.
  16. 0:53 The only thing that you need to focus on is the layer between the model and something that a human can interact with.
  17. 1:00 That layer is what I'm going to talk about.
  18. 1:02 The models and agents are here to stay, and we should learn how to make it friendly to the humans.
  19. 1:09 Hello, welcome to my talk.
  20. 1:11 I'm Bala Ramdas.
  21. 1:13 I have been building customer-facing apps for over a decade, and the past six years I've spent building Amazon Lens.
  22. 1:22 Amazon Lens is our suite of camera features powered by AI.
  23. 1:26 It enables you to shop using images, screenshots, and barcodes to discover visually similar products.
  24. 1:33 If you have Amazon app installed, I encourage you to try.
  25. 1:36 This talk comes from my experience building customer-facing products that are in millions and millions of mobile devices.
  26. 1:45 Quick disclaimer before I go further.
  27. 1:48 I am giving this talk on my own.
  28. 1:50 The opinions in this talk are mine and not my employers.
  29. 1:56 When you think of building UX for an agentic AI, obviously ChatGPT comes into mind.
  30. 2:02 How do you draw these cards?
  31. 2:03 How do you choose carousel versus list?
  32. 2:06 How do you craft this so the human can understand and interact better?
  33. 2:12 When you attempt to build something like this, you run into a wall.
  34. 2:16 There are problems that need to be solved from the very early stages of your system.
  35. 2:22 Is that experience going to be snappy or slow?
  36. 2:26 Do we get all the information at first or one thing at a time?
  37. 2:32 And how do you scale this for mobile apps where versions and device capabilities are fragmented?
  38. 2:38 If you notice, none of these problems are due to the model itself.
  39. 2:42 The model does its job well.
  40. 2:44 These are delivery problems, and they live in between the model output and what's on the screen.
  41. 2:50 That is the layer that decides whether your product succeeds or not.
  42. 2:56 So for this delivery problem, if you had asked me about this layer two years ago, I wouldn't have had a name for it.
  43. 3:05 I had built a couple of
  44. 3:07 AI features by then, and every single time, I solved this part from scratch.
  45. 3:13 A bespoke pattern, shaped around whatever system I was in.
  46. 3:18 There was no shared vocabulary for any of it.
  47. 3:22 What gets me is that this thing now has a name, Generative UI, and there's an open spec for it, A2UI from Google.
  48. 3:31 Instead of the agent handing you a raw text or HTML, it describes the UI as data, a list of components, and the client renders them with its own native widgets.
  49. 3:43 A problem I used to solve alone is becoming something teams get to start from, and that's exactly what I'm excited to dig into.
  50. 3:52 Normal API returns data, and the client decides how to draw it.

Open at this second