vidtheque.

read-only demo

Videos 31GUkCBD-Uc

Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber

index_state ready data_status ok

AI Engineer· published 2026-07-24· 0:21:38· en-US· indexed 2026-08-10 19:40

Open on YouTube

Scene timeline

  1. Shot 0, 0:00 to 0:03, 1 of 1 keyframes kept
  2. Shot 1, 0:03 to 0:05, 1 of 1 keyframes kept
  3. Shot 2, 0:05 to 0:12, 1 of 1 keyframes kept
  4. Shot 3, 0:12 to 0:25, 1 of 1 keyframes kept
  5. Shot 4, 0:25 to 0:54, 1 of 1 keyframes kept
  6. Shot 5, 0:54 to 1:29, 1 of 1 keyframes kept
  7. Shot 6, 1:29 to 1:58, 1 of 1 keyframes kept
  8. Shot 7, 1:58 to 2:31, 1 of 1 keyframes kept
  9. Shot 8, 2:31 to 3:14, 1 of 1 keyframes kept
  10. Shot 9, 3:14 to 3:41, 1 of 1 keyframes kept
  11. Shot 10, 3:41 to 4:17, 1 of 1 keyframes kept
  12. Shot 11, 4:17 to 5:04, 1 of 1 keyframes kept
  13. Shot 12, 5:04 to 5:20, 1 of 1 keyframes kept
  14. Shot 13, 5:20 to 5:51, 1 of 1 keyframes kept
  15. Shot 14, 5:51 to 6:22, 1 of 1 keyframes kept
  16. Shot 15, 6:22 to 7:10, 1 of 1 keyframes kept
  17. Shot 16, 7:10 to 7:15, 1 of 1 keyframes kept
  18. Shot 17, 7:15 to 7:42, 1 of 1 keyframes kept
  19. Shot 18, 7:42 to 8:14, 1 of 1 keyframes kept
  20. Shot 19, 8:14 to 8:46, 0 of 1 keyframes kept
  21. Shot 20, 8:46 to 8:47, 1 of 1 keyframes kept
  22. Shot 21, 8:47 to 8:50, 0 of 1 keyframes kept
  23. Shot 22, 8:50 to 9:16, 0 of 1 keyframes kept
  24. Shot 23, 9:16 to 9:43, 0 of 1 keyframes kept
  25. Shot 24, 9:43 to 10:08, 1 of 1 keyframes kept
  26. Shot 25, 10:08 to 10:33, 1 of 1 keyframes kept
  27. Shot 26, 10:33 to 10:58, 0 of 1 keyframes kept
  28. Shot 27, 10:58 to 11:32, 1 of 1 keyframes kept
  29. Shot 28, 11:32 to 12:07, 0 of 1 keyframes kept
  30. Shot 29, 12:07 to 12:08, 1 of 1 keyframes kept
  31. Shot 30, 12:08 to 12:15, 0 of 1 keyframes kept
  32. Shot 31, 12:15 to 12:47, 0 of 1 keyframes kept
  33. Shot 32, 12:47 to 13:18, 0 of 1 keyframes kept
  34. Shot 33, 13:18 to 13:27, 1 of 1 keyframes kept
  35. Shot 34, 13:27 to 14:09, 1 of 1 keyframes kept
  36. Shot 35, 14:09 to 14:49, 1 of 1 keyframes kept
  37. Shot 36, 14:49 to 15:17, 1 of 1 keyframes kept
  38. Shot 37, 15:17 to 15:44, 1 of 1 keyframes kept
  39. Shot 38, 15:44 to 16:02, 1 of 1 keyframes kept
  40. Shot 39, 16:02 to 16:39, 1 of 1 keyframes kept
  41. Shot 40, 16:39 to 17:16, 0 of 1 keyframes kept
  42. Shot 41, 17:16 to 17:38, 1 of 1 keyframes kept
  43. Shot 42, 17:38 to 17:52, 1 of 1 keyframes kept
  44. Shot 43, 17:52 to 18:31, 1 of 1 keyframes kept
  45. Shot 44, 18:31 to 18:37, 0 of 1 keyframes kept
  46. Shot 45, 18:37 to 19:09, 1 of 1 keyframes kept
  47. Shot 46, 19:09 to 19:44, 1 of 1 keyframes kept
  48. Shot 47, 19:44 to 20:25, 1 of 1 keyframes kept
  49. Shot 48, 20:25 to 20:50, 1 of 1 keyframes kept
  50. Shot 49, 20:50 to 21:15, 0 of 1 keyframes kept
  51. Shot 50, 21:15 to 21:21, 1 of 1 keyframes kept
  52. Shot 51, 21:21 to 21:38, 0 of 1 keyframes kept

52 shot(s).

keyframes kept every frame deduplicated

What was stored

cues
251
whisperx 251
chunks
39
from 251 cues
keyframes
39
kept of 52 captured
frames with text
39
785 lines read
chapters
0
from the source metadata
keyframe bytes
4.9 MB
word timings on 251 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage state model started took
fetch done — 2026-08-09 22:45 0s
stt done — 2026-08-09 07:03 29s
chunk done — 2026-08-09 07:03 0s
text_embed done — 2026-08-10 19:40 0s
keyframe done — 2026-08-09 07:03 2m 42s
ocr done — 2026-08-09 07:06 15s
frame_embed done — 2026-08-10 19:40 7s

Frames, and what the machine read

  • 0:02 #0 done2 line(s)

    shot 0·sharpness 454.1

    1. AlEngineer0.95
    2. World's Fair0.97

  • 0:03 #1 done2 line(s)

    shot 1·sharpness 663.0

    1. AlEngineer0.95
    2. World's Fair0.99

  • 0:10 #2 done24 line(s)

    shot 2·sharpness 2732.6

    1. LAB & PLATINUM SPONSORS0.99
    2. Amazon AGI Lab0.98
    3. ANTHROP\C1.00
    4. Google DeepMind1.00
    5. MINIMAX0.95
    6. OpenAI0.92
    7. Akamai1.00
    8. arize1.00
    9. aws1.00
    10. Braintrust bright data0.98
    11. B1.00
    12. Browserbase1.00
    13. docker1.00
    14. :neo4j0.93
    15. ORACLE1.00
    16. PayPal1.00
    17. qodo1.00
    18. reducto1.00
    19. Sonar1.00
    20. Makers of0.99
    21. togetherai1.00
    22. Unblocked1.00
    23. WorkOS1.00
    24. SonarQube1.00

  • 0:16 #3 done2 line(s)

    shot 3·sharpness 217.8

    1. AlEngineer1.00
    2. World's Fair1.00

  • 0:42 #4 done13 line(s)

    shot 4·sharpness 2139.9

    1. AlEngineer0.98
    2. World's Fair0.96
    3. AIE WF 260.96
    4. Building Closed-Loop Evals1.00
    5. PRESENTED BY1.00
    6. Microsoft1.00
    7. for a Multimodal Agent at1.00
    8. Uber Scale0.98
    9. Soumya Gupta, Sr. Software Engineer1.00
    10. Jai Chopra, Sr. Product Manager1.00
    11. Uber1.00
    12. World's Fair0.97
    13. Engineering the future of Al0.99

  • 1:08 #5 done17 line(s)

    shot 5·sharpness 1624.5

    1. AlEngineer0.99
    2. World'sFair1.00
    3. Introduction1.00
    4. Scale of Uber Eats today0.99
    5. PRESENTED BY1.00
    6. ~$90b1.00
    7. ~millions1.00
    8. Microsoft1.00
    9. Annualized run rate1.00
    10. New items added / month0.99
    11. ~20%1.00
    12. ~10,000+1.00
    13. YoY Growth0.98
    14. Cities globally1.00
    15. World's Fair0.96
    16. TRACK 5· JULY 1, 20260.96
    17. Evals1.00

  • 1:52 #6 done21 line(s)

    shot 6·sharpness 1753.5

    1. AlEngineer0.99
    2. World's Fair0.97
    3. Introduction1.00
    4. Visual content plays0.98
    5. 9:190.89
    6. Uber Mission Bay 30.97
    7. Uber Mission Blay 30.95
    8. an important role for1.00
    9. Rdm0.59
    10. Rdes0.68
    11. Grooery0.88
    12. Most popular local restaurants0.98
    13. Discover a new favourite dish1.00
    14. the user experience1.00
    15. Thai for you0.99
    16. Popular in your area1.00
    17. Soups and salads1.00
    18. Explore retail0.96
    19. World's Fai0.98
    20. TRACK 5• JULY 1, 20260.97
    21. Evals1.00

  • 2:27 #7 done28 line(s)

    shot 7·sharpness 2110.5

    1. AlEngineer0.99
    2. World's Fair0.96
    3. Introduction1.00
    4. Our smaller, independent Merchants don't have1.00
    5. photos that reflect the quality of their food0.99
    6. Lack of Time1.00
    7. Lack of1.00
    8. Costs ($)0.98
    9. Know-How1.00
    10. · Photos aren't top-of-mind0.98
    11. • No equipment or expertise for0.97
    12. • Studio shoots are costly and0.97
    13. during busy service1.00
    14. pro photography1.00
    15. need space0.99
    16. • Prep required to0.95
    17. · Upload tools are hard to find &0.98
    18. Professional shoots don't scale0.98
    19. plate/compose the each dish1.00
    20. discover1.00
    21. for a global Marketplace1.00
    22. item1.00
    23. · Manual editing required0.97
    24. Dish items are0.96
    25. updated/removed1.00
    26. World's Fai0.97
    27. TRACK 5· JULY 1, 20260.95
    28. Evals1.00

  • 3:05 #8 done20 line(s)

    shot 8·sharpness 2124.4

    1. AlEngineer0.98
    2. World's Fair0.97
    3. Introduction1.00
    4. This problem is uniquely challenging at scale1.00
    5. Eaters prefer authentic, real-looking photos; a meaningful fraction actively distrust1.00
    6. anything that looks Al-generated.0.99
    7. Stay faithful1.00
    8. Preserve brand1.00
    9. Avoid sameness1.00
    10. No added or removed0.99
    11. Each merchant's1.00
    12. Enhanced images can't all1.00
    13. ingredients, no changed1.00
    14. packaging, plating and1.00
    15. collapse to the same look0.99
    16. portions1.00
    17. identity is consistent0.99
    18. World's Fai0.99
    19. TRACK 5· JULY 1,20260.96
    20. Evals1.00

  • 3:38 #9 done13 line(s)

    shot 9·sharpness 1992.6

    1. AlEngineer0.99
    2. World'sFair1.00
    3. Introduction1.00
    4. This problem is uniquely challenging at scale0.99
    5. Lack of sharpness1.00
    6. Poor composition1.00
    7. Not centered1.00
    8. Distracting background1.00
    9. Dark / poor colors0.96
    10. User-generated content1.00
    11. World'sFai0.94
    12. TRACK 5· JULY 1, 20260.94
    13. Evals1.00

  • 3:45 #10 done20 line(s)

    shot 10·sharpness 1735.4

    1. AlEngineer0.99
    2. World's Fair0.98
    3. Introduction0.98
    4. Our Goals1.00
    5. Preserve authenticity & trust1.00
    6. Remain faithful to the original image0.99
    7. Improve quality when needed1.00
    8. Edit images selectively0.98
    9. Optimize globally for the Marketplace0.99
    10. Raise within-merchant coherence without1.00
    11. collapsing cross-merchant diversity0.99
    12. Ship safely0.98
    13. Don't violate policy issues0.99
    14. Learn continuously1.00
    15. Self-learning from feedback loop(s)0.99
    16. Operate at scale1.00
    17. Reliably and cost-efficiently0.99
    18. World's Fai0.95
    19. TRACK 5· JULY 1, 20260.94
    20. Evals1.00

  • 4:49 #11 done12 line(s)

    shot 11·sharpness 1562.9

    1. AlEngineer0.98
    2. World'sFair1.00
    3. Introduction1.00
    4. Agents are well suited to solve this problem, as they0.99
    5. allow us to generalize1.00
    6. ← more control, less flexibility, brittle0.96
    7. more agency, more creativity →0.99
    8. High-agency, generalizes across the1.00
    9. Marketplace, safely0.99
    10. World'sFai1.00
    11. TRACK 5· JULY 1, 20260.95
    12. Evals1.00

  • 5:15 #12 done8 line(s)

    shot 12·sharpness 783.6

    1. AlEngineer0.99
    2. World'sFair1.00
    3. 010.98
    4. Part11.00
    5. Agent Design & Evals1.00
    6. World's Fai0.97
    7. TRACK 5· JULY 1, 20260.96
    8. Evals1.00

  • 5:47 #13 done23 line(s)

    shot 13·sharpness 1434.2

    1. AlEngineer0.99
    2. World's Fair0.96
    3. Agent Design & Evals0.99
    4. Agent Orchestration1.00
    5. Retry1.00
    6. Retrieval1.00
    7. PRESENTED BY1.00
    8. Input1.00
    9. Understanding1.00
    10. Image Quality1.00
    11. Routing1.00
    12. Prompting1.00
    13. Generation1.00
    14. LLM QA0.95
    15. Enhance1.00
    16. Microsoft1.00
    17. Skip1.00
    18. Keep Original1.00
    19. Final Scoring/1.00
    20. QA1.00
    21. World's Fai0.95
    22. TRACK 5· JULY 1, 20260.96
    23. Evals1.00

  • 6:18 #14 done31 line(s)

    shot 14·sharpness 1789.2

    1. AlEngineer0.98
    2. World's Fair0.96
    3. Agent Design & Evals0.98
    4. Agent Orchestration0.98
    5. Retry1.00
    6. Retrieval1.00
    7. PRESENTED BY1.00
    8. Input1.00
    9. Understanding1.00
    10. Image Quality1.00
    11. Routing1.00
    12. Prompting1.00
    13. Generation1.00
    14. LLM QA0.96
    15. Enhance1.00
    16. Microsoft1.00
    17. Skip1.00
    18. Pass1.00
    19. Post-Processi1.00
    20. Keep Original1.00
    21. ng0.81
    22. Final Scoring/1.00
    23. QA1.00
    24. Do Not Publish1.00
    25. Publish Ready1.00
    26. QA1.00
    27. Logging/Traces1.00
    28. Menu Output1.00
    29. World's Fai0.97
    30. TRACK 5 · JULY 1, 20260.92
    31. Evals1.00

  • 6:50 #15 done64 line(s)

    shot 15·sharpness 1251.1

    1. AlEngineer0.99
    2. World's Fair0.99
    3. Agent Design & Evals0.98
    4. End to end logging is important to setup first0.99
    5. "context"1.00
    6. "enhancement_name":0.97
    7. "timestamp"1.00
    8. "dish_description": "Street Tacos"0.99
    9. "image_url"0.98
    10. "store"1.00
    11. "2026-04-05T06:04:57Z"0.97
    12. "enhancement_pipeline_v1"0.99
    13. combo, fries & sauces unchanged.0.99
    14. "generated_prompt"0.99
    15. "1lm_model"0.88
    16. "template"1.00
    17. prompt1.00
    18. "scoring_based_improvement_v1"1.00
    19. "Replace background with dark stained wood; keep1.00
    20. "inage_quality"0.95
    21. "overall_score"1.00
    22. . image understanding0.98
    23. "lighting_quality"1.00
    24. "item_proportion"0.99
    25. "score"1.00
    26. more criteria"1.00
    27. "conf"0.99
    28. "..."0.59
    29. "***"0.55
    30. "processing_time_ms0.96
    31. generation + post-processing ..."0.95
    32. "model_name"1.00
    33. "compression"0.99
    34. "color_enhancement"0.99
    35. "upload": "success"0.99
    36. "format": "JPEG", "size_reduction_percent"0.98
    37. ["hdr", "sharpness"]0.98
    38. "dish_accuracy"0.99
    39. "reasoning": "grilled chicken, cilantro, onion - matches iten name"0.98
    40. "score"1.00
    41. "conf": -"0.82
    42. .. LLM QA gate ...0.77
    43. "final_assessment"1.00
    44. "iteration_count"0.99
    45. "passed",0.98
    46. "enhancement_decision": "genai_editing".0.97
    47. "directives"1.00
    48. routing - directives ...": (0.94
    49. "iten_proportion"0.87
    50. "lighting_quality":0.99
    51. "flat, slightly dim; muted colors"0.99
    52. "food fills 38-48% of frame, too small"),0.89
    53. "checks"1.00
    54. "spillage_on_container"1.00
    55. "distracting_background"0.99
    56. "unrealistic_food_or_plating":1.00
    57. all passed ...0.95
    58. "passed"1.00
    59. "passed"1.00
    60. "passed"0.96
    61. "background_quality": "typical surface, not styled"1.00
    62. World's Fai0.98
    63. TRACK 5· JULY 1, 20260.95
    64. Evals1.00

  • 7:11 #16 done29 line(s)

    shot 16·sharpness 713.4

    1. AlEngineer0.99
    2. World's Fair0.96
    3. Agent Design & Evals0.98
    4. Retry1.00
    5. Retrieval1.00
    6. Input1.00
    7. Understanding1.00
    8. Image Quality1.00
    9. Routing1.00
    10. Prompting1.00
    11. Generation1.00
    12. LLM QA0.94
    13. Enhance1.00
    14. Skip1.00
    15. Fail1.00
    16. Pass1.00
    17. Post-Processi1.00
    18. Keep Original0.98
    19. ng1.00
    20. Final Scoring/1.00
    21. QA1.00
    22. Do Not Publish0.99
    23. Publish Ready1.00
    24. QA1.00
    25. Logging/Traces1.00
    26. Menu Output1.00
    27. World's Fai0.98
    28. TRACK 5· JULY 1, 20260.95
    29. Evals1.00

  • 7:21 #17 done25 line(s)

    shot 17·sharpness 905.3

    1. AlEngineer0.99
    2. World'sFair1.00
    3. Agent Design & Evals0.98
    4. Routing1.00
    5. 1·REFER TO IMAGE, TEXT INPUT0.96
    6. 2·GRADE AGAINST RUBRIC0.97
    7. 3·DECIDE0.98
    8. Technical1.00
    9. Below bar1.00
    10. Composition1.00
    11. Pass1.00
    12. Content1.00
    13. Pass1.00
    14. →0.97
    15. Decide whether to enhance or1.00
    16. skip1.00
    17. Policy1.00
    18. Pass1.00
    19. Dish: "Street Tacos - grilled0.99
    20. Dish match1.00
    21. Below bar1.00
    22. chicken, cilantro, onion"0.99
    23. World's Fai0.97
    24. TRACK 5· JULY 1, 20260.96
    25. Evals1.00

  • 7:58 #18 done22 line(s)

    shot 18·sharpness 1030.5

    1. AlEngineer0.99
    2. World's Fair0.99
    3. Agent Design & Evals0.96
    4. Routing Evals1.00
    5. Model: ENHANCE0.99
    6. Model: SKIP1.00
    7. Correct enhance0.98
    8. TP1.00
    9. False skip0.99
    10. FN1.00
    11. Bad photo, we fixed it0.99
    12. Silent regression0.98
    13. False enhance1.00
    14. FP1.00
    15. Correct skip0.99
    16. TN1.00
    17. wasted spend on an OK photo0.96
    18. already good, left alone0.98
    19. We measure it like a classifier, i.e.precision & recall against human labels0.99
    20. World's Fai0.97
    21. TRACK 5· JULY 1,20260.96
    22. Evals1.00

  • 8:30 #19 skipped

    shot 19·duplicate of #18

  • 8:47 #20 done25 line(s)

    shot 20·sharpness 1088.7

    1. AlEngineer0.99
    2. World's Fair0.99
    3. Agent Design & Evals0.98
    4. Routing: Offline Human Alignment0.98
    5. Human Labelers0.97
    6. Ground Truth (Labelled Dataset)0.99
    7. rubric-anchored grading1.00
    8. 由0.95
    9. 巫0.69
    10. Routing1.00
    11. Evaluate1.00
    12. Tune1.00
    13. Ship0.98
    14. scores image → enhance /0.98
    15. agent decision vs. golden-set0.99
    16. raise the bar, bias toward recall0.98
    17. deploy once precision & recall0.99
    18. keep1.00
    19. truth → precision & recall0.95
    20. (catch bad photos)1.00
    21. clear the bar0.98
    22. re-sample unseen golden-set images0.96
    23. World's Fai0.95
    24. TRACK 5· JULY 1, 20260.95
    25. Evals1.00

  • 8:49 #21 skipped

    shot 21·duplicate of #18

  • 9:06 #22 skipped

    shot 22·duplicate of #20

  • 9:22 #23 skipped

    shot 23·duplicate of #20

Transcript

251 cues· 3,605 words· 19,541 chars

  1. 0:17 My name is Jay, and I'm here with Somya.
  2. 0:19 We are part of the computer vision team at Uber.
  3. 0:23 And we're going to talk to you about a real world production use.
  4. 0:27 Oh, mic's not on?
  5. 0:28 OK. OK. Don't worry, I'll manage.
  6. 0:34 Can you hear me now?
  7. 0:36 OK, so we're going to talk to you today about a real world production use case.
  8. 0:41 And specifically, we're going to dive into how we design the evals and the eval loops.
  9. 0:48 So let's see this.
  10. 0:54 All right, cool.
  11. 0:55 So just before we get into the agent design, we're going to talk a little bit about the use case.
  12. 1:01 So our delivery marketplace, Uber Eats, we do about 90 billion run rate per year at the moment.
  13. 1:09 We're adding millions of items to the marketplace each and every year.
  14. 1:14 Sorry, every month.
  15. 1:16 We're growing at 20% year on year, and we operate in 10,000 cities globally.
  16. 1:21 So not many people actually know this, but our delivery marketplace is just as big as the mobility side on Uber today.
  17. 1:30 Visual content actually plays a really important role for the user experience.
  18. 1:35 So a photo is quite often the first signal that a customer gets that gives them that initial impression about a merchant.
  19. 1:44 So a good photo can make the difference between someone scrolling through the feed and actually clicking on an item and adding to the cart.
  20. 1:52 And more and more, we're seeing different modalities on Uber Eats, especially video content.
  21. 1:59 But this is a problem.
  22. 2:01 So our smaller independent merchants simply just don't have the level of quality for their photos that reflect what the eater is actually going to get.
  23. 2:11 And when we speak to our merchants, there are three themes that kind of emerge.
  24. 2:16 Lack of time.
  25. 2:17 lack of know-how, and costs, because these professional photo shoots actually cost a lot of money.
  26. 2:24 And this can be especially problematic if the merchant is updating their menu over time.
  27. 2:32 So this problem is actually pretty challenging to solve for at scale.
  28. 2:37 Because our consumers, they want authentic, real-looking photos.
  29. 2:41 But a meaningful fraction of consumers actually distrust anything that is AI-generated.
  30. 2:48 So if you open up the Uber Eats app, the last thing that you want is to be scrolling through food photography that looks like AI slot.
  31. 2:56 So threading the needle here, we need to be able to stay faithful to the original image, preserve the brand of the merchant, and avoid everything looking the same.
  32. 3:06 We have the same prompt for every photo that we're editing.
  33. 3:09 The diversity of the marketplace is going to collapse.
  34. 3:15 We also, because we operate globally, we also have this long tail distribution of different quality that we see across the marketplace.
  35. 3:24 So we've got some examples here.
  36. 3:26 You might see food photography that is poor sharpness, poor composition, not centered, or poor colors as well.
  37. 3:35 We also have a wide range of spectrum of user generated content on the platform as well.
  38. 3:42 So what are our goals when we're designing these agents?
  39. 3:44 When you think through these goals, you might actually be thinking through your own agents that you're building yourself.
  40. 3:50 But for us, it's about one, preserving authenticity and trust.
  41. 3:54 Two, improving the quality when we need to.
  42. 3:56 So we want to be able to improve quality selectively.
  43. 4:00 We want to optimize globally for the entire marketplace.
  44. 4:03 We don't want to cannibalize certain merchants.
  45. 4:07 We want to ship safely.
  46. 4:08 And this is going to be an important theme throughout the talk.
  47. 4:11 We want to learn continuously, and we want to operate at scale in a cost-efficient manner.
  48. 4:18 So agents are actually really well-suited to solve this problem.
  49. 4:23 So if you imagine a spectrum, on the one side, you've got something that's more deterministic.
  50. 4:28 It's more rules-based.

Open at this second