Computer-Use 2.0: Agents Just Got Multi-Cursor — Francesco Bonacci, Cua

What was stored

cues

154

whisperx 154

chunks

29

from 154 cues

keyframes

39

kept of 49 captured

frames with text

38

862 lines read

chapters

6

from the source metadata

keyframe bytes

5.3 MB

word timings on 154 cues

Provenance

Frames, and what the machine read

• 

  0:02 #0 done2 line(s)

  shot 0·sharpness 453.3

  1. AlEngineer0.96
  2. World's Fair0.97
• 

  0:03 #1 done2 line(s)

  shot 1·sharpness 662.8

  1. AlEngineer0.95
  2. World's Fair0.99
• 

  0:10 #2 done24 line(s)

  shot 2·sharpness 2748.4

  1. LAB & PLATINUM SPONSORS0.99
  2. Amazon AGI Lab0.98
  3. ANTHROP\C1.00
  4. Google DeepMind1.00
  5. MINIMAX0.91
  6. OpenAI0.92
  7. Akamai1.00
  8. arize1.00
  9. aws1.00
  10. Braintrust bright data0.98
  11. B1.00
  12. Browserbase1.00
  13. docker1.00
  14. :neo4j0.90
  15. ORACLE1.00
  16. PayPal1.00
  17. qodo1.00
  18. reducto1.00
  19. Sonar1.00
  20. Makers of1.00
  21. togetherai1.00
  22. Unblocked1.00
  23. WorkOS1.00
  24. SonarQube1.00
• 

  0:18 #3 done2 line(s)

  shot 3·sharpness 175.5

  1. AlEngineer0.99
  2. World's Fair0.99
• 

  0:35 #4 done3 line(s)

  shot 4·sharpness 168.1

  1. Cua1.00
  2. AlEngineer0.98
  3. World's Fair0.99
• 

  1:04 #5 done2 line(s)

  shot 5·sharpness 165.1

  1. AlEngineer0.96
  2. World's Fair1.00
• 

  1:33 #6 done49 line(s)

  shot 6·sharpness 1733.2

  1. AlEngineer0.99
  2. World'sFair1.00
  3. Cua1.00
  4. THE OLD LOOP0.99
  5. 1.0 = the Human Loop0.99
  6. The agent watched the screen, chose the next move, acted like a user, and repeated.1.00
  7. PRESENTED BY1.00
  8. SCREENSHOT1.00
  9. REASON1.00
  10. CLICK TARGET1.00
  11. Safari1.00
  12. Edit0.98
  13. View1.00
  14. Window1.00
  15. Help1.00
  16. "The user asked for a0.97
  17. Safari1.00
  18. Edit1.00
  19. View1.00
  20. Go1.00
  21. Window1.00
  22. Help1.00
  23. Microsoft1.00
  24. PDF. The export button0.97
  25. is visible in the lower1.00
  26. Reports1.00
  27. Quarterly report0.96
  28. Export PDF1.00
  29. right."0.99
  30. Reports1.00
  31. Quarterly report1.00
  32. Export0.89
  33. Export1.00
  34. Choose one action, then observe again.0.99
  35. 1.0 METHODS0.99
  36. screenshots1.00
  37. OCR / SoM0.99
  38. mouse1.00
  39. keyboard1.00
  40. scroll1.00
  41. type1.00
  42. wait1.00
  43. pyautogui1.00
  44. screen first - action next · foreground repeat0.97
  45. cua.ai1.00
  46. D0.62
  47. Cua0.80
  48. Engineering the future of Al0.98
  49. World's Fair0.99
• 

  1:51 #7 done47 line(s)

  shot 7·sharpness 1668.1

  1. AlEngineer0.99
  2. World'sFair1.00
  3. Cua1.00
  4. THE OLD LOOP0.96
  5. 1.0 = the Human Loop0.98
  6. The agent watched the screen, chose the next move, acted like a user, and repeated.1.00
  7. PRESENTED BY1.00
  8. SCREENSHOT1.00
  9. REASON1.00
  10. CLICK TARGET1.00
  11. Safari1.00
  12. View1.00
  13. Window1.00
  14. Help1.00
  15. "The user asked for a1.00
  16. Safari1.00
  17. Edit1.00
  18. View1.00
  19. Go1.00
  20. Window1.00
  21. Help1.00
  22. Microsoft1.00
  23. PDF. The export button0.99
  24. is visible in the lower0.97
  25. Reports1.00
  26. Quarterly report0.99
  27. Export PDF1.00
  28. right."0.99
  29. Reports1.00
  30. Quarterly report1.00
  31. Choose one action, then observe again.0.99
  32. 1.0 METHODS0.99
  33. screenshots1.00
  34. OCR / SoM0.99
  35. mouse1.00
  36. keyboard1.00
  37. scroll1.00
  38. type1.00
  39. wait1.00
  40. pyautogui1.00
  41. screen first · action next · foreground repeat0.97
  42. cua.ai0.99
  43. 00.57
  44. 000.64
  45. TRACK 7·JULY 1,20260.95
  46. ComputerUse1.00
  47. World's Fair0.97
• 

  2:33 #8 done40 line(s)

  shot 8·sharpness 1425.0

  1. AlEngineer0.98
  2. World's Fair0.97
  3. Cua1.00
  4. COMPUTER-USE 1.0 EXAMPLE0.98
  5. A GUI-first operator0.99
  6. The old loop wired the model straight to the GUl: prompt and screenshot in, action out.0.99
  7. computer_use_setup.py1.00
  8. MODEL1.00
  9. response = client.responses.create(1.00
  10. computer-use-preview1.00
  11. model="computer-use-preview",1.00
  12. The loop is native to the model.1.00
  13. tools=[{1.00
  14. "type": "computer_use_preview",0.99
  15. "display_width": 1024,0.99
  16. "display_height": 768,1.00
  17. TOOL1.00
  18. },0.52
  19. "environment": "mac",0.98
  20. provider-native tools1.00
  21. input=[{0.99
  22. "role": "user",0.98
  23. OpenAI Responses API or Anthropic beta.0.97
  24. "content": [0.99
  25. {"type": "input_text", "text": task},0.99
  26. {"type": "input_image", "image_url": screenshot_url},0.98
  27. INPUT1.00
  28. H],0.50
  29. text + screenshot1.00
  30. truncation="auto",1.00
  31. The next action depends on pixels.1.00
  32. Computer-Use model setup · prompt + screenshot loop0.98
  33. cua.ai1.00
  34. OUTPUT1.00
  35. next: computer_call1.00
  36. 40.95
  37. TRACK 7·JULY 1,20260.96
  38. World's Fair1.00
  39. AlEngineet0.89
  40. Computer Use0.98
• 

  2:38 #9 done23 line(s)

  shot 9·sharpness 1387.1

  1. AlEngineer1.00
  2. World's Fair0.96
  3. Cua1.00
  4. BACKGROUND CONTROL1.00
  5. À la SkyLight Recipe0.96
  6. Focus without raise, post to pid, prime Chromium, keep AX live0.99
  7. yabai pattern1.00
  8. 011.00
  9. Flip AppKit active without raising the window or switching0.99
  10. Spaces.1.00
  11. SLEventPostToPid1.00
  12. 820.87
  13. SkyLight sends mouse events to the target pid. outside the0.99
  14. HID stream.1.00
  15. primer click1.00
  16. 031.00
  17. An off-screen click at -1,-1 warms Chromium's activation gate.0.99
  18. live AX1.00
  19. 840.87
  20. Private AX SPI keeps occluded Electron trees fresh.0.99
  21. TRACK 7· JULY 1,20260.96
  22. Computer Use1.00
  23. World's Fair0.97
• 

  2:39 #10 skipped

  shot 10·duplicate of #7

• 

  2:46 #11 done24 line(s)

  shot 11·sharpness 1406.5

  1. AlEngineer0.99
  2. World's Fair0.99
  3. Cua1.00
  4. BACKGROUND CONTROL1.00
  5. À la SkyLight Recipe1.00
  6. Focus without raise, post to pid, prime Chromium, keep AX live1.00
  7. yabai pattern1.00
  8. 011.00
  9. 58°1.00
  10. Flip AppKit active without raising the window or switching0.99
  11. Spaces.1.00
  12. SLEventPostToPid1.00
  13. 020.86
  14. SkyLight sends mouse events to the farget pid. outside the0.99
  15. HID stream.1.00
  16. primer click1.00
  17. 030.99
  18. An off-screen click at -1,-1 warms Chromium's activation gate.0.99
  19. live AX1.00
  20. 840.94
  21. Private AX SPI keeps occluded Electron trees fresh.1.00
  22. TRACK 7· JULY 1,20260.95
  23. ComputerUse1.00
  24. World's Fair0.99
• 

  3:26 #12 skipped

  shot 12·duplicate of #11

• 

  3:49 #13 skipped

  shot 13·duplicate of #11

• 

  4:10 #14 done30 line(s)

  shot 14·sharpness 1573.4

  1. World's Fair0.99
  2. AlEngineer0.99
  3. Cua1.00
  4. CROSS-PLATFORM1.00
  5. One interface to rule them all.0.98
  6. Different operating systems, different display servers, one agent-facing contract.1.00
  7. MAY 28260.96
  8. JUNE 20260.98
  9. Windows1.00
  10. Linux1.00
  11. UIA + messages1.00
  12. AT-SPI first0.99
  13. State comes from UIA/MSAA: pixels from PrintWindow/WGC.1.00
  14. Xll and Wayland expose opposite input models, so the driver0.99
  15. PostMessage handles many apps: SendInput is the explicit0.99
  16. keeps the agent contract semantic.0.97
  17. foreground rung.1.00
  18. X11 / XWAYLAND0.95
  19. background pixels, XTEST keys, recording0.98
  20. UIA / MSAA0.99
  21. PrintWindow / WGC0.98
  22. PostMessage1.00
  23. NATIVE WAYLAND1.00
  24. AT-SPI actions; raw keys need fallback0.98
  25. Windows: UIA + messages - Linux: X11 routes pixels, Wayland routes semantics0.98
  26. cua.ai1.00
  27. 000.66
  28. TRACK 7· JULY 1,20260.93
  29. ComputerUse1.00
  30. World's Fair0.96
• 

  4:41 #15 skipped

  shot 15·duplicate of #8

• 

  5:16 #16 skipped

  shot 16·duplicate of #8

• 

  5:51 #17 done68 line(s)

  shot 17·sharpness 1531.2

  1. AlEngineer0.98
  2. World'sFair1.00
  3. Cua1.00
  4. REGRESSION MATRIX1.00
  5. Every action path gets tested0.99
  6. 500+1.00
  7. Tests run against real toolkits and app-state changes.1.00
  8. SCORED ACTION CHECKS0.99
  9. effect plus background-held contract1.00
  10. 5 MODALITY RUNS0.99
  11. ACTION PATH X DISPATCH X SCOPE0.98
  12. TOOLKIT LANES1.00
  13. MODE0.99
  14. TARGET1.00
  15. DISPATCH1.00
  16. VERDICT0.98
  17. WINDOWS1.00
  18. WPF1.00
  19. WinUI31.00
  20. WebView21.00
  21. Electron1.00
  22. ax-bg1.00
  23. element_index1.00
  24. background1.00
  25. effect + held0.97
  26. px-bg1.00
  27. x,y0.91
  28. background1.00
  29. effect + held0.97
  30. QLINUX0.98
  31. GTK3/41.00
  32. Qt5/61.00
  33. Electron1.00
  34. ax-fg1.00
  35. element_index1.00
  36. foreground1.00
  37. effect1.00
  38. MACOS0.99
  39. AppKit1.00
  40. SwiftUI1.00
  41. WKWebView1.00
  42. Electron1.00
  43. px-fg1.00
  44. x,y0.93
  45. foreground1.00
  46. effect1.00
  47. px-desktop1.00
  48. screen pixels1.00
  49. desktop1.00
  50. effect1.00
  51. EFFECT1.00
  52. CONTRACT1.00
  53. Did the app state actually1.00
  54. Did the target stay0.99
  55. change?1.00
  56. backgrounded?1.00
  57. 6 PARITY CONTROLS1.00
  58. checkbox1.00
  59. click target1.00
  60. slider1.00
  61. scroll1.00
  62. text input1.00
  63. 8 actions x 6 controls x 10+ lanes = 500+ scored checks0.99
  64. cua.ai0.98
  65. Oun0.73
  66. TRACK 7· JULY 1,20260.95
  67. Computer Use0.98
  68. World's Fair0.96
• 

  6:21 #18 done31 line(s)

  shot 18·sharpness 1671.8

  1. AlEngineer0.99
  2. World'sFair1.00
  3. Cua1.00
  4. USED TODAY1.00
  5. Our early adopters1.00
  6. These products and agent labs use the same MCP and CLI driver for desktop apps.1.00
  7. PRESENTED BY1.00
  8. Microsoft1.00
  9. 1OH0.55
  10. Clicky1.00
  11. Hermes1.00
  12. Qwen Code0.98
  13. H Company1.00
  14. Droid Factory1.00
  15. CUA PRODUCT0.96
  16. AGENT RUNTIME1.00
  17. CODING AGENT0.99
  18. AGENT PLATFORM0.99
  19. FACTORY1.00
  20. ENTRY POINTS0.99
  21. TARGETS1.00
  22. LOOP1.00
  23. MCP, CLI, and SDK0.96
  24. real desktop apps0.98
  25. observe, act, verify0.97
  26. MCP and CLI across real apps1.00
  27. cua.ai0.99
  28. n0.73
  29. TRACK 7· JULY 1,20260.95
  30. ComputerUse1.00
  31. World's Fair0.98
• 

  6:31 #19 done23 line(s)

  shot 19·sharpness 1875.4

  1. AlEngineer0.99
  2. World's Fair0.98
  3. Cua1.00
  4. INTELLIGENCE1.00
  5. Now: can the agent use the computer?1.00
  6. PRESENTED BY1.00
  7. The driver gives agents hands. The next question is whether those hands1.00
  8. produce reliable work.1.00
  9. Microsoft1.00
  10. SURFACE1.00
  11. TEST1.00
  12. SPEAKER1.00
  13. Cua Driver1.00
  14. Cua Bench0.95
  15. Dillon DuPont0.97
  16. observe, act, verify0.97
  17. measure real task progress0.99
  18. intelligence layer1.00
  19. Cua Bench · intelligence section0.96
  20. cua.ai1.00
  21. TRACK 7· JULY 1, 20260.93
  22. ComputerUse1.00
  23. World's Fair0.99
• 

  6:42 #20 done22 line(s)

  shot 20·sharpness 1399.2

  1. AlEngineer0.98
  2. World's Fair0.99
  3. Cua1.00
  4. ACTIONS → CAN WE TRUST THEM0.98
  5. Cua Driver gives agents actions.1.00
  6. CuaBench measures whether those actions solve the task.1.00
  7. PRESENTED BY1.00
  8. Microsoft1.00
  9. 011.00
  10. 020.98
  11. 030.77
  12. setup1.00
  13. act1.00
  14. score1.00
  15. known state1.00
  16. the agent works1.00
  17. checks pass or fail0.99
  18. verifiable environments · every result reproducible0.98
  19. cua.ai1.00
  20. TRACK 7· JULY 1,20260.95
  21. World's Fair0.99
  22. Computer Use1.00
• 

  7:01 #21 done21 line(s)

  shot 21·sharpness 1171.0

  1. AlEngineer0.98
  2. World'sFair1.00
  3. Cua1.00
  4. A TASK = THREE PIECES0.99
  5. Every task is three pieces.1.00
  6. task.py1.00
  7. PRESENTED BY1.00
  8. from cua_bench import setup_task, solve_task, evaluate_task0.99
  9. Microsoft1.00
  10. @setup_task1.00
  11. def setup(env): env.open("invoice.xlsx")0.99
  12. @solve_task1.00
  13. def solve(agent): agent.run("Put the total in B2")1.00
  14. @evaluate_task1.00
  15. def check(env): return env.cell("B2") = "42"0.99
  16. setup · solve · evaluate · five platforms0.93
  17. cua.ai1.00
  18. TRACK 7· JULY 1, 20260.96
  19. World's Fair0.97
  20. AlEngineer0.99
  21. Computer Use0.97
• 

  7:46 #22 done21 line(s)

  shot 22·sharpness 1213.5

  1. AlEngineer0.98
  2. World'sFair1.00
  3. Cua1.00
  4. ONE FILE, ANY PLATFORM1.00
  5. Environments take scale and expertise.1.00
  6. env.py1.00
  7. env window1.00
  8. from bench_ui import launch_window, get_element_rect1.00
  9. # inline HTML, no browser, no server1.00
  10. rect + agent0.94
  11. pid = launch_window(html="<h1>Invoice</h1>")0.99
  12. Invoice0.99
  13. # real screen-space coords for the agent0.99
  14. rect = get_element_rect(pid, "h1")0.99
  15. THE AGENT SEES EXACTLY WHAT YOU DREW0.98
  16. author · launch· inspect · in one file0.92
  17. cua.ai1.00
  18. TRACK 7· JULY 1, 20260.96
  19. AlEngineer1.00
  20. Computer Use0.97
  21. World'sFair1.00
• 

  8:19 #23 done18 line(s)

  shot 23·sharpness 1135.3

  1. AlEngineer0.99
  2. World'sFair1.00
  3. Cua1.00
  4. 130 TASKS· CUA-BENCH-KICAD0.96
  5. One open catalog.1.00
  6. 1301.00
  7. 420.86
  8. 51.00
  9. VERIFIABLE TASKS1.00
  10. ENVIRONMENTS1.00
  11. PLATFORMS1.00
  12. $ cb run dataset cua-bench-kicad -m sonnet-4-51.00
  13. linux · windows · android · macos · web0.93
  14. cua.ai1.00
  15. TRACK 7· JULY 1,20260.96
  16. AlEngineer0.99
  17. Computer Use0.96
  18. World'sFair1.00

Transcript

154 cues· 2,311 words· 12,442 chars

Chapters

1. 0:00 Introduction and Vision of Cua
2. 2:40 Overview of Cua Driver and Background Operation
3. 6:34 Introduction to Cua Bench and Agent Evaluation
4. 10:50 Cua Fleet and GPU Infrastructure Optimization
5. 15:08 Q&A Session
6. 15:44 Discussion on Mobile and Android Support