vidtheque.

read-only demo

Videos q4Tr-DknG2M

Recursive Model Improvement — Lee Robinson, Cursor, SpaceXAI

index_state ready data_status ok

AI Engineer· published 2026-07-15· 0:20:32· en· indexed 2026-08-11 02:56

Open on YouTube

Scene timeline

  1. Shot 0, 0:00 to 0:03, 1 of 1 keyframes kept
  2. Shot 1, 0:03 to 0:05, 1 of 1 keyframes kept
  3. Shot 2, 0:05 to 0:12, 1 of 1 keyframes kept
  4. Shot 3, 0:12 to 0:21, 1 of 1 keyframes kept
  5. Shot 4, 0:21 to 0:29, 1 of 1 keyframes kept
  6. Shot 5, 0:29 to 0:32, 1 of 1 keyframes kept
  7. Shot 6, 0:32 to 0:35, 1 of 1 keyframes kept
  8. Shot 7, 0:35 to 0:38, 1 of 1 keyframes kept
  9. Shot 8, 0:38 to 0:42, 1 of 1 keyframes kept
  10. Shot 9, 0:42 to 0:50, 1 of 1 keyframes kept
  11. Shot 10, 0:50 to 0:56, 1 of 1 keyframes kept
  12. Shot 11, 0:56 to 1:15, 1 of 1 keyframes kept
  13. Shot 12, 1:15 to 1:44, 1 of 1 keyframes kept
  14. Shot 13, 1:44 to 2:12, 1 of 1 keyframes kept
  15. Shot 14, 2:12 to 2:41, 0 of 1 keyframes kept
  16. Shot 15, 2:41 to 2:59, 0 of 1 keyframes kept
  17. Shot 16, 2:59 to 3:21, 1 of 1 keyframes kept
  18. Shot 17, 3:21 to 3:45, 1 of 1 keyframes kept
  19. Shot 18, 3:45 to 4:01, 1 of 1 keyframes kept
  20. Shot 19, 4:01 to 4:28, 1 of 1 keyframes kept
  21. Shot 20, 4:28 to 4:34, 1 of 1 keyframes kept
  22. Shot 21, 4:34 to 4:54, 1 of 1 keyframes kept
  23. Shot 22, 4:54 to 5:29, 1 of 1 keyframes kept
  24. Shot 23, 5:29 to 5:55, 0 of 1 keyframes kept
  25. Shot 24, 5:55 to 6:23, 1 of 1 keyframes kept
  26. Shot 25, 6:23 to 6:50, 1 of 1 keyframes kept
  27. Shot 26, 6:50 to 7:03, 0 of 1 keyframes kept
  28. Shot 27, 7:03 to 7:37, 1 of 1 keyframes kept
  29. Shot 28, 7:37 to 8:10, 1 of 1 keyframes kept
  30. Shot 29, 8:10 to 8:26, 1 of 1 keyframes kept
  31. Shot 30, 8:26 to 9:05, 0 of 1 keyframes kept
  32. Shot 31, 9:05 to 9:50, 1 of 1 keyframes kept
  33. Shot 32, 9:50 to 10:20, 1 of 1 keyframes kept
  34. Shot 33, 10:20 to 10:49, 1 of 1 keyframes kept
  35. Shot 34, 10:49 to 11:32, 1 of 1 keyframes kept
  36. Shot 35, 11:32 to 11:39, 1 of 1 keyframes kept
  37. Shot 36, 11:39 to 12:06, 1 of 1 keyframes kept
  38. Shot 37, 12:06 to 12:34, 1 of 1 keyframes kept
  39. Shot 38, 12:34 to 12:48, 1 of 1 keyframes kept
  40. Shot 39, 12:48 to 13:05, 1 of 1 keyframes kept
  41. Shot 40, 13:05 to 13:49, 1 of 1 keyframes kept
  42. Shot 41, 13:49 to 14:17, 1 of 1 keyframes kept
  43. Shot 42, 14:17 to 14:44, 1 of 1 keyframes kept
  44. Shot 43, 14:44 to 15:29, 1 of 1 keyframes kept
  45. Shot 44, 15:29 to 15:51, 1 of 1 keyframes kept
  46. Shot 45, 15:51 to 16:19, 1 of 1 keyframes kept
  47. Shot 46, 16:19 to 16:47, 0 of 1 keyframes kept
  48. Shot 47, 16:47 to 17:15, 0 of 1 keyframes kept
  49. Shot 48, 17:15 to 17:47, 1 of 1 keyframes kept
  50. Shot 49, 17:47 to 18:30, 1 of 1 keyframes kept
  51. Shot 50, 18:30 to 19:04, 1 of 1 keyframes kept
  52. Shot 51, 19:04 to 19:52, 1 of 1 keyframes kept
  53. Shot 52, 19:52 to 20:11, 1 of 1 keyframes kept
  54. Shot 53, 20:11 to 20:15, 1 of 1 keyframes kept
  55. Shot 54, 20:15 to 20:30, 0 of 1 keyframes kept
  56. Shot 55, 20:30 to 20:31, 1 of 1 keyframes kept

56 shot(s).

keyframes kept every frame deduplicated

What was stored

cues
199
whisperx 199
chunks
35
from 199 cues
keyframes
48
kept of 56 captured
frames with text
47
1,213 lines read
chapters
13
from the source metadata
keyframe bytes
6.2 MB
word timings on 199 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage state model started took
fetch done — 2026-08-11 02:52 1m 15s
stt done — 2026-08-11 02:53 23s
chunk done — 2026-08-11 02:53 0s
text_embed done — 2026-08-11 02:53 1s
keyframe done — 2026-08-11 02:53 2m 11s
ocr done — 2026-08-11 02:56 20s
frame_embed done — 2026-08-11 02:56 8s

Frames, and what the machine read

  • 0:02 #0 done2 line(s)

    shot 0·sharpness 455.1

    1. AlEngineer0.96
    2. World's Fair0.97

  • 0:03 #1 done2 line(s)

    shot 1·sharpness 656.7

    1. AlEngineer0.96
    2. World's Fair0.99

  • 0:10 #2 done24 line(s)

    shot 2·sharpness 2735.8

    1. LAB & PLATINUM SPONSORS0.99
    2. Amazon AGI Lab0.96
    3. ANTHROP\C1.00
    4. Google DeepMind1.00
    5. MINIMAX0.96
    6. OpenAI0.92
    7. Akamai1.00
    8. arize1.00
    9. aws1.00
    10. Braintrust bright data0.99
    11. B1.00
    12. Browserbase1.00
    13. docker1.00
    14. :neo4j0.92
    15. ORACLE1.00
    16. PayPal1.00
    17. qodo1.00
    18. reducto1.00
    19. Sonar1.00
    20. Makers of0.99
    21. togetherai1.00
    22. Unblocked1.00
    23. WorkOS1.00
    24. SonarQube1.00

  • 0:16 #3 done2 line(s)

    shot 3·sharpness 197.8

    1. CURSOR1.00
    2. AIE1.00

  • 0:28 #4 done4 line(s)

    shot 4·sharpness 571.1

    1. LEE ROBINSON0.99
    2. ML, MODEL BEHAVIOR0.99
    3. CURSOR1.00
    4. AIE0.97

  • 0:32 #5 done18 line(s)

    shot 5·sharpness 949.5

    1. LEE ROBINSON0.99
    2. sneos0.61
    3. promptql0.92
    4. jineer0.90
    5. 'sFair0.99
    6. ML, MODEL BEHAVIOR0.98
    7. WoridsFair0.93
    8. CURSOR1.00
    9. Cleric0.99
    10. Vorld's Fair0.86
    11. Microsoht0.92
    12. AIE1.00
    13. Greducto0.99
    14. Worids Fair0.94
    15. World's Fair0.93
    16. Ravenna0.99
    17. Workd'sFa0.93
    18. eNCORD0.93

  • 0:33 #6 done91 line(s)

    shot 6·sharpness 2855.1

    1. World'sFair1.00
    2. World's Fair0.98
    3. World's Fai0.97
    4. World's Fair0.94
    5. World's Fair0.97
    6. Red Hat0.96
    7. World's Fain0.95
    8. World's Fair0.99
    9. mezmo*0.94
    10. fiddler0.97
    11. sigg0.79
    12. World's Fair0.96
    13. Cate0.72
    14. INNGEST0.97
    15. Snorkel0.98
    16. World's Fair0.96
    17. PRIOR1.00
    18. World's Fair0.97
    19. BAND0.98
    20. Keycard1.00
    21. AUTOMATTIC1.00
    22. orld's Fair0.97
    23. Z.AI0.95
    24. World's Fair0.93
    25. Worlds Fair0.83
    26. WorkOS0.97
    27. :neo4j0.97
    28. World's Fair0.98
    29. Google DeepMind1.00
    30. World's Fair0.99
    31. qodo0.99
    32. World's Fain0.94
    33. Cleric1.00
    34. Wor0.99
    35. A ATLASSIAN0.92
    36. GGRAVITEE0.97
    37. Meticulous1.00
    38. rld's Fair0.99
    39. World's Fair0.99
    40. Braintrust1.00
    41. Amazon AGI Lab1.00
    42. World's Fair0.97
    43. World's Fair0.96
    44. World's Fair0.98
    45. Microsoft1.00
    46. World's Fair0.96
    47. arize0.96
    48. Pr0.99
    49. reducto1.00
    50. Gradium0.97
    51. FACTORY1.00
    52. Microsoft1.00
    53. ANTHROPIC1.00
    54. Ddud0.56
    55. OpenAl0.90
    56. aws1.00
    57. OpenAl0.92
    58. World's Fair0.91
    59. bright data0.86
    60. Worids Fair0.88
    61. World's Fair0.99
    62. MINIMAX1.00
    63. World's Fair0.98
    64. L.AI0.90
    65. World'sFai1.00
    66. ANTHROPIC1.00
    67. World's Fair0.95
    68. d's Fair0.98
    69. World's Fair0.98
    70. DATADOG1.00
    71. World's Fair0.96
    72. Resolve.ai1.00
    73. CODER1.00
    74. d'sFai0.94
    75. @Airbyte0.95
    76. Optiver0.99
    77. greptle0.91
    78. World's Fair0.97
    79. builder.io1.00
    80. World's Fair0.98
    81. Ravenna1.00
    82. World'sFair1.00
    83. tKi0.96
    84. 's Fair0.98
    85. World's Fair0.97
    86. cognee1.00
    87. World'sFair1.00
    88. BeNCORD0.89
    89. I'sF0.80
    90. World's Far0.97
    91. tgus0.77

  • 0:35 #7 done210 line(s)

    shot 7·sharpness 3498.8

    1. Fair0.98
    2. ORACLE1.00
    3. World's Fair0.95
    4. arize1.00
    5. World's Fair0.96
    6. Google DeepMind1.00
    7. World's Fair0.89
    8. :neo4j0.95
    9. World's Fair0.90
    10. Z.AI0.99
    11. World's Fair0.85
    12. bright data0.99
    13. World's Fair0.96
    14. Browserba1.00
    15. mpute co.0.98
    16. World's Fair0.90
    17. extend1.00
    18. World's Fair0.96
    19. vast.ai1.00
    20. World's Fair0.95
    21. Ref.1.00
    22. World's Fair0.98
    23. RedHat1.00
    24. World's Fair0.99
    25. mezmo*0.97
    26. Worid's Fair0.96
    27. stigg0.92
    28. World's Fair0.94
    29. sFair0.86
    30. RELI0.95
    31. World's Fair0.98
    32. VAPI1.00
    33. World's Fair0.99
    34. Modal1.00
    35. World's Fair0.98
    36. promptql1.00
    37. World's Fair0.96
    38. THE VELOCITY ROOM0.97
    39. World's Fair0.96
    40. fiddler0.94
    41. World's Fair0.98
    42. Superconduc0.98
    43. eal0.96
    44. World's Fair0.96
    45. ZERO0.88
    46. World's Fair0.99
    47. AlEngineer0.97
    48. comet1.00
    49. World's Fair0.97
    50. authe0.98
    51. World's Fair0.95
    52. sFair0.91
    53. SOIO.IO0.90
    54. World's Fair0.96
    55. granica1.00
    56. World's Fair0.99
    57. dash00.99
    58. World's Fair0.99
    59. PRIOR1.00
    60. World's Fair1.00
    61. alOcean1.00
    62. World's Fair0.95
    63. Venmce0.79
    64. World's Fair0.95
    65. POSTMAN1.00
    66. World's Fair0.99
    67. Composio1.00
    68. World's Fair0.95
    69. sFair0.90
    70. Modular1.00
    71. Worid's Fair0.97
    72. MERGE1.00
    73. World's Fair0.97
    74. AUTOMATTIC1.00
    75. World's Fair0.98
    76. Buildkit0.99
    77. byteDB1.00
    78. Worid's Fair0.93
    79. Zed1.00
    80. World's Fair0.94
    81. Keycard1.00
    82. Worid's Fair0.94
    83. Meticulous0.99
    84. World's Fair0.91
    85. sFair0.93
    86. Daytona1.00
    87. World's Fair0.99
    88. twilio0.96
    89. World's Fair0.96
    90. C GRAVITEE0.93
    91. World's Fair0.93
    92. PlanetScal0.98
    93. nporal0.98
    94. World's Fair0.97
    95. Llamalndex0.95
    96. World's Fair0.98
    97. INNGEST0.94
    98. World's Fair0.96
    99. © BAND0.88
    100. World's Fair0.95
    101. Cleric1.00
    102. World's Fair0.98
    103. À ATLASSIAN0.95
    104. World's Fair0.98
    105. FACTORY1.00
    106. World's Fain0.93
    107. sFair0.88
    108. baseten1.00
    109. World's Fair0.98
    110. Snorkel1.00
    111. World's Fair0.93
    112. Z.AI0.97
    113. World's Fair0.97
    114. qodo0.95
    115. World's Fair0.96
    116. PayPal1.00
    117. World's Fair0.95
    118. Gradium1.00
    119. World's Fair0.96
    120. ANTHROPV0.99
    121. AGI Lab1.00
    122. World's Fair0.98
    123. Browserbase1.00
    124. World's Fair1.00
    125. :neo4j0.81
    126. World's Fair0.98
    127. Google DeepMind0.99
    128. World's Fair0.96
    129. arize1.00
    130. World's Fair0.96
    131. reducto1.00
    132. World's Fair0.94
    133. Microsoft0.95
    134. World's Fai0.95
    135. sFair0.86
    136. OpenAl0.98
    137. World's Fair0.98
    138. WorkOS0.88
    139. World's Fair0.97
    140. Amazon AGI Lab1.00
    141. World's Far0.94
    142. Microsoft1.00
    143. World's Fair0.96
    144. ORACLE1.00
    145. World's Fair0.89
    146. bright data0.98
    147. World's Fair1.00
    148. Google DeepM1.00
    149. rosoft1.00
    150. World's Fair0.91
    151. docker1.00
    152. World's Fair0.95
    153. Braintrust1.00
    154. World's Fair0.97
    155. OpenA'0.99
    156. rid's Fair0.96
    157. MINIMAX0.99
    158. World's Fair0.97
    159. Z.AI0.99
    160. World's Fair0.99
    161. ANTHROPIC1.00
    162. World's Fai0.96
    163. Fair0.86
    164. snyk1.00
    165. World's Fair0.97
    166. aws0.98
    167. World's Fair0.94
    168. togetherai1.00
    169. World'sF0.99
    170. Akamai0.98
    171. World's Fair0.94
    172. Unblocked0.96
    173. Worldin0.87
    174. Airbyte1.00
    175. Worl0.87
    176. ghtrun1.00
    177. ain0.97
    178. World's Fair0.97
    179. World's Fair0.98
    180. DATADOC0.95
    181. orld's Fair0.93
    182. Resolve.ai1.00
    183. Word's Far0.86
    184. World's Fa0.94
    185. Op1.00
    186. Worid's Fal0.92
    187. Fair1.00
    188. LanceDB0.97
    189. World's Fair0.98
    190. builderio0.95
    191. World's Fair0.97
    192. : Ravenna0.97
    193. World's Fair0.99
    194. CopilotKit0.99
    195. VERIS1.00
    196. Wor1.00
    197. World's Far0.93
    198. pe0.98
    199. Microsoft0.97
    200. World's Fair0.99
    201. TOP1.00
    202. World's Fair0.95
    203. cognee1.00
    204. forld's Fair0.92
    205. eNCORD0.93
    206. World's Fair0.92
    207. rid'sFa0.97
    208. ti0.66
    209. World's Fai0.98
    210. ●。。0.85

  • 0:40 #8 done9 line(s)

    shot 8·sharpness 451.1

    1. AlEngineer0.98
    2. Amazon A1.00
    3. World's Fair0.99
    4. AIEngineer0.95
    5. OpenAI0.94
    6. World's Fair1.00
    7. AlEngineer1.00
    8. together1.00
    9. World's Fair0.99

  • 0:45 #9 done10 line(s)

    shot 9·sharpness 373.2

    1. orkOS0.99
    2. WorFair0.98
    3. Amazon1.00
    4. gineer1.00
    5. AIEngi0.94
    6. I's Fair0.90
    7. World1.00
    8. VS0.95
    9. toge0.98
    10. 70.54

  • 0:54 #10 done11 line(s)

    shot 10·sharpness 426.3

    1. AlEngineer0.99
    2. WorkOS1.00
    3. orld's Fair1.00
    4. Ama1.00
    5. AlEngineer0.98
    6. lorld's Fair0.96
    7. intrust1.00
    8. W1.00
    9. eer1.00
    10. aws1.00
    11. sFair1.00

  • 0:58 #11 done15 line(s)

    shot 11·sharpness 2089.6

    1. AlEngineer0.98
    2. World'sFair1.00
    3. Goal: build the best general Al model1.00
    4. AlEngineer1.00
    5. AGI Lab0.97
    6. Norld'sl0.96
    7. 'sFair0.93
    8. neer1.00
    9. Open1.00
    10. The Future of Cursor0.98
    11. - AlEngineer0.93
    12. etherai0.99
    13. rld'sl0.95
    14. Lee Robinson / ML, Model Behavior0.97
    15. CURSOR1.00

  • 1:27 #12 done18 line(s)

    shot 12·sharpness 2191.4

    1. AlEngineer0.99
    2. World's Fair0.97
    3. User feedback1.00
    4. Scale/improve data1.00
    5. Scale/improve training1.00
    6. Better model1.00
    7. AIE1.00
    8. WorkOs0.95
    9. Worl1.00
    10. - AlEnginer0.96
    11. orld'!0.75
    12. Bra0.91
    13. The Future of Cursor1.00
    14. al0.97
    15. /orl0.98
    16. — AIE0.90
    17. Lee Robinson / ML, Model Behavior0.99
    18. CURSOR1.00

  • 1:52 #13 done20 line(s)

    shot 13·sharpness 2637.8

    1. AlEngineer0.98
    2. World's Fair0.97
    3. User feedback, online metrics1.00
    4. High quality evals, hard training tasks1.00
    5. Outer loop0.99
    6. Inner loop:0.97
    7. Climb evals1.00
    8. Collect & create data, shape reward0.99
    9. Better model1.00
    10. d'sFair1.00
    11. azon1.00
    12. AlEngine0.98
    13. intru1.00
    14. World's1.00
    15. The Future of Cursor1.00
    16. d'sFai0.96
    17. ngineer1.00
    18. togel0.95
    19. Lee Robinson / ML, Model Behavior0.98
    20. CURSOR1.00

  • 2:26 #14 skipped

    shot 14·duplicate of #12

  • 2:48 #15 skipped

    shot 15·duplicate of #12

  • 3:10 #16 done29 line(s)

    shot 16·sharpness 1577.4

    1. AlEngineer0.98
    2. Terminal-bench 2.0 Score0.99
    3. Composer 2.51.00
    4. World'sFair1.00
    5. 70%1.00
    6. Composer 20.99
    7. 60%1.00
    8. 50%1.00
    9. Composer 1.50.99
    10. Composer 10.96
    11. WorkOS1.00
    12. Wc0.99
    13. 40%1.00
    14. Nov'251.00
    15. Dec'251.00
    16. Jan'260.99
    17. Feb '260.94
    18. Mar'261.00
    19. Apr'260.99
    20. May'261.00
    21. Jun '260.95
    22. AlEngineer1.00
    23. Norld's1.00
    24. E0.98
    25. The Future of Cursor1.00
    26. av0.94
    27. Ic0.75
    28. Lee Robinson / ML, Model Behavior0.98
    29. CURSOR1.00

  • 3:40 #17 done44 line(s)

    shot 17·sharpness 1850.7

    1. 75% CursorBench 3.1 score0.99
    2. AlEngineer1.00
    3. Fable 5 high1.00
    4. World'sFair1.00
    5. 70%1.00
    6. (default)1.00
    7. 65%1.00
    8. Composer 2.50.98
    9. 60%1.00
    10. GPT-5.5 medium1.00
    11. PRESENTED BY0.96
    12. 55%1.00
    13. Opus 4.8 high1.00
    14. (default)1.00
    15. (default)1.00
    16. Microsoft1.00
    17. 50%1.00
    18. Gemini 3.5 Flash1.00
    19. Sonnet 4.6 high1.00
    20. 45%1.00
    21. (default)1.00
    22. Kimi 2.60.99
    23. 40%1.00
    24. 35%1.00
    25. 30%1.00
    26. AlEngineer0.99
    27. $201.00
    28. $161.00
    29. $121.00
    30. $81.00
    31. $41.00
    32. $00.99
    33. rld's Fai0.94
    34. OR/0.96
    35. Average cost per task1.00
    36. AIEr0.77
    37. NM0.66
    38. Worl0.92
    39. The Future of Cursor1.00
    40. AlI0.60
    41. rl0.99
    42. Un1.00
    43. Lee Robinson / ML, Model Behavior0.98
    44. CURSOR1.00

  • 3:56 #18 done61 line(s)

    shot 18·sharpness 2358.6

    1. AlEngineer0.99
    2. World'sFair1.00
    3. Artificial Analysis Coding Agent Index1.00
    4. Composite average pass@1 across SWE-Bench-Pro-Hard-AA, Terminal-Bench v2, and1.00
    5. SWE-Atlas-QnA· Higher is better0.98
    6. Artificial Analysis0.98
    7. PRESENTED BY0.98
    8. Microsoft1.00
    9. 651.00
    10. 631.00
    11. 611.00
    12. 581.00
    13. 531.00
    14. 501.00
    15. 501.00
    16. 481.00
    17. 431.00
    18. Al Claude Code0.93
    19. Codex1.00
    20. Cursor CLI0.95
    21. Cursor CLI0.92
    22. Cursor CUI0.90
    23. Al Claude Code0.94
    24. Al Claude Code0.94
    25. Al Claude Code0.98
    26. Cursor CLI1.00
    27. G Gemini CU0.94
    28. Al Opus 4.70.96
    29. (Max)0.99
    30. GPT-5.50.99
    31. (XHigh)0.96
    32. Composer 2.51.00
    33. Fast1.00
    34. Al Opus 4.70.97
    35. (Medium)0.98
    36. (Medium)0.97
    37. GPT-5.51.00
    38. GLM-510.98
    39. 区0.62
    40. Kimi K2.61.00
    41. ar0.63
    42. DeepSeek V40.98
    43. (High)1.00
    44. Pro1.00
    45. Composer 21.00
    46. G1.00
    47. Gemini 3.10.99
    48. (High)0.92
    49. Pro0.97
    50. AlEngineer1.00
    51. rld's Fai0.93
    52. Amazor1.00
    53. AIEr0.90
    54. raintrust1.00
    55. 'ork0.97
    56. The Future of Cursor0.98
    57. AlEngineer0.98
    58. rld's Fair0.98
    59. tog0.93
    60. Lee Robinson/ ML, Model Behavior0.99
    61. CURSOR1.00

  • 4:04 #19 done25 line(s)

    shot 19·sharpness 2340.2

    1. AlEngineer0.99
    2. What to improve1.00
    3. World's Fair0.98
    4. • Bigger& smarter model0.96
    5. Faster training stack0.94
    6. PRESENTED BY0.99
    7. • Full pretrain to control model behavior0.97
    8. · Intelligent beyond coding0.96
    9. Microsoft1.00
    10. Multi-effort0.96
    11. • More diverse, harder data0.96
    12. • New evals0.95
    13. ·Scale RL further0.95
    14. Al0.94
    15. WorkO1.00
    16. Wor1.00
    17. — AlEngineer0.97
    18. orld's1.00
    19. Br1.00
    20. The Future of Cursor0.99
    21. av.0.93
    22. − AI0.71
    23. orl0.93
    24. Lee Robinson/ ML, Model Behavior0.98
    25. CURSOR1.00

  • 4:32 #20 done5 line(s)

    shot 20·sharpness 1925.1

    1. AlEngineer0.99
    2. World's Fair0.97
    3. PRESENTED BY1.00
    4. Improving the outer loop1.00
    5. Microsoft1.00

  • 4:42 #21 done

    shot 21·sharpness 1580.1

  • 4:58 #22 done

    shot 22·sharpness 2503.9

  • 5:52 #23 skipped

    shot 23·duplicate of #13

The page's on-screen-text budget of 600 lines is spent, so the last cards in this grid list fewer lines than they hold. Narrow the page with ?frames= to read them.

Transcript

199 cues· 3,959 words· 20,860 chars

  1. 0:13 Please welcome to the stage the machine learning engineer, Model Behavior at Cursor, Lee Robinson.
  2. 0:36 All right.
  3. 0:36 Hey, everyone.
  4. 0:39 I'm excited to be here, excited to be back at AI Engineer and talk a little bit about how we're training models at Cursor.
  5. 0:47 So how we train the models and also how the model learns to train itself or recursive model improvement.
  6. 0:54 So our goal at Cursor is to build the best possible AI models, which might make sense.
  7. 1:01 You might have heard of this equation of if we just give the models more compute, we can get a better model out.
  8. 1:06 And I think this is a helpful simplification of the problem, but I want to actually click in a few layers deeper in the talk today and talk about all the different pieces that go into training these models.
  9. 1:16 So we can think about this loop.
  10. 1:19 We put a model out into the world, and then we get feedback from you all when you use the model, what goes well or places we can improve.
  11. 1:25 We use that to scale and improve the data that we do for the next round of training.
  12. 1:29 And we also then increase the amount of compute and scale up training overall to make a new model.
  13. 1:35 And this is a loop that can just go over and over again.
  14. 1:38 However, if you see my helpful snail or turtle to bunny meter down in the bottom right, it's pretty slow.
  15. 1:45 This is gonna be a serial process.
  16. 1:47 And you can only do, in this instance, one big run at a time.
  17. 1:51 So we wanna make this a little bit faster.
  18. 1:53 But I'll actually go another layer deeper and add some more color here.
  19. 1:57 There's actually two loops, the outer loop and the inner loop.
  20. 2:01 On the outer loop, we have the feedback coming in, but we also have data like online metrics.
  21. 2:07 So running A-B tests and seeing what users prefer a different checkpoint of a model.
  22. 2:11 That's going to then flow into hopefully making better high quality evals that
  23. 2:16 help ensure we're getting the right behaviors we want out of the model, and also being able to create much more difficult problems for the models to try to solve where we then can kind of shape the rewards that we want to get during training.
  24. 2:28 So we want to climb that inner loop as well.
  25. 2:33 We have been training models for about a year at the large scale at Cursor, and I want to talk about some of our progress so far.
  26. 2:41 So we put out Composer 2.5 in May, and it's now the most popular model in Cursor, which is exciting.
  27. 2:48 And we scaled up training here quite a bit by generating more RL environments, trying out some new methods for learning, and also just making more ambitious problems for the models to solve.
  28. 3:00 And the results have been pretty promising so far.
  29. 3:03 Like I mentioned, this is still a new effort for us.
  30. 3:06 ML has really been in the blood of Cursor since the start where we were training more specialized models for things like tab or code autocomplete.
  31. 3:13 But really in the past year we've staffed up and built a team with ambitions to train state of the art models and we made some pretty good progress just in the last 12 months.
  32. 3:22 People like Composer right now, I think because it is both fast and pretty smart, and also cost effective.
  33. 3:29 And as we've heard from other speakers today, I think there is a space in the market right now for that type of model, in addition to also having the most intelligent models in the world.
  34. 3:39 And we think it's important to have a good selection of both of these type of things.
  35. 3:42 So Composer we think is serving a good niche here.
  36. 3:46 And when we released it, we were honestly pretty impressed with some of the public evals.
  37. 3:51 It did a little better than we expected.
  38. 3:53 On artificial analysis, it was a pretty modest jump.
  39. 3:56 However, there were a lot of behaviors that we found that we really wanted to improve for the next version of the model.
  40. 4:02 Notably, we wanted to have a much bigger and smarter model.
  41. 4:05 We wanted to control every aspect of training.
  42. 4:07 So ideally, doing a full pre-train from scratch versus the previous open source base of Kimmy that we were using.
  43. 4:13 We wanted to infuse new data so that we can make the model great outside of more things than just coding, but more of a general model.
  44. 4:20 And then also just scale up every part of the training process.
  45. 4:24 More data, more compute, and really pushing RL as far as we can.
  46. 4:29 So first I want to talk about improving the outer loop, and then we'll drill into the inner loop.
  47. 4:35 If you haven't used Cursor in a while, you might think about it as this IDE or tab autocomplete thing.
  48. 4:42 And in reality, the vast, vast majority of our revenue today comes from agent usage.
  49. 4:47 And that means that all of the data inside of Cursor is also coming from agent usage, and we can use that to train better models.
  50. 4:55 For example, we have two different buckets of feedback.

Chapters

  1. 0:00 <Untitled Chapter 1>
  2. 0:37 Introduction and recursive model improvement overview
  3. 1:55 The two-loop training framework (inner and outer loops)
  4. 2:33 Progress and success of Composer 2.5
  5. 4:31 Improving the outer loop with user feedback
  6. 5:40 Climbing the inner loop with high-quality evals
  7. 6:52 Solving reward hacking in public benchmarks
  8. 8:27 Scaling training with ambitious problems
  9. 9:53 New learning methods: Teacher-student textual feedback
  10. 11:34 Scaling compute infrastructure with SpaceX and Colossus
  11. 13:06 Understanding compute allocation in model training
  12. 15:28 Agent-based automation and research efficiency
  13. 18:30 The recursive future: Models training models

Open at this second