Stop Burning Tokens: Why self-improvement needs domain expertise first - Annabell Schäfer, Langfuse

What was stored

cues

166

whisperx 166

chunks

31

from 166 cues

keyframes

22

kept of 40 captured

frames with text

22

504 lines read

chapters

13

from the source metadata

keyframe bytes

4.5 MB

word timings on 166 cues

Provenance

Frames, and what the machine read

• 

  0:20 #0 done8 line(s)

  shot 0·sharpness 1129.3

  1. ealangfuse0.84
  2. AI ENGINEER· SAN FRANCISCO0.96
  3. by ClickHouse0.99
  4. A MINIMAL SELF-OPTIMIZATION EXPERIMENT1.00
  5. Stop burning tokens1.00
  6. Self-improvement needs domain expertise first.0.99
  7. Annabell Schäfer - Growth Engineer0.99
  8. SELF-IMPROVING AGENTS0.98
• 

  0:49 #1 done25 line(s)

  shot 1·sharpness 1071.8

  1. langfuse0.87
  2. pre.0.99
  3. karpathy / autoresearch0.98
  4. pops1.00
  5. Peter Steinberger0.97
  6. 0 ..0.59
  7. @steipete1.00
  8. :ode1.00
  9. Issues 541.00
  10. * Pull requests 130.96
  11. Vertaling tonen0.99
  12. Here's your monthly reminder that you shouldn't be prompting coding0.98
  13. agents anymore.1.00
  14. autoresea0.99
  15. Public1.00
  16. You should be designing loops that prompt your agents.0.99
  17. 11:58 a.m. - 7 jun. 2026 · 8,3 mln. Weergaven0.94
  18. master1.00
  19. 3Branches0 Ta0.96
  20. "I don't write prompts anymore. I1.00
  21. Stop prompting coding agents. Design the loops1.00
  22. autoresearch —autonomous research loops,0.97
  23. have loops."0.99
  24. that prompt them.1.00
  25. 87.2k★.1.00
• 

  1:22 #2 done2 line(s)

  shot 2·sharpness 545.3

  1. langfuse0.90
  2. code_compiles1.00
• 

  1:40 #3 done8 line(s)

  shot 3·sharpness 1617.9

  1. langfuse0.99
  2. The target you give an agent is1.00
  3. always incomplete1.00
  4. optimal1.00
  5. destination1.00
  6. original1.00
  7. aim1.00
  8. START1.00
• 

  2:30 #4 done37 line(s)

  shot 4·sharpness 2286.2

  1. langfuse0.92
  2. 02 - CONTEXT0.99
  3. Langfuse builds the infrastructure to run0.99
  4. improvement loops - for humans and agents1.00
  5. ONLINE1.00
  6. ONLINE1.00
  7. OFFLINE1.00
  8. OFFLINE1.00
  9. OFFLINE1.00
  10. Trace1.00
  11. Monitor1.00
  12. Build1.00
  13. Experiment1.00
  14. Evaluate1.00
  15. datasets0.98
  16. traces· sessions0.97
  17. dashboards · LLM-as-0.96
  18. datasets· features-as-0.96
  19. prompts · models - code0.91
  20. judges - custom evals0.96
  21. agents· prompts0.95
  22. judge . feedback0.95
  23. tests1.00
  24. variants1.00
  25. annotation1.00
  26. DEPLOY1.00
  27. Know what's1.00
  28. Find what's1.00
  29. Capture what you1.00
  30. Test hypothesis1.00
  31. Check if it1.00
  32. going on0.96
  33. interesting/1.00
  34. want to work1.00
  35. worked1.00
  36. wrong1.00
  37. STOP BURNING TOKENS0.99
• 

  2:45 #5 done6 line(s)

  shot 5·sharpness 1445.2

  1. elangfuse0.98
  2. 03 - THE LOOPS FOR AGENT SYSTEMS1.00
  3. What is the clearest cut target function we can find?0.99
  4. What can we learn about the role of target functions from running1.00
  5. auto-optimization on it?1.00
  6. STOP BURNING TOKENS0.98
• 

  3:14 #6 done16 line(s)

  shot 6·sharpness 1939.9

  1. elangfuse0.97
  2. 03 - THE CLEAR CUT TARGET1.00
  3. A single-label classification task has a clear cut target1.00
  4. function1.00
  5. Item to categorize0.98
  6. Available labels1.00
  7. Accuracy1.00
  8. Order1.00
  9. True label = predicted0.99
  10. label?1.00
  11. Complaint1.00
  12. yes/no1.00
  13. Inquiry1.00
  14. True label0.99
  15. Order1.00
  16. STOP BURNING TOKENS1.00
• 

  3:30 #7 done10 line(s)

  shot 7·sharpness 1237.6

  1. langfuse0.93
  2. The 'Agent'0.92
  3. Single label classifier1.00
  4. The optimizer1.00
  5. Target function0.99
  6. Single label classification1.00
  7. Claude running in loop to get0.99
  8. ground truth1.00
  9. to 92% accuracy1.00
  10. STOP BURNING TOKENS1.00
• 

  3:31 #8 done5 line(s)

  shot 8·sharpness 616.4

  1. PART TWO - THE EXPERIMENT0.98
  2. 011.00
  3. A minimal self-optimization loop0.99
  4. RANDOM PETER1.00
  5. our shorthand for a prompt hand-seeded by domain experts. The human baseline.0.99
• 

  3:56 #9 done32 line(s)

  shot 9·sharpness 2433.6

  1. langfuse0.96
  2. 05 - THE EXPERIMENT0.97
  3. The setup1.00
  4. The Target Function1.00
  5. Classified arXiv paper— primary label, from title +0.99
  6. TASK1.00
  7. abstract1.00
  8. Classification allows for a clean0.99
  9. accuracy signal0.99
  10. DATASET1.00
  11. 200 fit· 100 validate· 300 test0.98
  12. The'Agent'0.99
  13. CLASSIFIER1.00
  14. GPT-5-4 nano—the cheaper model0.97
  15. Simple on purpose, to constrain the1.00
  16. degrees of freedom for change1.00
  17. START PROMPT1.00
  18. A flat list of labels0.99
  19. OPTIMIZER1.00
  20. Claude Opus4.8 — proposes prompt updates0.96
  21. The Optimization process1.00
  22. Powerful model as optimizer, helpful1.00
  23. GPT-5-4 prompting guide1.00
  24. information as context1.00
  25. REFERENCES1.00
  26. task.md for overall loop process description0.99
  27. COMPARISON1.00
  28. GPT 5-5 baseline performed with 84.2%0.99
  29. accuracy1.00
  30. STOP BURNING TOKENS1.00
  31. Source: Data licensed under Apache License 2.00.98
  32. https://www.kaggle.com/datasets/sumitm004/arxiv-scientific-research-papers-dataset1.00
• 

  4:35 #10 skipped

  shot 10·duplicate of #9

• 

  5:21 #11 done103 line(s)

  shot 11·sharpness 2308.0

  1. langfuse0.94
  2. 05 - THE INPUTS0.94
  3. The target function maps inputs to their expected labels1.00
  4. Dataset arxiv-cs10-singlelabel-v1-small-fit0.99
  5. + New item0.99
  6. Upload CSV0.97
  7. 个0.95
  8. Items1.00
  9. Experiments1.00
  10. Q Search...0.95
  11. IDs1.00
  12. Filters1.00
  13. Columns 8/81.00
  14. 目1.00
  15. Fit set:1.00
  16. Item id1.00
  17. ①Status0.96
  18. C. Input0.99
  19. Expected Output1.00
  20. Metadata1.00
  21. Actions1.00
  22. Run, look at errors, form0.99
  23. f71f8d3d-aa6...0.96
  24. Active1.00
  25. 2021.00
  26. 06-1.00
  27. 05:1.00
  28. Team, Faster than the1.00
  29. title: "Faster than the1.00
  30. 1 Itens0.86
  31. Engineering"1.00
  32. label: "Software0.98
  33. 11 Items0.86
  34. seed: 202606151.00
  35. split: "fit"0.98
  36. hypothesis for prompt updates0.99
  37. Customer: Tool Integration,1.00
  38. Collaboration, and1.00
  39. }0.92
  40. source_id: "abs-1.00
  41. 2606.01772v1"1.00
  42. Validation set:1.00
  43. 460a43e7-56...1.00
  44. Active1.00
  45. 05:0.99
  46. 2021.00
  47. 06-1.00
  48. 3 Itens0.91
  49. title: "From Failed0.99
  50. Repairing Harness Flaws"1.00
  51. Trajectories to Reliable1.00
  52. LLM Agents: Diagnosing and1.00
  53. 1 Itens0.94
  54. }0.95
  55. Engineering"0.99
  56. label: "Software0.98
  57. 11 Items0.91
  58. seed: 202606151.00
  59. split: "fit"0.99
  60. source_id: "abs-0.98
  61. 2606.06324v1"1.00
  62. Run to decide if a change is1.00
  63. accepted1.00
  64. 1079da80-d9...1.00
  65. Active1.00
  66. 2021.00
  67. 06-1.00
  68. 05:0.96
  69. title: "Agentic Persona1.00
  70. Generation with Critique-0.99
  71. Engineering"1.00
  72. label: "Software0.98
  73. 11 Itens0.91
  74. seed: 202606151.00
  75. split: "fit"0.96
  76. ***0.87
  77. Test set:1.00
  78. Refinement: An Industrial1.00
  79. Evaluation"1.00
  80. }0.90
  81. source_id: "abs-1.00
  82. 2606.09637v1"1.00
  83. Run after loop converged to check0.99
  84. 2a4fd057-92...1.00
  85. Active1.00
  86. 06-1.00
  87. 05:0.99
  88. 2021.00
  89. 3 Itens0.97
  90. Code Smells Across Project1.00
  91. title: "Comparing ML-1.00
  92. Specific and General Python1.00
  93. }0.81
  94. Engineering"1.00
  95. label: "Software0.98
  96. 11 Items0.89
  97. seed: 202606151.00
  98. split: "fit"0.99
  99. source_id: "abs-1.00
  100. generalization1.00
  101. Characteristics"1.00
  102. 2606.01882v1"0.99
  103. STOP BURNING TOKENS1.00
• 

  5:41 #12 done26 line(s)

  shot 12·sharpness 1553.7

  1. langfuse0.96
  2. 05 - THE INPUTS0.94
  3. The base prompt is a flat list of available labels0.99
  4. flat_list_categories1.00
  5. #11.00
  6. pc-flat-loop1.00
  7. Probe1.00
  8. Prompt Config Linked Generations Use Prompt0.99
  9. Text Prompt1.00
  10. Classify this paper with a label.1.00
  11. Available labels:1.00
  12. - Artificial Intelligence1.00
  13. - Computation and Language (Natural Language Processing)1.00
  14. - Computer Vision and Pattern Recognition0.99
  15. - Cryptography and Security0.98
  16. - Databases1.00
  17. - Distributed, Parallel, and Cluster Computing1.00
  18. - Information Retrieval0.99
  19. - Networking and Internet Architecture1.00
  20. - Robotics1.00
  21. - Software Engineering0.99
  22. Title: {{title}}0.99
  23. Authors: {{authors}}0.98
  24. Summary: {{summary}}0.98
  25. Return only the label name, nothing else.0.99
  26. STOP BURNING TOKENS0.99
• 

  6:12 #13 done68 line(s)

  shot 13·sharpness 4813.6

  1. 11.00
  2. # Prompt Auto-Improvement Loop1.00
  3. 21.00
  4. 06 - THE Optimization process0.98
  5. 31.00
  6. ## Context1.00
  7. 41.00
  8. You are improving a classification prompt through an automated1.00
  9. The loop is step by step1.00
  10. fit/validate/test loop.1.00
  11. 51.00
  12. The baseline prompt lives in Langfuse prompt management - fetch0.99
  13. it via the CLI at the start.1.00
  14. instructed via task.md0.98
  15. 61.00
  16. You interact with Langfuse exclusively via the Langfuse CLI.1.00
  17. 71.00
  18. Do not touch the test dataset until a stopping condition is met.0.99
  19. 81.00
  20. 90.99
  21. ## Datasets0.99
  22. 101.00
  23. - Fit: https://cloud.langfuse.com/project/0.99
  24. Run the base prompt on fit + validate1.00
  25. cmq74a8mb01yqad0j43ke3ax7/datasets/cmqfems63091rad0cmfg8q8nd/1.00
  26. items1.00
  27. Score accuracy — per item and overall0.97
  28. 111.00
  29. - Validate: https://cloud.langfuse.com/project/0.99
  30. cmq74a8mb01yqad0j43ke3ax7/datasets/cmqfenrra093mad0ctht6cntj/1.00
  31. Error analysis on the fit set — find patterns0.99
  32. items1.00
  33. 121.00
  34. - Test: untouched until a stopping condition is met1.00
  35. 131.00
  36. Propose an update for the biggest error category1.00
  37. 141.00
  38. ## Stopping conditions1.00
  39. 151.00
  40. Stop the loop when either of these is met:0.97
  41. Re-run — accept only if validate improves0.98
  42. 161.00
  43. 1. 92% accuracy on t0.97
  44. validate dataset0.99
  45. 171.00
  46. 2. 15 improvement loops completed1.00
  47. STOP IF 15 runs completed OR 92%1.00
  48. 181.00
  49. 191.00
  50. When a stopping condition is met, run the test dataset once and1.00
  51. accuracy reached1.00
  52. report final accuracy across all three splits.1.00
  53. 201.00
  54. Perform final run on test set1.00
  55. 211.00
  56. ## Loop0.99
  57. 221.00
  58. 231.00
  59. ### Step 0 - Fetch the prompt0.96
  60. 241.00
  61. Pull the latest production version0.99
  62. via the CLI. This is your baseline0.99
  63. STOP BURNING TOKENS0.99
  64. 251.00
  65. 261.00
  66. ### Step 1 - Baseline0.96
  67. 271.00
  68. 1. Run the current prompt against t0.98
• 

  6:34 #14 skipped

  shot 14·duplicate of #13

• 

  7:25 #15 skipped

  shot 15·duplicate of #5

• 

  7:55 #16 done37 line(s)

  shot 16·sharpness 1748.0

  1. langfuse0.89
  2. 07 - RESULTS·QUALITY0.97
  3. The loop improves accuracy of the classification1.00
  4. Self-improvement undeniably lifts performance, and, by holding it on a cheaper model, saves cost.0.99
  5. Validation accuracy per prompt iteration — keep/discard decision boundary0.99
  6. Kept1.00
  7. Discarded1.00
  8. 1.01.00
  9. EXacth act-rac0.57
  10. 0.81.00
  11. 0.7801.00
  12. 0.7701.00
  13. 0.8301.00
  14. 0.7801.00
  15. 0.7701.00
  16. 0.790.8300.96
  17. Overall level remains0.97
  18. aroun80%0.94
  19. 0.6801.00
  20. -0.6800.99
  21. 0.61.00
  22. +15% from0.98
  23. 0.41.00
  24. baseline1.00
  25. Test set1.00
  26. 0.21.00
  27. accuracy:1.00
  28. 80.2%1.00
  29. 0.01.00
  30. 21.00
  31. 31.00
  32. 41.00
  33. 51.00
  34. 61.00
  35. 71.00
  36. Prompt version0.99
  37. STOP BURNING TOKENS1.00
• 

  8:14 #17 skipped

  shot 17·duplicate of #16

• 

  8:52 #18 done43 line(s)

  shot 18·sharpness 2097.1

  1. langfuse1.00
  2. 07 - RESULTS · QUALITY0.96
  3. The loop clarified decision boundaries and added1.00
  4. examples1.00
  5. Changes v1 → v40.95
  6. Classify this paper with a label.0.99
  7. Classify this paper with a label.0.99
  8. Added general classification approach0.99
  9. transfor1.00
  10. Reserve Artificial Intelligence for a0.96
  11. Added information on how to decide1.00
  12. "Uses a neural network / LLM / learning" is not by itself a reason0.94
  13. generating, translating.0.96
  14. sunnarizing, dialogue, QA,0.87
  15. between two similar classes1.00
  16. - A faster approximate-mearest-neighbour / vector index for search -» Databases (the0.87
  17. Added examples on specific frequently0.99
  18. mendations for a user's query -> Infarnation0.91
  19. confused labels0.99
  20. Scheduling or running jobs across a Gu cluster - Bistributed, Parallel, and Cluster0.83
  21. sage, even if it retrieves or ranks passages as a step.0.99
  22. Available labels:0.95
  23. - Artificial Intelligence0.98
  24. - Computation and Language (Natural Language Pracessing)0.95
  25. Computer Wision and Pattern Recognition0.93
  26. Available labels:0.96
  27. Artificial Intelligence0.97
  28. - Seftware Engineering0.97
  29. Sunnary: ((suary))0.76
  30. - Distributed, Paratlel, and Cluster Computing0.99
  31. Enformation Retrieval0.90
  32. - Software Engineering0.97
  33. Titte: ((tite))0.81
  34. Authors: €authors?}0.88
  35. Sunnary: (<summary))0.75
  36. - Distributed, Parallel, and Cluster Conputing0.97
  37. - Information Retrieval0.94
  38. - Robotics0.95
  39. Networking and Baternet Architecture0.90
  40. Return only the label nase,0.93
  41. Return only the label name, nothing else.0.97
  42. Improvement loop added rules and examples1.00
  43. STOP BURNING TOKENS0.99
• 

  9:09 #19 skipped

  shot 19·duplicate of #18

• 

  9:26 #20 skipped

  shot 20·duplicate of #18

• 

  9:59 #21 done24 line(s)

  shot 21·sharpness 3214.8

  1. ealangfuse0.92
  2. 07 - RESULTS· QUALITY0.97
  3. The classes and the clear right/wrong give high signal0.99
  4. feedback for the optimizer to improve the setup1.00
  5. Both baselines complete and logged: Fit 68.0%· Validate 68.0%.0.97
  6. Error analysis (fit, 64 errors): The dominant pattern is unambiguous - "Artificial Intelligence" is0.99
  7. the #1 wrong-answer sink (22/64), "Computation and Language (NLP)" #2 (11/64). The biggest0.99
  8. specific confusions are Information Retrieval → AI (8×) and Networking → AI (5×). The bare baseline0.98
  9. has zero guidance, so papers that merely use ML get dumped into the two method-magnet labels0.99
  10. instead of their true application domain.1.00
  11. Loop-1 will address exactly this pattern: add a concise "classify by primary domain/contribution,1.00
  12. not the technique; AI and NLP are over-chosen method labels" rule. Let me update tracking and write0.99
  13. the loop-1 build script.0.99
  14. Clearly quantifiable1.00
  15. Data backed'update'1.00
  16. Enough examples to draw0.98
  17. and reliable failure0.97
  18. +0.90
  19. hypothesis1.00
  20. +0.89
  21. general conclusions from0.98
  22. =0.87
  23. modes1.00
  24. STOP BURNING TOKENS0.98
• 

  10:02 #22 done9 line(s)

  shot 22·sharpness 1845.4

  1. ight/wrong give high signal0.99
  2. to improve the setup1.00
  3. date68.0%.1.00
  4. n is unambiguous - "Artificial Intelligence" is0.99
  5. anguage (NLP)"#2(11/64). The biggest0.98
  6. (8×) and Networking → AI(5×). The bare baseline0.97
  7. dumped into the two method-magnet labels1.00
  8. cise "classify by primary domain/contribution,0.99
  9. dlabels" rule. Let me update tracking and write0.98
• 

  10:04 #23 done9 line(s)

  shot 23·sharpness 1214.3

  1. idate 68.0%.0.89
  2. n is unambiguous - "Artificial Intelligence" is0.98
  3. anguage (NLP)"#2(11/64). The biggest0.98
  4. (8×) and Networking → AI(5×). The bare baseline0.98
  5. dumped into the two method-magnet labels1.00
  6. cise "classify by primary domain/contribution,0.98
  7. labels" rule. Let me update tracking and write1.00
  8. ough examples to draw0.99
  9. eneral conclusions from1.00

Transcript

166 cues· 3,151 words· 17,090 chars

Chapters

1. 0:00 Introduction and the goal of avoiding token waste
2. 0:24 The current trend of "designing loops" vs. prompt engineering
3. 0:47 Why coding (with its clear "compile" target) set a false precedent
4. 1:20 Challenges of defining target functions in non-coding domains
5. 2:30 Case study: The arXiv paper classification experiment
6. 3:30 Setup of the minimal self-optimization loop
7. 5:48 The step-by-step iteration process and stopping criteria
8. 7:00 Results: Achieving a 15% improvement in accuracy
9. 9:53 Deep dive into the reasoning behind the improvement
10. 11:26 Translating binary "high signal" feedback to other applications
11. 13:12 Defining what "good" looks like for your specific domain
12. 15:05 The importance of human-agent collaboration and data review
13. 15:57 Don't review it only with your coding agents, but review it as a human.