Harnesses in AI: A Deep Dive — Tejas Kumar, IBM

What was stored

cues

410

whisperx 410

chunks

37

from 410 cues

keyframes

43

kept of 79 captured

frames with text

43

1,258 lines read

chapters

13

from the source metadata

keyframe bytes

8.8 MB

word timings on 410 cues

Provenance

Frames, and what the machine read

• 

  0:03 #0 done2 line(s)

  shot 0·sharpness 658.4

  1. AlEngineer0.98
  2. EUROPE1.00
• 

  0:08 #1 done2 line(s)

  shot 1·sharpness 827.9

  1. PRESENTINGSPONSOR1.00
  2. Google DeepMind1.00
• 

  0:13 #2 done3 line(s)

  shot 2·sharpness 907.7

  1. PLATINUM SPONSORS0.98
  2. # Braintrust0.96
  3. WorkOS OpenAI0.95
• 

  0:37 #3 done6 line(s)

  shot 3·sharpness 299.0

  1. X @tejaskun0.93
  2. Spotity0.95
  3. Teja0.99
  4. Like “conta0.98
  5. AlEngineer0.99
  6. EUROPE1.00
• 

  1:01 #4 done9 line(s)

  shot 4·sharpness 1126.0

  1. X @tejaskumar_0.98
  2. AIE1.00
  3. AI Harnesses0.96
  4. ★1.00
  5. ★1.00
  6. From first principles1.00
  7. AlEngineer1.00
  8. Engineering the future of Al1.00
  9. 20260.91
• 

  1:19 #5 skipped

  shot 5·duplicate of #4

• 

  1:43 #6 done11 line(s)

  shot 6·sharpness 1117.9

  1. X @tejaskumar_0.96
  2. AIE1.00
  3. AI Harnesses1.00
  4. ★1.00
  5. ★1.00
  6. ★1.00
  7. From first principles0.98
  8. AlEngineer0.97
  9. AlEngineer0.96
  10. 20260.88
  11. EUROPE1.00
• 

  2:36 #7 done9 line(s)

  shot 7·sharpness 988.6

  1. X @tejaskumar_0.98
  2. AIE1.00
  3. Why Harness?1.00
  4. ★1.00
  5. ★1.00
  6. AlEngineer1.00
  7. AlEngineer0.97
  8. 20260.90
  9. EUROPE1.00
• 

  3:06 #8 skipped

  shot 8·duplicate of #4

• 

  3:15 #9 done5 line(s)

  shot 9·sharpness 869.8

  1. ***0.52
  2. AIE1.00
  3. Engineering the future of Al1.00
  4. AlEngineer0.99
  5. 20260.88
• 

  3:40 #10 done2 line(s)

  shot 10·sharpness 279.8

  1. AlEngineer0.98
  2. EUROPE1.00
• 

  3:59 #11 done15 line(s)

  shot 11·sharpness 1874.6

  1. X @tejaskumar_0.98
  2. AIE1.00
  3. ★1.00
  4. What Harness?1.00
  5. ★1.00
  6. ★1.00
  7. Eval Harness (ML Engineering)1.00
  8. Agent Harness (AI Engineering)1.00
  9. What1.00
  10. Eval Harness (ML Engineer0.99
  11. Agent Harness (AI Enginee1.00
  12. AlEngineer0.98
  13. AlEngineer0.97
  14. 20260.95
  15. EUROPE1.00
• 

  4:47 #12 done11 line(s)

  shot 12·sharpness 1261.6

  1. X @tejaskumar_0.98
  2. Agent Harness1.00
  3. AIE1.00
  4. • Tool Registry0.96
  5. ★1.00
  6. ★1.00
  7. Ager1.00
  8. • Tool Re0.90
  9. AlEngineer0.96
  10. Engineering the future of Al1.00
  11. 20260.92
• 

  5:16 #13 done20 line(s)

  shot 13·sharpness 1968.0

  1. X @tejaskumar_0.97
  2. Agent Harness1.00
  3. *★★0.67
  4. AIE1.00
  5. Tool Registry0.98
  6. ★1.00
  7. ★1.00
  8. Model1.00
  9. Context Management1.00
  10. Guardrails1.00
  11. Agent Loop1.00
  12. Ager1.00
  13. • Tool Re0.93
  14. Model1.00
  15. Context Management1.00
  16. Guardrails1.00
  17. Agent Loop1.00
  18. AlEngineer0.97
  19. Engineering the future of Al1.00
  20. 20260.85
• 

  5:55 #14 done9 line(s)

  shot 14·sharpness 493.4

  1. Agent Harness1.00
  2. Tool Registry1.00
  3. Model1.00
  4. Context Management1.00
  5. Guardrails1.00
  6. Agent Loop1.00
  7. lerify0.90
  8. AlEngineer0.98
  9. EUROPE1.00
• 

  6:24 #15 done3 line(s)

  shot 15·sharpness 278.2

  1. dem1.00
  2. AlEngineer0.99
  3. EUROPE1.00
• 

  6:36 #16 done55 line(s)

  shot 16·sharpness 2182.9

  1. Keynote1.00
  2. File0.99
  3. Edit0.99
  4. Insert1.00
  5. Slide1.00
  6. Format1.00
  7. Arrange1.00
  8. Piay0.90
  9. Window1.00
  10. Help1.00
  11. Thu Apr 9 14 35 480.97
  12. aie-europe0.99
  13. 00.65
  14. 63 %0.77
  15. Add Slide0.94
  16. 用0.51
  17. 80.90
  18. ①0.67
  19. Format0.98
  20. Slide1.00
  21. Blank1.00
  22. Appearance1.00
  23. Title0.93
  24. X etejaskumar0.83
  25. Slide Number0.96
  26. Body1.00
  27. Background1.00
  28. Standard1.00
  29. Dynamic0.97
  30. ★1.00
  31. ★1.00
  32. ★0.89
  33. AIE1.00
  34. ★1.00
  35. ★1.00
  36. ★1.00
  37. ★1.00
  38. ★1.00
  39. demo1.00
  40. Preview Motion0.99
  41. Colors0.96
  42. Speed1.00
  43. 4,480.97
  44. Height0.96
  45. Peaks1.00
  46. 0.10.94
  47. Slack1.00
  48. $I0.67
  49. 米0.99
  50. A0.99
  51. Edt Slide Layouf0.97
  52. AlEngineer0.97
  53. AlEngineer0.96
  54. 0260.86
  55. EUROPE1.00
• 

  6:45 #17 done68 line(s)

  shot 17·sharpness 1961.7

  1. Run1.00
  2. Window Help0.98
  3. harness0.96
  4. 7-index.ts ×0.96
  5. 日0.50
  6. agent ) Ts 7-Index.ts .0.84
  7. import { createTools ) from "./1-tools.js";0.97
  8. import { createContext } from "./3-context.js";0.98
  9. import { runLoop } from "./5-loop.js";0.97
  10. import { BrowserSession ) from "./browser.js";0.96
  11. // try a shitty model1.00
  12. const MODEL = "openai/gpt-3.5-turbo-0613*;0.97
  13. 81.00
  14. const TASK =0.99
  15. 101.00
  16. Upvote a story on Hacker News.1.00
  17. ★1.00
  18. 111.00
  19. AIE1.00
  20. ★1.00
  21. 121.00
  22. 131.00
  23. Go to https://news.ycombinator.com.0.99
  24. Call browser_get_stories to see ranked stories with their IDs and voted status.0.99
  25. ★1.00
  26. 151.00
  27. 141.00
  28. Find the highest-ranked story where alreadyVoted is false.1.00
  29. Click its upvote arrow using the exact selector: a[id="up_STORYID"] (replace STORYID with the actual id).0.99
  30. ★1.00
  31. ★1.00
  32. ★1.00
  33. 161.00
  34. .trim();0.99
  35. 171.00
  36. 181.00
  37. console.log( Model: $(MODEL)');0.95
  38. 191.00
  39. console.log('Task: upvote on Hacker News\n');0.99
  40. 20 210.85
  41. const session = new BrowserSession();0.98
  42. 221.00
  43. 231.00
  44. try0.95
  45. 241.00
  46. await session.open();0.97
  47. 251.00
  48. 261.00
  49. const tools = createTools(session);1.00
  50. 271.00
  51. const messages = createContext(TASK);1.00
  52. 281.00
  53. const result = await runLoop(MODEL, messages, tools);0.99
  54. 291.00
  55. 301.00
  56. console.log('\nAnswer: $(result.answer)');0.96
  57. 311.00
  58. console.log('Stopped by: $(result.stoppedBy));0.99
  59. 321.00
  60. console.log('Iterations: $(result.iterations)');0.97
  61. 331.00
  62. finally{0.93
  63. 341.00
  64. await session.close();0.99
  65. 0 harmess 1 0 Live Share0.86
  66. Engineering the future of Al0.99
  67. AlEngineer1.00
  68. 20260.98
• 

  6:53 #18 done77 line(s)

  shot 18·sharpness 2853.4

  1. Edit1.00
  2. Selection1.00
  3. GQ0.83
  4. Thu Apr 9 14:36:060.97
  5. □←0.89
  6. harness0.99
  7. Agents Window+0.99
  8. Ts 7-index.ts ×0.96
  9. Select Color Theme (detect system color mode disabled)0.99
  10. ①0.85
  11. agent > TS 7-index.ts > ...0.93
  12. Cursor Light1.00
  13. light themes0.96
  14. 21.00
  15. 11.00
  16. import { createT0.99
  17. import { createC1.00
  18. Light Modern Default Light Modern0.98
  19. Light+ Default Light+1.00
  20. Light (Visual Studio) Visual Studio Light0.99
  21. 31.00
  22. import { runLoop0.98
  23. Quiet Light1.00
  24. 41.00
  25. import { Browser0.96
  26. Solarized Light1.00
  27. 51.00
  28. Abyss1.00
  29. dark themes1.00
  30. 61.00
  31. // try a shitty0.97
  32. Cursor Dark0.96
  33. ★1.00
  34. 71.00
  35. const MODEL = "o0.99
  36. Cursor Dark Midnight1.00
  37. AIE1.00
  38. ★0.87
  39. 81.00
  40. Dark (Visual Studio) Visual Studio Dark1.00
  41. ★1.00
  42. 91.00
  43. const TASK =0.97
  44. Dark Modern Default Dark Modern1.00
  45. ★1.00
  46. ★1.00
  47. ★1.00
  48. 100.83
  49. Upvote a story on Hacker News.1.00
  50. 110.97
  51. 121.00
  52. Go to https://news.ycombinator.com.1.00
  53. 131.00
  54. Call browser_get_stories to see ranked stories with their IDs and voted status.0.99
  55. 141.00
  56. Find the highest-ranked story where alreadyVoted is false.1.00
  57. 151.00
  58. Click its upvote arrow using the exact selector: a[id="up_STORYID"] (replace STORYID with the actual id).0.99
  59. 161.00
  60. .trim();0.99
  61. 171.00
  62. 181.00
  63. console.log(Model: ${MODEL}');0.97
  64. 191.00
  65. console.log('Task: upvote on Hacker News\n`);0.98
  66. 201.00
  67. 211.00
  68. const session = new BrowserSession();1.00
  69. 221.00
  70. ↓90 harness0.91
  71. 10Live Share1.00
  72. Cursor Tab -- Tejas Kumar (1 week ago)0.97
  73. Ln 25, Col 10.98
  74. Spaces: 2 UTF-8 LF () TypeScript Prettier0.95
  75. Engineering the future of Al0.98
  76. AlEngineer1.00
  77. 20260.99
• 

  6:59 #19 skipped

  shot 19·duplicate of #18

• 

  7:23 #20 done62 line(s)

  shot 20·sharpness 2975.0

  1. Edit1.00
  2. Selection1.00
  3. Thu Apr 9 14:36:350.98
  4. □←0.98
  5. harness1.00
  6. Agents Window +0.96
  7. Ts 7-index.ts ×0.96
  8. agent > TS 7-index.ts > [0] TASK0.95
  9. 81.00
  10. 91.00
  11. const TASK =1.00
  12. 101.00
  13. Upvote a story on Hacker News.0.99
  14. 111.00
  15. 121.00
  16. Go to https://news.ycombinator.com.0.99
  17. 131.00
  18. Call browser_get_stories to see ranked stories with their IDs and voted status.1.00
  19. ★1.00
  20. 141.00
  21. Find the highest-ranked story where alreadyVoted is false.1.00
  22. AIE1.00
  23. ★1.00
  24. 151.00
  25. Click its upvote arrow using the exact selector: a[id="up_STORYID"] (replace STORYID with the actual id).0.98
  26. ★1.00
  27. 161.00
  28. .trim();0.98
  29. ★1.00
  30. ★1.00
  31. ★1.00
  32. 171.00
  33. 181.00
  34. console.log(Model: ${MODEL}');0.97
  35. 191.00
  36. console.log('Task: upvote on Hacker News\n');0.97
  37. 201.00
  38. 211.00
  39. const session = new BrowserSession();1.00
  40. 221.00
  41. 231.00
  42. try {0.94
  43. 241.00
  44. await session.open();1.00
  45. 251.00
  46. 261.00
  47. const tools = createTools(session);0.99
  48. 271.00
  49. const messages = createContext(TASK);0.99
  50. 281.00
  51. const result = await runLoop(MODEL, messages, tools);0.99
  52. 290.89
  53. 300.96
  54. conenle loa('lnAnewer: $/reeult anewerl.).0.75
  55. 90 harness 1 0 Live Share0.89
  56. Cursor Tab1.00
  57. -- Tejas Kumar (1 week ago)0.96
  58. Ln 15, Col 40.97
  59. Spaces: 2 UTF-8 LF () TypeScript Prettier0.97
  60. Engineering the future of Al0.98
  61. AlEngineer1.00
  62. 20260.98
• 

  7:48 #21 done64 line(s)

  shot 21·sharpness 3151.4

  1. Edit0.86
  2. Selection View Go Run Terminal Window Help0.98
  3. Thu Apr 9 14 37 000.98
  4. □←0.68
  5. harness1.00
  6. Agents Window +0.96
  7. Ts 7-index.ts0.93
  8. Ts browser.ts ×0.95
  9. agent Ts browser.ts BrowserSession >getText0.93
  10. 41.00
  11. export class BrowserSession {1.00
  12. 231.00
  13. async getText(): Promise<string> {1.00
  14. LT0.61
  15. 251.00
  16. return text.slice(0, 4000);0.98
  17. 261.00
  18. 271.00
  19. ★1.00
  20. 281.00
  21. async fill(selector: string, value: string): Promise<string> {0.99
  22. AIE1.00
  23. ★0.99
  24. 291.00
  25. await this.page!.fill(selector, value);1.00
  26. ★1.00
  27. ★1.00
  28. ★1.00
  29. ★1.00
  30. 301.00
  31. 311.00
  32. 321.00
  33. }0.80
  34. return `Filled "$(selector}";0.96
  35. 331.00
  36. async click(selector: string): Promise<string> {0.99
  37. 341.00
  38. // Capture the id of the element before clicking (navigation may change the page)0.99
  39. 351.00
  40. const elementId = await this.page!.locator(selector).first().getAttribute("id");0.99
  41. 361.00
  42. 371.00
  43. await this.page!.click(selector, { timeout: 10000 });1.00
  44. 381.00
  45. await this.page!.waitForLoadState("domcontentloaded", { timeout: 10000 });0.99
  46. 391.00
  47. 401.00
  48. const clicked = elementId ? `element id="${elementId}"' : '"${selector}"`;0.95
  49. 411.00
  50. return Clicked ${clicked} - now at ${this.page!.url()}`;0.98
  51. 421.00
  52. }0.86
  53. 431.00
  54. 441.00
  55. // Returns a structured list of HN front-page stories so the agent can0.99
  56. 90④0.77
  57. harness 1 0 Live Share0.95
  58. Cursor Tab -o- Tejas Kumar (1 week ago)0.97
  59. Q0.83
  60. Ln 25, Col 21 (5 selected)1.00
  61. Spaces: 2 UTF-8 LF (} TypeScript Prettier0.94
  62. Google DeepMind1.00
  63. AlEngineer1.00
  64. 20260.98
• 

  7:55 #22 skipped

  shot 22·duplicate of #18

• 

  8:21 #23 done74 line(s)

  shot 23·sharpness 2894.1

  1. Edit1.00
  2. Selection View Go Run Terminal Window Help0.99
  3. Thu Apr 9 14 37 340.97
  4. □←0.96
  5. harness1.00
  6. Agents Window +0.97
  7. Ts 7-index.ts0.92
  8. Ts 1-tools.ts ×0.96
  9. agent > Ts 1-tools.ts > createTools > [] tools execute0.92
  10. 191.00
  11. export function createTools(session: BrowserSession, hooks?: ToolHooks): ToolRegistry {0.99
  12. 201.00
  13. const tools: Tool[] =[0.97
  14. 411.00
  15. definition: {1.00
  16. 431.00
  17. function: {0.97
  18. 700.59
  19. WUL LC UnL UI LIC CUTLL NayC. VOU LITO LU UOLOVL0.57
  20. pa0.99
  21. 461.00
  22. parameters: { type: "object", properties: {}, required: [] },0.98
  23. 471.00
  24. 3.0.56
  25. ★1.00
  26. AIE1.00
  27. ★1.00
  28. 481.00
  29. }.0.57
  30. ★1.00
  31. 491.00
  32. 501.00
  33. }.0.70
  34. execute: async () => session.getUrl(),0.99
  35. ★1.00
  36. ★1.00
  37. ★1.00
  38. 511.00
  39. 521.00
  40. 531.00
  41. definition: {0.99
  42. 541.00
  43. type: "function",0.99
  44. 551.00
  45. function: {0.97
  46. 561.00
  47. name: "browser_get_text",0.99
  48. 571.00
  49. description: "Get the visible text content of the current page.",1.00
  50. 581.00
  51. parameters: { type: "object", properties: {}, required: [] },0.98
  52. 591.00
  53. },0.87
  54. 601.00
  55. }.0.57
  56. 611.00
  57. execute: async () => session.getText(),0.99
  58. 621.00
  59. },0.67
  60. 631.00
  61. 90④0.78
  62. harness1.00
  63. ②10Live Share0.93
  64. Cursor Tab1.00
  65. -o- Tejas Kumar (1 week ago)0.98
  66. Ln 37, Col 22 (1 selected)1.00
  67. Spaces: 20.99
  68. UTF-81.00
  69. LF1.00
  70. () TypeScript Prettier0.95
  71. AlEngineer0.98
  72. Braintrust1.00
  73. WorkOS OpenAI0.98
  74. 20260.96

Transcript

410 cues· 3,978 words· 20,664 chars

Chapters

1. 0:00 Introduction to Tejas Kumar and AI Harnesses
2. 1:45 Why we use harnesses: Reliability and control
3. 3:00 Defining an agent harness from first principles
4. 4:32 Key components of an agent harness (Tooling, Context, Guardrails)
5. 5:59 Starting the demo: Building a browser agent
6. 7:00 Inspecting the initial agent loop
7. 8:12 The problem: Agent failure and hallucination
8. 10:20 Adding guardrails and context management
9. 11:54 Refactoring into a formal harness
10. 13:02 Implementing a verify step to catch lies
11. 15:36 Implementing a login handler for programmatic access
12. 17:42 Final demonstration: Successful autonomous upvoting
13. 18:34 Summary and the future of dynamic harnesses