Videos 2e9ANoOEn28
What if the harness mattered more than the model? - Aditya Bhargava, Etsy
Scene timeline
187 shot(s).
keyframes kept every frame deduplicated
What was stored
- cues
- 318
- whisperx 318
- chunks
- 56
- from 318 cues
- keyframes
- 74
- kept of 187 captured
- frames with text
- 74
- 1,239 lines read
- chapters
- 0
- from the source metadata
- keyframe bytes
- 19.1 MB
- word timings on 318 cues
Provenance
| stage | state | model | started | took |
|---|---|---|---|---|
fetch |
done | — | 2026-08-11 04:51 | 1m 32s |
stt |
done | — | 2026-08-11 04:53 | 31s |
chunk |
done | — | 2026-08-11 04:53 | 0s |
text_embed |
done | — | 2026-08-11 04:53 | 1s |
keyframe |
done | — | 2026-08-11 04:53 | 2m 36s |
ocr |
done | — | 2026-08-11 04:56 | 43s |
frame_embed |
done | — | 2026-08-11 04:57 | 10s |
Frames, and what the machine read
-
- slides0.99
- -bash1.00
- 110.61
- slides0.99
- N20.61
- -bash1.00
- 1630.68
- About me1.00
- • Staff engineer at Etsy0.95
- Also Etsy's ICInitiative Lead for Agentic Commerce0.99
- Wrote GrokkingAlgorithms, an illustrated book on algorithms1.00
- Blog at ducktyped.org0.97
- Aditya Bhargava What if the harness mattered more than the model?0.99
-
- , that you can keep the harness simple. Just give the model a1.00
- reliant on fancy proprietary models0.98
- open source models that can be run locally?0.99
- Y0.78
-
- roprietarymodels1.00
- that can be run locally?0.98
- hat we get the performance of a cutting edge model through0.99
- X0.59
-
- slides0.99
- -bash1.00
- 1010.55
- slides0.91
- 320.77
- -bash1.00
- 2e30.62
- Rest of this talk: Lets build a coding agent in Agency0.99
- We'll see the agent evolve over 7 examples0.99
- ● Same model, same task, different harness each time0.98
- We'll see the harness improve the agent's performance over time0.99
- Goals1.00
- Understand thebasics of building a harness1.00
- ● Learn about agent safety0.95
- Aditya Bhargava What if the harness mattered more than the model?0.98
-
- slides1.00
- -bash1.00
- 110.58
- slides1.00
- 320.82
- -bash1.00
- 230.70
- Rest of this talk: Lets build a coding agent in Agency1.00
- We'll see the agent evolve over 7 examples0.99
- ● Same model, same task, different harness each time0.98
- We'll see the harness improve the agent's performance over time0.99
- Goals1.00
- Understand the basics of building a harness0.99
- Learn about agent safety0.98
- Aditya Bhargava What if the harness mattered more than the model?0.99
-
- Same model, same task, different harness each time0.99
- We'll see the harness improve the agent's performanc1.00
- Goals1.00
- Understand the basics of building a harness1.00
- Learn about agent safety0.99
- Intro to more advanced concepts like subagents and s0.99
-
- ne bug:0.98
- demo/median.py1.00
- def median(numbers):0.99
- ordered = sorted(numbers)0.98
- middle = len(ordered)// 20.95
- returnordered[middle]1.00
- 1 # wrongfor even-length lists!0.96
- demo/test_median.py1.00
- fails:1.00
- AssertionError: median() returned 3, expected 2.50.98
- ame model, same task for every example. Only the harness chang1.00
Transcript
318 cues· 4,221 words· 22,314 chars
- 0:01 Hi, everyone.
- 0:02 My name is Audit, pronounced Audit like a text, Audit.
- 0:06 And I'm here to talk to you about what if the harness mattered more than the model?
- 0:11 A little bit about me first.
- 0:13 I'm a staff engineer at Etsy.
- 0:15 I'm also Etsy's IC Initiative Lead for Agentic Commerce.
- 0:18 I wrote Crocking Algorithms, which is an illustrated book on algorithms.
- 0:22 And I also do illustrated posts on DuckTyped.org.
- 0:29 And I want to talk to you about what if the harness mattered more than the model, by which I mean, what if we should be paying more attention to building expertise in harnesses?
- 0:45 And my controversial take here is I hear a lot of people say, oh, the models are so good that you can just keep the harness simple, just give the model a few tools and it'll do the rest.
- 0:57 And I've been hearing this so often that it has now started becoming the wisdom of the tech industry.
- 1:07 To which my answer is yes, but that's moving in the wrong direction because that is making us reliant on fancy proprietary models that can't be run locally.
- 1:20 So what if we instead talked about the other direction?
- 1:23 What if we focused on open source models that can be run locally?
- 1:28 What if we focused on building a harness that is so good that we can get the performance of a cutting edge model through a local open source model?
- 1:37 So what if the harness mattered more than the model?
- 1:41 What if our focus should be on the harness?
- 1:45 How much does the harness matter?
- 1:47 This is something I have been researching a lot in my spare time.
- 1:53 There's a really interesting paper that talks about harness bench, which is a benchmark for harnesses, which is something we haven't seen a lot of yet.
- 2:04 It's got 106 tasks.
- 2:07 It's testing these different models and different harnesses.
- 2:10 But the point is that it's running the same setup, it's doing the same evaluation, same model, but testing different harnesses.
- 2:22 And the results are pretty interesting.
- 2:25 So scores range from 52.4% to 76.2%.
- 2:27 So more than a 20-point difference.
- 2:33 and only the harness changed.
- 2:37 And the really interesting thing is that for weaker models, the harness matters more.
- 2:44 If the harness matters more than the model, that's great news.
- 2:48 Because if the model matters more, then we're dependent on a handful of companies who are able to build and train these models.
- 2:58 But if a good harness can compensate and can make a weaker model perform better, then we can build our own harnesses.
- 3:07 And that's something that any of us can do.
- 3:09 And we don't have to depend on paid models.
- 3:14 So I have been spending a lot of time and doing a lot of research specifically into what would it take to build a really good harness.
- 3:25 And
- 3:29 I realized early on that none of the existing tools or frameworks were able to do what I wanted.
- 3:37 And I realized that what was actually needed was a language level solution.
- 3:43 So this is my other controversial take in this talk is I think building a good harness actually requires language level support.
- 3:53 And this is kind of where my research has gone in the last six months, and I've spent a good part of that six months building a new language called Agency, which is a language for building agents.
- 4:12 and just to talk about the terminology there real quick because the terms agent and harness aren't super crisp in the industry yet they're not super well defined um but in this talk
- 4:26 I'll kind of use them interchangeably.
- 4:28 But when I talk about the agent, I really mean the harness and the model.
- 4:33 So if you take Cloud Code, for example, Cloud Code is an agent.
- 4:38 It uses Opus, which is the model, and then everything else is the harness.
- 4:45 But when we talk about agency, it's really for building agents, but the thing it's providing is the harness.
- 4:55 um agency is really good uh you know it's still new building a language is a lot of work and it's only six months in uh but i love this quote by alan k simple things should be simple complex things should be possible
- 5:13 That's something I love about agency is that I think when people build agents, there are a lot of primitives, a lot of things that every agent will need to be able to do, and those things should be simple to do.
- 5:33 And there's a lot of complex things that we want to be able to do with agents.
- 5:38 And it should be possible to do those things in the framework of your choice.
- 5:43 And with agency, it is possible.
loading