The State of Model Routing — NVIDIA, Cognition, OpenRouter

1. chunk 0 · 0:12–0:53 · 149 words · 821 chars
2. 0:12 Those have been really exciting.
3. 0:13 We've tried to get a bunch of the industry leaders together to talk about some of the problems that we're facing as we try to run more on local.
4. 0:21 If you guys were here for the first panel, one of the things that we talked about was model routing.
5. 0:26 We firmly believe that we're in a multi-model world.
6. 0:29 I think you heard this from many of the panelists.
7. 0:32 Anyone who is deploying AI in production and who is doing so locally is seeing that multi-model world.
8. 0:37 That's why we released these Nemotron models at NVIDIA.
9. 0:40 Everything is released from the data sets to the weights with recipes so that you can customize them.
10. chunk 1 · 0:44–1:28 · 148 words · 831 chars
11. 0:44 We do that because we know that people customizing models is going to be huge.
12. 0:49 And so this panel is really exciting because we're going to talk specifically about model routing.
13. 0:54 So as you are picking which model to use, how does that how essentially how does that tooling itself look?
14. 1:00 Do you guys want to introduce yourselves?
15. 1:03 Yeah, sure.
16. 1:04 I'm Walden.
17. 1:04 I'm the co-founder of Cognition.
18. 1:06 We build Devon, a software engineer.
19. 1:08 In addition to the product, we spend a lot of time partnering with our customers to figure out how they should deploy these models and these agents.
20. chunk 2 · 1:15–1:48 · 114 words · 640 chars
21. 1:15 And one of the things they're constantly asking us nowadays is basically, how do I know the ROI of our models?
22. 1:20 And how do I know which tasks I can actually let our engineers spend the most expensive models on versus letting them use a more cost-efficient model?
23. 1:29 And so that's why we're also thinking a lot more about multi-model routing nowadays.
24. 1:32 Totally.
25. 1:33 I'm Carter.
26. 1:34 You guys heard from me a little bit earlier, but if you weren't here, I'm a developer tech engineer at Nvidia and ultimately I spent a lot of time thinking about how to get intelligence into as many developers hands as possible and something that is continually becoming a
27. chunk 3 · 1:49–2:31 · 118 words · 658 chars
28. 1:49 not an issue, but something that is top of mind for a lot of developers is as you use more intelligence and the frontier models get more expensive, it becomes somewhat cost prohibitive to use the best tools, what feels like the best tools, as much as you would like to use them.
29. 2:04 And so this has become a recent focus is how can we still get the same desired outputs, but actually both as an individual developer, but also imagine startups and small companies
30. 2:16 How can you leverage this incredible tool without totally breaking the bank?
31. chunk 4 · 2:21–3:05 · 126 words · 691 chars
32. 2:21 I'm Tane.
33. 2:22 I work on model evaluations, both in terms of its accuracies and efficiency and cost understanding of the model.
34. 2:31 And then I try and understand those, implement those learnings and help build a router.
35. 2:37 So it's basically...
36. 2:40 My job is to understand the behavior of the model on an intimate level and then use those learnings to both improve the model and try and design a system of model that can work together with each other.
37. chunk 5 · 2:52–3:28 · 134 words · 729 chars
38. 2:52 Totally.
39. 2:54 Yeah, I love a lot of the research that you're doing at NVIDIA as we kind of see the space through.
40. 2:58 I think what's really interesting is model routing itself is pretty new still.
41. 3:01 And so what you'll notice is there isn't a very clear solution here.
42. 3:05 That was something that came up on the first panel is that there is a lot of space for startups and for companies in the ecosystem to fill in a solution here because we're still figuring out how to best do these patterns.
43. 3:15 And I think, Walden, I want to kind of ask you.
44. 3:17 So Cognition just released Fusion, your guys' model router.
45. 3:21 And when you guys released it, you, in your blog, said that you're actually getting better performance than Fable, than these frontier models.
46. chunk 6 · 3:28–4:07 · 106 words · 663 chars
47. 3:28 And I feel like that was a very surprising statement to hear, because we're thinking that you're getting as good or close enough, usually, when we're running on Edge, when we're running local, in these compute-strained, smaller footprint models.
48. 3:40 But you guys are getting better.
49. 3:41 Can you explain how?
50. 3:43 Yeah, absolutely.
51. 3:44 I also want to be clear about something here.
52. 3:48 We're not saying that we gap above Fable-level performance in the same way that maybe Fable-level performance gaps above other models.
53. 3:54 I think actually there's this really unintuitive dynamic where smarter models actually get better and better at delegating work.
54. chunk 7 · 4:05–4:46 · 154 words · 823 chars
55. 4:05 we had with building model routers.
56. 4:07 We don't want to route people to a dumber model and then suddenly you're stuck with a model that doesn't know how to do your task and next thing you know you're switching yourself back to a smarter model anyways and now taking that expensive cost.
57. 4:18 In general we think a lot of the existing model routing systems out there are probably the same ones people have been using like a year ago and so we really wanted to put out a new framework that actually lets people
58. 4:28 still feel like and still have a frontier model in their system while getting all these cost benefits.

loading