Learning on the Job: The Future of Post-Training — Raymond Feng, Applied Compute

1. chunk 0 · 0:12–0:44 · 79 words · 466 chars
2. 0:12 Yeah, thank you, Jack.
3. 0:14 Really grateful for the opportunity to speak here.
4. 0:16 Today I'm gonna be sharing some of our frontier work on post-training and how we envision a future where agents can learn new skills on the job.
5. 0:27 So over the last year or so, we've seen agents develop really strong reasoning skills
6. 0:34 And they've learned to use agentic harnesses to solve longer and longer horizon tasks, which involve many turns and tool calls on complicated environment states.
7. chunk 1 · 0:46–1:23 · 87 words · 463 chars
8. 0:46 We're seeing an increasing demand for agents that can just be deployed in a plug and play way into how enterprises use the agents.
9. 0:59 So for instance, if they already have some method of
10. 1:02 calling the agent to do a task, they would want to be able to train a custom model to do that task instead.
11. 1:11 And that requires new ways of looking at post-training that allow you to adapt to any harness, including ones that you don't necessarily have access to the source code of.
12. chunk 2 · 1:25–2:08 · 109 words · 585 chars
13. 1:25 So I wanted to talk about a few different levels of post-training where each one builds on top of the last.
14. 1:34 One way that we kind of think of this is a framework comparing it to how humans do learning where you learn simple tasks first and you can sort of compound your understanding to more and more complicated tasks.
15. 1:48 So
16. 1:49 Over the last year, we've sort of, I would say, mastered or gotten a lot of reps with these simple single turn Q&A tasks and some longer horizon synthetic environment tasks.
17. chunk 3 · 2:02–2:45 · 121 words · 657 chars
18. 2:02 But what we're increasingly seeing is we want to be able to adapt to custom harnesses and
19. 2:09 be able to train directly on those instead.
20. 2:12 And we kind of think of those kind of like internships where you want the model to do a specific task, but you don't necessarily know how exactly the task will play out because you don't own the harness.
21. 2:26 And then finally, I would want to share some visions we have for the future of the custom model training space, where we think that there will be these kind of agentic citizens, which you can just deploy once, and they'll be able to adapt to many different types of distribution tasks and learn from their interactions.
22. chunk 4 · 2:47–3:31 · 133 words · 707 chars
23. 2:47 So first, I just want to talk about the training setup for these simple Q&A tasks.
24. 2:54 We have something that looks like this where you have an orchestrator, and the orchestrator is in charge of driving the rollouts.
25. 3:02 The orchestrator holds a task spec which you can think of for now as just the simple prompt and answer, so something like a math question and a corresponding numerical answer.
26. 3:13 The orchestrator will send this prompt to a model and then get an answer back.
27. chunk 5 · 3:17–3:55 · 114 words · 614 chars
28. 3:17 Then it will send the answer to a grader and have it be graded.
29. 3:20 So once all of this is done, we want to improve our model based on that interaction or maybe like a batch of interactions.
30. 3:28 And the way we do that is through a training engine.
31. 3:31 which takes in the graded chats and produces a wait update.
32. 3:35 That wait update is then synced to some inference engines, and once those inference engines are updated, then we can start this entire process over again, where the orchestrator will have new problems to send to the model completion endpoint, and then you'll be able to get more chats, grade them, and train again.
33. chunk 6 · 3:56–4:40 · 126 words · 688 chars
34. 3:56 The key thing to note here is that the only thing you need for improving your model is the graded chats in some format.
35. 4:04 And once you have those, the training engine can compute wait updates to improve your model.
36. 4:11 What's important here is that the chats are in a very specific format because we're sort of constraining everything to be inside of our training stack.
37. 4:19 So in this simple setup for Q&A, you don't have anything living outside of the training stack.
38. 4:24 You basically have the code of how to run the rollout and how everything is formatted.
39. chunk 7 · 4:30–5:14 · 114 words · 627 chars
40. 4:30 So it's like in a very controlled environment.
41. 4:34 However, this is kind of limited because we can only kind of do single turn tasks in this way.
42. 4:40 If we want to do longer and long horizon tasks and we want to build like higher order skills into our models, we need to also increase the complexity of our environment.
43. 4:51 So with synthetic environments, we have a very similar setup, but we offload a lot of the environment state outside of the training stack.
44. chunk 8 · 5:00–5:42 · 118 words · 644 chars
45. 5:00 So you still have the same orchestrator from before, but
46. 5:07 The task spec is maybe a little bit more complicated and the environment state is living outside of the training stack.
47. 5:14 So the task spec might now include things like tool cost specs or like maybe an initial state for your environment like a file system.
48. 5:23 And the orchestrator is now in charge of running many turns in series where maybe first it asks the model for how it wants to respond.
49. chunk 9 · 5:32–6:15 · 142 words · 734 chars
50. 5:32 And then if the model wants to call some tools, it will then call the sandbox to actually modify the environment state or read the environment state and then return those results back to the model.
51. 5:43 After all that is said and done, you get a full task trace out of this.
52. 5:48 And that task trace is then sent to a grader for grading.
53. 5:51 And very similar to what we had before, you'll be able to take the graded chats.
54. 5:56 You'll be able to then use them to do a weight update.
55. 6:01 The main thing to highlight here is that this orchestrator and sandbox setup is replayable, which is basically just saying that for any specific prompt, you can always roll back to the initial state and rerun it.
56. chunk 10 · 6:13–6:45 · 88 words · 486 chars
57. 6:13 You can do that in parallel or you can do that in series.
58. 6:16 But the reason that's important is because
59. 6:19 the main sort of method that we use for reinforcement learning today is grpo and that involves comparing many rollouts for the same prompt and then comparing like relatively which one is better than the other and the training engine will then up like make an edit to the model to upweight the trajectories that were more successful and then down with the ones that were less successful
60. chunk 11 · 6:48–7:27 · 99 words · 571 chars
61. 6:48 So some challenges that we face in this setup is that the environment is something that you want to basically use to replicate reality so that after you're done training, the improvements that you've seen actually translate to when you deploy these models into production.
62. 7:05 And the main sort of problem is

loading