Videos 9HbzAWnKbo4

From Signal to PR: Anatomy of a Self-Improving Agent — Jason Lopatecki, Arize

index_state ready data_status ok

AI Engineer· published 2026-07-24· 0:20:35· en-US· indexed 2026-08-10 19:40

Open on YouTube

68 shot(s).

keyframes kept every frame deduplicated

What was stored

cues: 228; whisperx 228
chunks: 36; from 228 cues
keyframes: 48; kept of 68 captured
frames with text: 48; 1,766 lines read
chapters: 12; from the source metadata
keyframe bytes: 7.7 MB; word timings on 228 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage	state	model	started	took
`fetch`	done	—	2026-08-09 22:50	0s
`stt`	done	—	2026-08-09 07:13	22s
`chunk`	done	—	2026-08-09 07:14	0s
`text_embed`	done	—	2026-08-10 19:40	1s
`keyframe`	done	—	2026-08-09 07:14	2m 49s
`ocr`	done	—	2026-08-09 07:17	31s
`frame_embed`	done	—	2026-08-10 19:40	8s

Frames, and what the machine read

0:02 #0 done2 line(s)

shot 0·sharpness 459.9
1. AlEngineer0.96
2. World's Fair1.00
0:03 #1 done2 line(s)

shot 1·sharpness 662.1
1. AlEngineer0.95
2. World's Fair0.99
0:10 #2 done24 line(s)

shot 2·sharpness 2744.8
1. LAB & PLATINUM SPONSORS0.99
2. Amazon AGI Lab0.98
3. ANTHROP\C1.00
4. Google DeepMind1.00
5. MINIMAX0.94
6. OpenAI0.92
7. Akamai1.00
8. arize1.00
9. aws1.00
10. Braintrust bright data0.98
11. B1.00
12. Browserbase1.00
13. docker1.00
14. :neo4j0.92
15. ORACLE1.00
16. PayPal1.00
17. qodo1.00
18. reducto1.00
19. Sonar1.00
20. Makers of1.00
21. togetherai1.00
22. Unblocked1.00
23. WorkOS1.00
24. SonarQube1.00
0:28 #3 done3 line(s)

shot 3·sharpness 208.6
1. arize0.85
2. AlEngineer0.99
3. World's Fair1.00
0:57 #4 done2 line(s)

shot 4·sharpness 216.6
1. AlEngineer0.99
2. World's Fair1.00
1:21 #5 done10 line(s)

shot 5·sharpness 1471.9
1. AlEngineer0.98
2. World's Fair0.97
3. arize1.00
4. We make agents work1.00
5. PRESENTED BY1.00
6. From Signal to PR0.98
7. Microsoft1.00
8. Observability & Evaluation1.00
9. World's Fair0.98
10. Engineering the future of Al0.99
1:46 #6 done13 line(s)

shot 6·sharpness 1423.3
1. AlEngineer0.97
2. World'sFair1.00
3. 02:14 - ERROR RATE SPIKING0.99
4. PRESENTED BY1.00
5. It's 2am, the platform is down and0.99
6. Microsoft1.00
7. the fix is still hours away.1.00
8. A human still has to wake up, find the trace using1.00
9. and reconstruct what1.00
10. happened before a single line changes.0.98
11. World's Fair0.97
12. TRACK 5· JULY 1, 20260.95
13. Evals1.00
1:48 #7 done24 line(s)

shot 7·sharpness 1166.6
1. AlEngineer0.98
2. WHY NOW1.00
3. World's Fair0.96
4. Who consumes your telemetry?0.99
5. Phase 1· Software 1.00.93
6. Phase 2 · Software 2.00.98
7. Phase 3·Autonomous1.00
8. PRESENTED BY1.00
9. Application1.00
10. Application1.00
11. Application1.00
12. Microsoft1.00
13. Observability0.93
14. Observability0.98
15. Observability1.00
16. Human dev1.00
17. + agent0.94
18. Coding agent1.00
19. The human is the only reader.0.98
20. Human reads, prompts an agent to write.1.00
21. The agent reads it directly.1.00
22. Worild's Fair0.93
23. TRACK 5· JULY 1, 20260.95
24. Evals1.00
1:50 #8 skipped

shot 8·duplicate of #6
1:59 #9 skipped

shot 9·duplicate of #7
2:41 #10 done22 line(s)

shot 10·sharpness 1044.7
1. AlEngineer0.98
2. WHY NOW1.00
3. World's Fair0.99
4. Who consumes your telemetry?1.00
5. Phase 1· Software 1.00.94
6. Phase 2 · Software 2.00.97
7. Phase 3·Autonomous0.99
8. Application1.00
9. Application1.00
10. Application1.00
11. Observability1.00
12. Observability0.96
13. Observability1.00
14. Human dev1.00
15. + agent0.95
16. Coding agent1.00
17. The human is the only reader.0.98
18. Human reads, prompts an agent to write.1.00
19. The agent reads it directly.1.00
20. World's F0.94
21. TRACK 5· JULY 1, 20260.95
22. Evals1.00
3:04 #11 skipped

shot 11·duplicate of #10
3:35 #12 done11 line(s)

shot 12·sharpness 919.0
1. AlEngineer0.97
2. World'sFair1.00
3. THE GAP1.00
4. You can now build at agent speed0.98
5. Coding agents· minutes0.97
6. You still can't improve at agent speed0.99
7. Humans reading dashboards· days0.97
8. arize0.96
9. World'sFair0.99
10. TRACK 5· JULY 1, 20260.96
11. Evals1.00
3:58 #13 done14 line(s)

shot 13·sharpness 1088.4
1. AlEngineer0.97
2. World'sFair1.00
3. The bottleneck is not the fix. It's building1.00
4. the evidence and confidence in the fix.1.00
5. Finding the traces, reading the evals, digesting the data and forming a hypothesis.0.99
6. Find the traces1.00
7. Read the evals0.99
8. Digest the data1.00
9. Form a hypothesis0.97
10. Localize the bug1.00
11. Change one line1.00
12. World'sFa0.96
13. TRACK 5· JULY 1,20260.94
14. Evals1.00
4:28 #14 done16 line(s)

shot 14·sharpness 741.2
1. AlEngineer0.99
2. THE MOVE1.00
3. World'sFair1.00
4. Invert the loop1.00
5. before1.00
6. Evidence1.00
7. Human debugs1.00
8. Agent writes fix0.99
9. Ship1.00
10. after1.00
11. Evidence1.00
12. Agent investigates & writes fix1.00
13. Human reviews1.00
14. World'sF1.00
15. TRACK 5· JULY 1, 20260.95
16. Evals1.00
4:44 #15 skipped

shot 15·duplicate of #14
5:24 #16 done20 line(s)

shot 16·sharpness 1261.8
1. AlEngineer0.99
2. World's Fair0.98
3. THE ANATOMY1.00
4. What a self-improving agent is made of0.99
5. 011.00
6. what happened1.00
7. 021.00
8. enough to fix it1.00
9. 031.00
10. · when it runs0.97
11. Event Evidence1.00
12. Context + Skills1.00
13. Trigger1.00
14. Traces + evals0.99
15. Logs, APM + the repo1.00
16. Periodic / on error0.97
17. Evidence + context is everything the agent needs to fix it. A trigger decides when it runs.0.99
18. World's Fai0.94
19. TRACK 5· JULY 1,20260.95
20. Evals1.00
5:57 #17 skipped

shot 17·duplicate of #16
6:12 #18 done21 line(s)

shot 18·sharpness 1131.7
1. AlEngineer0.99
2. Evidence1.00
3. Context1.00
4. Trigger1.00
5. World'sFair0.99
6. PRESENTED BY1.00
7. ANATOMY·010.98
8. Traces1.00
9. Microsoft1.00
10. Evidence1.00
11. Every span, input, output, latency, and error the full execution path.1.00
12. Traces are what happened. Evals are whether it0.98
13. was good. Together they're the ground truth an1.00
14. Evals1.00
15. agent reasons from the same evidence you'd1.00
16. LLM-as-a-judge and code checks that turn raw runs into pass / fail1.00
17. scores.1.00
18. open first.1.00
19. World'sF0.93
20. TRACK 5· JULY 1, 20260.95
21. Evals1.00
6:46 #19 done21 line(s)

shot 19·sharpness 973.6
1. AlEngineer0.98
2. Evidence1.00
3. Context1.00
4. Trigger1.00
5. World's Fair0.96
6. ANATOMY· 020.94
7. Context1.00
8. Evidence says something's wrong. The context plus the skills help you fix.1.00
9. PRESENTED BY1.00
10. Microsoft1.00
11. Traces + evals0.99
12. Logs & APM0.97
13. The repo1.00
14. Enough to fix it1.00
15. the evidence1.00
16. embedded agents· Datadog0.98
17. where the fix lands1.00
18. pinpoints the exact line0.99
19. World's Fain0.91
20. TRACK 5· JULY 1,20260.96
21. Evals1.00
6:57 #20 done21 line(s)

shot 20·sharpness 966.6
1. AlEngineer0.97
2. Evidence1.00
3. Context1.00
4. Trigger1.00
5. World's Fair0.99
6. ANATOMY· 020.91
7. Context1.00
8. Evidence says something's wrong. The context plus the skills help you fix.1.00
9. PRESENTED BY1.00
10. Microsoft1.00
11. Traces + evals0.99
12. Logs & APM0.97
13. The repo1.00
14. Enough to fix it1.00
15. the evidence1.00
16. embedded agents· Datadog0.98
17. where the fix lands0.99
18. pinpoints the exact line1.00
19. Wor0.97
20. TRACK 5· JULY 1,20260.96
21. Evals0.93
7:46 #21 done22 line(s)

shot 21·sharpness 1105.6
1. AlEngineer0.97
2. FROM LOCAL TO LONG-RUNNING1.00
3. World's Fair0.99
4. The same agent you run locally runs on events1.00
5. RUNNING LOCAL1.00
6. SANDBOX1.00
7. ·event-triggered0.96
8. Human driving1.00
9. In the cloud1.00
10. Coding agent1.00
11. on events1.00
12. same agent, running unattended0.98
13. Coding agent1.00
14. Arize1.00
15. Pyroscope0.96
16. Gcloud Logs1.00
17. Arize1.00
18. Pyroscope1.00
19. Gcloud Logs0.99
20. World's Fair0.94
21. TRACK 5· JULY 1, 20260.95
22. Evals1.00
8:00 #22 done17 line(s)

shot 22·sharpness 895.5
1. AlEngineer0.98
2. Evidence1.00
3. Context1.00
4. Trigger1.00
5. World'sFair1.00
6. ANATOMY· 030.93
7. Trigger1.00
8. periodic1.00
9. event-driven1.00
10. On a schedule0.98
11. On every errored trace1.00
12. Sweep recent traces and evals on any cadence nightly, hourly, or near1.00
13. The moment a span fails, the loop kicks off.0.99
14. real-time. Batch up what's worth fixing.0.99
15. Work's Fair0.92
16. TRACK 5· JULY 1, 20260.96
17. Evals1.00
8:08 #23 done13 line(s)

shot 23·sharpness 785.5
1. AlEngineer0.97
2. ASSEMBLED1.00
3. World'sFair1.00
4. The loop, assembled0.97
5. Evidence1.00
6. Agent investigates0.98
7. Pull request1.00
8. You review1.00
9. &merge1.00
10. deploy → the next trace feeds the next pass1.00
11. World's Fair0.96
12. TRACK 5· JULY 1, 20260.95
13. Evals1.00

Transcript

228 cues· 3,262 words· 16,875 chars

chunk 0 · 0:12–0:46 · 67 words · 328 chars
0:12 Well, thank you all.
0:15 Let me just get set up here.
0:16 So not just the founder of Verizon, but I tend to build an incredible amount of stuff.
0:25 Let's see if we get this going here.
0:29 Oh, sorry, one more second.
0:34 So not just a founder here, but also a builder.
0:38 And I do my best to,
chunk 1 · 0:42–1:26 · 119 words · 603 chars
0:42 to build agents, assistants.
0:45 We have an agent in product.
0:48 We have an agent in product called Alex, and I think a lot of my experience has come from actually trying to make the stuff work and work well.
0:59 Our first version of our own agent frankly sucked.
1:03 It was many years ago, probably two years ago, we weren't the first in the space to do it.
1:09 And a lot of what we have built has come out of our own experience in building this agent.
chunk 2 · 1:16–1:54 · 114 words · 574 chars
1:16 And Signal is kind of our next generation of this, which is trying to automate a bunch of things, which we do every day, and build it into products that people can use.
1:26 So I'm going to go through this materials here.
1:29 I'll try to go fast and try to show you a lot of product, too.
1:32 I'm a product person.
1:36 if you built a startup before, you've experienced this, your platform's down, it's late at night and you wanna go fix it.
1:45 And really, it takes a lot of energy to go do that.
chunk 3 · 1:49–2:26 · 104 words · 571 chars
1:49 And we're going to talk about the automation we built a little bit and what the future looks like.
1:54 And I truly believe that the future of the observability space is actually changing massively right now.
2:02 Why is that?
2:03 Well, observability used to be for humans.
2:06 It used to be a UI you click, a graph you click, something you look at.
2:11 And today, I would argue it's a lot of 2.0, which is like this combination of coding agent.
2:17 So those of you who've built skills, skills for PyroScope, Google Cloud, or whatnot, these skills help you with your human debugging these systems.
chunk 4 · 2:27–3:12 · 108 words · 570 chars
2:27 And really, telemetry is like this smoke thrown off of your system that can allow these agents to go make fixes.
2:38 It tells you what path in the code it took.
2:41 Without that, you're guessing, and there's a million paths it could have taken.
2:44 The data thrown off by your system allows you to go use agents to go debug your software.
2:55 Evals add another layer to this, but really what we're at here is how do I build systems that autonomously fix themselves?
chunk 5 · 3:05–3:45 · 118 words · 646 chars
3:05 Really, that is what we're after, both AI agents.
3:08 I put AI into my system.
3:10 How do I have this thing just improve itself?
3:13 And today, we're kind of in the 2.0, which is a human making fixes and reviewing things, but there's a future we're all driving towards and throwing off traces, throwing off logs, throwing off way more than you normally would, and having agents run at this for a continuous loop is where we're going.
3:30 You can build at agent speed, but today,
3:33 you can't improve your systems really at this agent speed.
chunk 6 · 3:36–4:10 · 87 words · 440 chars
3:36 So those of us feel this kind of governor happening within our products.
3:41 And the bottleneck is actually not the fix anymore.
3:46 So those of us who've used these systems and used coding agents with skills, the bottlenecks, a lot of the confidence in, do I have it right?
3:57 A lot of this is about, is this fix the right one to push?
4:03 And so these are kind of the challenges here.
4:06 And then how do you build this loop in a way that just moves faster?
chunk 7 · 4:10–4:54 · 135 words · 668 chars
4:10 And a little bit of the way we've kind of come to do it, and we do it in our system, is we've kind of inverted this loop, which is like a human looks at things and an agent,
4:23 fixes it to a person now can wake up with an idea of the issues based upon the errors occurred in their system.
4:31 So the agent is actually, you know, maybe it's not a fix itself, but it's putting up an issue.
4:36 It's looking at the data before a human even looks at it.
4:40 And what you move from there is kind of humans grabbing tickets to having some amount of evidence
chunk 8 · 4:48–5:28 · 136 words · 724 chars
4:48 some deep evidence relative to whatever you're looking at already sitting in front of you by the time you actually even look at it.
4:55 And human review is kind of one thing, but a lot of times maybe you're driving this little investigation a bit from where it started.

Chapters

0:00 Arize, its agent Alyx, and why v1 sucked
1:36 Observability is changing: from dashboards to telemetry for agents
2:55 The goal: systems that fix themselves
4:14 Inverting the loop: the agent investigates first
6:08 Traces on a filesystem, the key unlock
7:23 From your laptop to sandboxes
8:13 A real fix: the Alyx stream canceled bug
9:39 Why you should trace ten times more
11:10 Product demo: Signal, AX, and Phoenix
13:09 Sandboxes, VPC, and why customers won't call out
16:20 Q&A: why not just point Claude Code at your data?
18:04 Q&A: where do the evals come in?