Videos H7puB0RwJMM

Citation Needed: Provenance for LLM-Built Knowledge Graphs — Daniel Chalef, Zep AI

index_state ready data_status ok

AI Engineer· published 2026-07-23· 0:20:54· en-US· indexed 2026-08-10 19:43

Open on YouTube

48 shot(s).

keyframes kept every frame deduplicated

What was stored

cues: 205; whisperx 205
chunks: 35; from 205 cues
keyframes: 13; kept of 48 captured
frames with text: 13; 220 lines read
chapters: 11; from the source metadata
keyframe bytes: 5.4 MB; word timings on 205 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage	state	model	started	took
`fetch`	done	—	2026-08-10 00:06	1m 51s
`stt`	done	—	2026-08-10 00:08	18s
`chunk`	done	—	2026-08-10 00:08	0s
`text_embed`	done	—	2026-08-10 19:43	0s
`keyframe`	done	—	2026-08-10 00:08	1m 59s
`ocr`	done	—	2026-08-10 00:10	5s
`frame_embed`	done	—	2026-08-10 19:43	3s

Frames, and what the machine read

0:02 #0 done2 line(s)

shot 0·sharpness 453.4
1. AlEngineer0.96
2. World's Fair0.97
0:03 #1 done2 line(s)

shot 1·sharpness 665.7
1. AlEngineer0.95
2. World's Fair0.99
0:10 #2 done24 line(s)

shot 2·sharpness 2743.2
1. LAB & PLATINUM SPONSORS0.98
2. Amazon AGI Lab0.98
3. ANTHROP\C1.00
4. Google DeepMind1.00
5. MINIMAX0.94
6. OpenAI0.92
7. Akamai1.00
8. arize1.00
9. aws1.00
10. Braintrust bright data0.99
11. B1.00
12. Browserbase1.00
13. docker1.00
14. :neo4j0.93
15. ORACLE1.00
16. PayPal1.00
17. qodo1.00
18. reducto1.00
19. Sonar1.00
20. Makers of0.99
21. togetherai1.00
22. Unblocked1.00
23. WorkOS1.00
24. SonarQube1.00
0:16 #3 done2 line(s)

shot 3·sharpness 141.0
1. AlEngineer0.99
2. World's Fair1.00
0:37 #4 done12 line(s)

shot 4·sharpness 1507.7
1. AlEngineer0.97
2. zep1.00
3. World'sFair1.00
4. AI ENGINEER WORLD'S FAIR 2026 - GRAPHS TRACK0.97
5. PRESENTED BY1.00
6. Citation needed0.99
7. Microsoft1.00
8. Provenance for LLM-built knowledge graphs0.99
9. Daniel Chalef0.97
10. FOUNDER, ZEP AI·GRAPHITI0.98
11. Engineering the future of Al0.98
12. World'sFair1.00
0:48 #5 skipped

shot 5·duplicate of #4
1:49 #6 done24 line(s)

shot 6·sharpness 1952.6
1. AlEngineer0.99
2. OPEN SOURCE1.00
3. World's Fair0.99
4. Graphiti1.00
5. getzep/graphiti1.00
6. ★ 28.2k0.97
7. Temporal knowledge graphs, built from your data by0.98
8. LLM extraction.0.99
9. RUNS ON1.00
10. The engine behind Zep's agent memory.0.99
11. Neo4j1.00
12. FalkorDB1.00
13. The lineage structure in this talk ships in Graphiti today.0.99
14. SHIPS IN GRAPHITI TODAY1.00
15. Amazon Neptune1.00
16. Zep's graph DB0.97
17. König0.99
18. PATTERN - BUILD ON IT0.97
19. The scoping patterns are what Zep has built on Graphiti.1.00
20. zep0.99
21. Citation Needed · AIE World's Fair 20260.98
22. TRACK 5• JULY 2,20260.95
23. World's Fair0.98
24. Graphs1.00
2:21 #7 skipped

shot 7·duplicate of #6
2:29 #8 skipped

shot 8·duplicate of #6
3:29 #9 done28 line(s)

shot 9·sharpness 1986.0
1. AlEngineer0.99
2. THE PROBLEM1.00
3. World's Fair0.98
4. Why a source link on the fact isn't enough0.99
5. The obvious fix: store a source_id on the fact.0.99
6. ETL - WHERE IT WORKS0.96
7. LLM EXTRACTION - WHERE IT BREAKS0.96
8. Several sources produce one fact. Its wording appears in0.99
9. many → one0.96
10. none.1.00
11. doc-17 · orders.csv0.97
12. merge1.00
13. "J. Smith" and "John Smith" merge into one identity.0.99
14. ↓1.00
15. invalidate1.00
16. Newer data supersedes the old fact.0.99
17. fact - value copied verbatim0.97
18. source_id: doc-17/0.94
19. One source, one value, copied. Lineage is one-to-one and0.98
20. sourcid: ?0.95
21. Lineage must be set-valued, survive mutation, answer0.99
22. deterministic.1.00
23. queries.1.00
24. zep1.00
25. Citation Needed · AlE World's Fair 20260.98
26. TRACK 5· JULY 2, 20260.96
27. World's Fair0.95
28. Graphs1.00
4:06 #10 skipped

shot 10·duplicate of #9
4:44 #11 done28 line(s)

shot 11·sharpness 1796.1
1. AlEngineer0.99
2. THE STRUCTURE0.99
3. World's Fair0.99
4. Provenance is a graph problem0.98
5. DERIVED - ENTITIES AND FACTS0.98
6. provenance link0.97
7. episode1.00
8. derived1.00
9. Patient1.00
10. HAS_ALLERGY1.00
11. Penicillin1.00
12. fact: "Patient has a penicillin allergy"1.00
13. GROUND TRUTH - EPISODES, STORED VERBATIM0.99
14. ep-4f21 ehr_export.json1.00
15. ep-7c88 lab_results.pdf0.98
16. ep-9a03 intake_chat0.98
17. { "allergy": "penicillin",0.99
18. "positive skin test,0.99
19. "I think I'm allergic0.98
20. "recorded_by": "Dr. Osei" }0.97
21. penicillin G._"0.94
22. to penicillin?"0.98
23. derived layer above, ground truth below0.99
24. zep1.00
25. Citation Needed · AIE World's Fair 20260.97
26. TRACK 5· JULY 2, 20260.95
27. World's Fair0.92
28. Graphs1.00
5:22 #12 skipped

shot 12·duplicate of #6
5:54 #13 skipped

shot 13·duplicate of #6
6:17 #14 done32 line(s)

shot 14·sharpness 2249.0
1. AlEngineer1.00
2. PATTERNS1.00
3. World's Fair0.99
4. Metadata projection: tag once, inherit everywhere1.00
5. Tag the episode once, at ingestion. Everything derived from it inherits the tag.0.99
6. DERIVED - INHERITS THROUGH EPISODE LINKS0.99
7. PRESENTED BY0.99
8. Patient1.00
9. HAS_ALLERGY1.00
10. Penicillin1.00
11. Microsoft1.00
12. source=EHR0.99
13. inherited1.00
14. fact: "Patient has a penicillin allergy"1.00
15. source=EHR- inherited0.97
16. source=EHR· inherited0.98
17. GROUND TRUTH - TAGGED ONCE, AT INGESTION0.98
18. source=EHR·verified=true0.99
19. ep-4f21 ehr_export.json1.00
20. ep-7c88 lab_results.pdf1.00
21. ep-9a03 intake_chat0.98
22. { "allergy": "penicillin",0.98
23. ".positive skin test,0.98
24. "I think I'm allergic1.00
25. "recorded_by": "Dr. Osei"}0.97
26. penicillin G."0.95
27. to penicillin?"1.00
28. zep0.99
29. Citation Needed · AIE World's Fair 20260.97
30. TRACK 5· JULY 2, 20260.95
31. World's Fair0.95
32. Graphs1.00
6:32 #15 skipped

shot 15·duplicate of #9
7:03 #16 skipped

shot 16·duplicate of #9
7:38 #17 skipped

shot 17·duplicate of #6
8:15 #18 skipped

shot 18·duplicate of #6
8:48 #19 done21 line(s)

shot 19·sharpness 2415.6
1. AlEngineer0.98
2. PATTERNS1.00
3. World'sFair1.00
4. Mixed-trust parentage: any parent, or all parents?0.99
5. A fact has three parents. Two are verified=t rue. One is not. Is the fact verified?0.98
6. fact - three parent episodes0.97
7. ep-4f21 · verified=true0.97
8. ep-7c88 · verified=true0.97
9. ep-9a03 · verified=false0.97
10. ANY1.00
11. "Patient has a penicillin allergy" — hearsay still blocks the prescription. The fatal failure is the false0.99
12. negative.1.00
13. ALL1.00
14. "Consent on file for the procedure" — proceed only when every source is verified. The fatal failure1.00
15. is the false positive.0.99
16. The right predicate depends on which failure hurts.1.00
17. zep0.99
18. Citation Needed · AIE World's Fair 20260.98
19. TRACK 5• JULY 2, 20260.96
20. World'sFair1.00
21. Graphs1.00
9:17 #20 skipped

shot 20·duplicate of #6
9:27 #21 skipped

shot 21·duplicate of #6
9:56 #22 skipped

shot 22·duplicate of #9
10:36 #23 done21 line(s)

shot 23·sharpness 2802.4
1. AlEngineer0.98
2. CLOSING1.00
3. World'sFair1.00
4. Key insights0.97
5. 011.00
6. LLM graph construction is lossy and generative0.99
7. PRESENTED BY1.00
8. — lineage must be engineered in, not logged after.1.00
9. Microsoft1.00
10. 021.00
11. Store sources verbatim; link every1.00
12. derived node and edge back to them.1.00
13. 031.00
14. The same edges give you compliance,0.99
15. verification, debugging, filtering, and deletion.1.00
16. github.com/getzep/graphiti1.00
17. zep1.00
18. Citation Needed - AIE World's Fair 20260.98
19. TRACK 5· JULY 2, 20260.95
20. World'sFair1.00
21. Graphs1.00

Transcript

205 cues· 2,379 words· 13,331 chars

chunk 0 · 0:12–0:50 · 74 words · 458 chars
0:12 So LLMs are really great at pulling together data from many sources, but they do so non-deterministically.
0:23 They interpret and synthesize data, generating a summary, an extracted fact, a structured record, and this output artifact may not appear verbatim in the source inputs.
0:36 Synthesis often destroys the paper trail of how these outputs were originated.
0:42 And I'm going to be talking today about provenance, which is tracing how an artifact was built and why.
chunk 1 · 0:51–1:35 · 82 words · 508 chars
0:51 Legal compliance often demands provenance, but it's also useful for debugging, deciding which sources you trust, and which artifacts to delete.
1:03 And solving this at scale presents a real engineering challenge.
1:10 Let me get my next slide going here.
1:14 So my team and I built Graffiti, the open source temporal graph framework.
1:19 And ZEP, our enterprise agent memory infrastructure, is built on Graffiti.
chunk 2 · 1:26–2:04 · 80 words · 479 chars
1:26 Our customers derive context or agent memory from many user touchpoints.
1:32 Those could be chat, but not only chat.
1:35 Often it's voice transcripts, email, business data.
1:39 And our customers have struggled with provenance.
1:42 Where did this fact come from?
1:45 What is the veracity of this fact?
1:47 And over the next few slides, I'll share how we engineered solutions to this problem.
1:55 So here's a stylized failure mode.
chunk 3 · 1:59–2:42 · 84 words · 496 chars
1:59 An agent retrieves context about a patient.
2:03 So this is a healthcare scenario.
2:06 And what comes back is a clean, confident fact, patient has a penicillin allergy.
2:13 And the context was synthesized from three sources, a lengthy EHR record, electronic health record, a PDF lab report, and something a patient typed into an AI intake chat.
chunk 4 · 2:30–3:10 · 82 words · 465 chars
2:30 If the agent presents the fact to a doctor in a treatment scenario without clearly indicating the source was from the patient themselves, it may mislead the doctor.
2:44 When an agent retrieves context, can we point to the exact source and its veracity?
2:51 For complex agent applications, the answer is often no.
2:58 So I can imagine you're probably thinking, but can't we just store like a source ID on the fact?
chunk 5 · 3:07–3:49 · 90 words · 532 chars
3:07 This can work well in structured data warehouses or data lakes.
3:11 A pipeline outputs one value, copied or mutated deterministically, and the sources are known and easily marked.
3:20 But with context pipelines run by LLMs, this breaks in several ways.
3:26 You prompt an LLM with several sources.
3:29 Many facts are each synthesized from one or more of the sources.
3:35 Somebody like Jay Smith and John Smith are merged into a single entity, one identity, and John's facts are derived from many different places.
chunk 6 · 3:46–4:30 · 86 words · 491 chars
3:46 So new data might invalidate old facts,
3:50 The store keeps changing underneath your pointer.
3:53 And a depend-only log, which often might come to mind here, gets very hard to manage at scale as there's so many changes occurring.
4:05 So lineage needs to be an evolving set and survive mutation.
4:13 So sets of links between facts and their sources can be modeled on a graph as relationships.
chunk 7 · 4:20–4:57 · 72 words · 414 chars
4:20 So provenance in a context store containing facts is a knowledge graph.
4:28 In this example, we have three source data.
4:31 In graffiti, they're turned episodes.
4:34 We have two entities extracted from the episodes, patient and penicillin, and an edge between them.
4:44 This graph triple, the two entities and the edge, can be hydrated as a fact
4:50 patient has a penicillin allergy.
chunk 8 · 4:54–5:30 · 69 words · 362 chars
4:54 Tracing a fact to its source is just a graph walk.
5:01 So it's pretty simple and easy to map source to fact on the first right.
5:09 But keeping it correct while the graph changes can be really hard.
5:16 When new data
5:18 So for example, when two entities merge, the merged entity needs to keep all source links from both, otherwise we silently drop a source and we lose lineage.
chunk 9 · 5:32–6:14 · 77 words · 438 chars
5:32 And when new data contradicts existing data, mutating it, we need to capture this lineage too.
5:41 In the rightmost card, a fact is rendered invalid by new data.
5:45 And in Graffiti, an invalid at date is added to the mutated edge.

Chapters

0:00 Why LLM synthesis destroys the paper trail
1:10 Graphiti, Zep, and the provenance problem
1:47 The failure mode: a penicillin allergy from three sources
2:53 Why a source ID does not survive an LLM pipeline
4:20 Provenance as a graph: tracing a fact is a walk
5:09 Keeping lineage correct through merges and invalidation
6:06 Metadata projection: tag a source once
7:25 Mixed trust parents: allergy flags versus consent
8:57 Deletion: GDPR erasure through the same edges
10:26 Benefits: compliance, veracity, and debuggability
11:31 Q&A: cost, dedup, and why not just markdown