When All Context Matters: Extended Cache Augmented Generation - Luis Romero-Sevilla, Orbis

1. chunk 0 · 0:10–0:51 · 115 words · 662 chars
2. 0:10 I'm Luis Romero Sevilla and I'm the VP of AI at Orbis Operations.
3. 0:15 I'm on a mission to solve knowledge representation when all context matters.
4. 0:19 So, let's start with a very specific example.
5. 0:24 Let's say we have a large number of documents.
6. 0:26 And all documents represent an event.
7. 0:28 And all documents in the collection are relevant to answer a set of questions that the user has.
8. 0:34 Not only that, there's one more challenge.
9. 0:38 the document in the collection becomes obsolete very fast and all documents get replaced with new information.
10. chunk 1 · 0:44–1:25 · 106 words · 612 chars
11. 0:44 Let's start with the simplest approach.
12. 0:46 We could start with a simple reg.
13. 0:48 For that we just need a vector database and an embedding model.
14. 0:53 An embedding model takes the documents and turns them into a learned numerical representation, a vector.
15. 1:00 Now we take those vectors and we store them in a database optimized for performing operations with vectors.
16. 1:07 Perfect.
17. 1:08 Now we can take all of our questions, turn them into vectors, and then look for vectors that are similar to the initial query.
18. chunk 2 · 1:18–2:00 · 112 words · 640 chars
19. 1:18 Those vectors that are within the similarity threshold are retrieved, and we can pass them to the LLM to answer the question.
20. 1:27 Inserting to a vector database, it's relatively fast.
21. 1:30 So whenever a collection becomes obsolete, we can just replace it with a new one.
22. 1:36 We still have one problem with our very specific scenario.
23. 1:41 All the documents in the collection are relevant for us to answer the question.
24. 1:45 So we can't just take all the documents in the collection and pass them to LLM.
25. chunk 3 · 1:51–2:32 · 106 words · 649 chars
26. 1:51 And that's just one of the many limitations with this approach.
27. 1:55 Now let's get a bit more sophisticated.
28. 1:58 All documents are relevant to answer a global question.
29. 2:01 Therefore, there must be some connections and relationship between the details within a document in the collections.
30. 2:08 For us to map out those relationship, we're going to need a knowledge graph.
31. 2:12 And one implementation we could try is GraphRag.
32. 2:16 GraphRag had many stats, but basically it uses an LLM to read through all the documents and extract key entities and relationship between them.
33. chunk 4 · 2:27–3:07 · 104 words · 668 chars
34. 2:27 It constructs a network, a knowledge graph, where all those connections and details are tied together.
35. 2:32 Then when a question is asked, it navigates this graph to synthesize a complete answer drawn across the entire collection.
36. 2:39 If your collection of documents isn't changing very often, GraphRack is an excellent approach for finding those relationships within details to answer the user's question.
37. 2:51 However, our very specific scenario states that our data is not only deeply interconnected, but also the data gets replaced very often.
38. chunk 5 · 3:01–3:44 · 135 words · 772 chars
39. 3:01 Recomputing a knowledge graph every time the data gets replaced is computationally very expensive and it takes a relatively long time.
40. 3:08 Okay, what if we take an even simpler approach and we continue to build on top of it?
41. 3:13 If we were to use something like GraphRack, each document needs to pass through an LLM for the entity and relationship structure anyway.
42. 3:21 Why can't we just throw all the documents into context?
43. 3:24 This approach will look something like cache amended generation, CAG, where we use a model with a large context window, load the documents into the context, and cache the context by storing the model's KB matrix.
44. chunk 6 · 3:38–4:19 · 111 words · 652 chars
45. 3:38 The problem here is that the context window is limited, and if you fill the context window too much, the quality of the answer gets degraded too.
46. 3:46 The solution, what if we use four kegs in parallel and distribute the documents across different context buckets?
47. 3:54 Now each keg can answer questions regarding its content.
48. 3:57 And now we just need something to ask the right questions to the right buckets.
49. 4:03 So for this, we can use a smarter model to interrogate each bucket and eventually synthesize an answer.
50. chunk 7 · 4:11–4:51 · 105 words · 676 chars
51. 4:11 How do we distribute the documents?
52. 4:13 It sounds tempting to organize the documents by domains and tell the supervisor, here are the different categories.
53. 4:20 But in practice, with very dense relationship between documents, the supervisor tends to ignore domains that at first glance seem irrelevant.
54. 4:30 For this reason, all documents are distributed in no particular order.
55. 4:34 The only requirement is to balance the number of documents in a way that the least amount of documents are needed.
56. 4:40 Then the supervisor model starts exploring the buckets and progressively builds its internal understanding.
57. chunk 8 · 4:47–5:07 · 45 words · 291 chars
58. 4:47 And if it finds something interesting, it can ask a specific bucket follow-up questions.
59. 4:52 Because all caches can be loaded in parallel, the knowledge building process is significantly faster than graph rag while providing more accurate answers than a simple rag.

loading