Module 09 — Advanced RAG & Knowledge Systems

Time: 7–10 days · Depends on: 07 Tools & RAG · Pairs with: 08 MCP if retrieval is exposed as a server · Next: Cost optimization

---

Learning objectives

By the end of this module you will be able to:

• Diagnose why naive top-k dense retrieval fails on real corpora
• Build hybrid retrieval (BM25 + dense) fused with Reciprocal Rank Fusion (RRF)
• Apply cross-encoder reranking after a cheap first-stage retrieve
• Design hierarchical indices (doc → section → span) and agentic multi-step retrieval
• Evaluate retrieval and generation separately (Hit@k, MRR, faithfulness, context precision)

---

Why this matters (CS engineer)

Ops bot answers “what does ERR_INV_88421 mean?” with a confident essay about inventory philosophy. The gold runbook title is ERR_INV_88421 — never retrieved. Dense-only search mapped the question to “inventory errors” prose and missed the rare token. Support escalates. Team “fixes quality” by switching to a larger generator. Bill goes up. Hit@5 stays flat. The crime scene was retrieval, not eloquence.

Basic RAG is a vector nearest-neighbor lookup plus a prompt. Production RAG is closer to a search system: inverted indices, multi-stage ranking, query understanding, freshness, and offline metrics.

If you treat embeddings as magic:

• SKU codes, error IDs, and proper nouns miss because dense models blur rare tokens
• Multi-hop questions (“compare policy A with the exception in B”) need multiple retrieves
• You optimize the LLM when the gold document never entered the context

Your job is the information path: query → candidates → ranking → packing → generation → attribution. Generation quality is capped by what that path delivers.

---

Mental model

flowchart LR
  Q[User question] --> QU[Query understanding]
  QU --> D[Dense top-N]
  QU --> S[Sparse / BM25 top-N]
  D --> F[RRF fusion]
  S --> F
  F --> R[Rerank top-k]
  R --> P[Pack + cite]
  P --> LLM[Generator]
  LLM --> A[Answer + sources]
  QU -.->|multi-hop / agentic| Q2[Next sub-query]
  Q2 --> D

First stage optimizes recall (get the right docs in a shortlist).
Second stage optimizes precision (put the best spans in the window).
Generation should only compose what retrieval already supports.

Intuition lock

Sticky picture: Hybrid search is keyword cop + semantic cop on the same case — one chases exact IDs, the other chases paraphrase. RRF fuses rankings, not scores (you don’t average Fahrenheit and Celsius). Agentic RAG is a detective with a step budget, not an infinite coffee tab.

Kill this idea: “Better embeddings (or a bigger LLM) fix all RAG failures.” → Replace with: Diagnose the path — sparse miss, bad chunk, packing drop, multi-hop need, or generator ignoring context — and measure retrieval separately from generation.

---

Core tutorial

1. Failure modes of basic RAG (start here)

Symptom	Likely cause	Direction
Misses keyword SKUs / IDs	Dense-only; rare tokens	Hybrid BM25 + dense
Right doc, wrong span	Chunks too big or overlapping poorly	Smaller / structure-aware chunks + rerank
Multi-hop fails	One-shot query	Decomposition / agentic loop
Contradictory sources	No time/version filter	Metadata filters + conflict-aware prompt
Stale answers	Index drift	Freshness TTL, re-embed policy
Fluent wrong answer	No faithfulness gate	Cite + verify against context

Naive pipeline from Module 07:

embed(query) → top-k cosine → stuff chunks → LLM

That is a good lab baseline and a bad production default.

Explainer

Embeddings recap (from Module 07): a dense retriever embeds the query and each chunk independently (bi-encoder) so you can precompute document vectors and search with nearest-neighbor. Nearby vectors mean semantic closeness, not shared keywords. BM25 scores term match with IDF weighting. Product codes (INV-88421) and stack traces are high-IDF; dense models often under-weight them. Hybrid search is not “enterprise theater” — it is covering two different failure modes with two rankers.

A cross-encoder (rerank stage) reads query and passage together. It is slower and cannot precompute the whole corpus, which is why it only sees a shortlist.

---

2. Query understanding before you retrieve

Start with the raw query as your retrieval baseline and measure it with the eval harness from §10. Add rewriting, decomposition, expansion, or HyDE only when that measurement shows a retrieval problem the raw query can't solve — each of these techniques has a real cost: an exact identifier (order number, SKU, error code) can disappear during rewriting, expansion can hallucinate terms that pull in irrelevant documents, decomposition can drift from user intent, and every extra LLM call adds latency. Treat rewriting as a fix for a measured failure, not a default step.

Decompose multi-part questions — reach for this once a golden-set eval shows single-shot retrieval missing multi-hop questions:

import json

DECOMPOSE = """
Break the user question into independent search queries.
Return JSON only:
{{"queries": ["...", "..."], "needs_multi_hop": bool}}

Question: {q}
"""

def parse_queries(model_json: str) -> list[str]:
    data = json.loads(model_json)
    return list(data.get("queries") or [])

Other useful rewrites:

Technique	Idea	When
HyDE	LLM writes a hypothetical answer; embed that	Vague questions, short queries
Step-back	Ask a more general question first	Policy / conceptual retrieval
Expand synonyms	Domain glossary expansion	Vertical jargon
Filter extract	Pull product=, date> from natural language	Structured metadata exists

Always log the rewritten queries. Eval failures often come from bad rewrites, not bad embeddings.

Think about it

Question: A user asks: “Did the 2024 refund policy change the 30-day window that applied to enterprise SKUs last year?” How many retrievals do you need, and what goes wrong if you only retrieve once?

Reveal a strong answer

At least two conceptual hops: (1) 2024 refund policy text for the new window, (2) prior enterprise SKU policy / 30-day rule. A single embedding of the full sentence may land near one policy doc and miss the other. Decomposition into “2024 refund policy window” + “enterprise SKU return window 2023” (plus a filter on product tier if available) raises the chance both sources enter the context so the model can *compare* rather than invent.

---

3. Hybrid search: dense + sparse

Dense path: embedding model → ANN index (FAISS, Qdrant, Pinecone, pgvector).
Sparse path: BM25 / Elasticsearch / OpenSearch / sparse vectors (SPLADE-style).

query → dense top-N  ─┐
                      ├→ fuse → shortlist
query → sparse top-N ─┘

Sketch (IDs only — swap in real scorers):

def dense_top(query: str, n: int = 50) -> list[str]:
    ...  # ANN over embeddings

def bm25_top(query: str, n: int = 50) -> list[str]:
    ...  # inverted index

def hybrid_candidates(query: str, n: int = 50) -> list[str]:
    return rrf([dense_top(query, n), bm25_top(query, n)])

This repo ships RRF in src.rag:

from src.rag import rrf

fused = rrf(
    [
        ["docA", "docB", "docC"],  # dense ranks
        ["docC", "docA", "docD"],  # bm25 ranks
    ],
    k=60,
)
# docs that rank well in *either* list rise; agreement boosts further

---

4. Reciprocal Rank Fusion (why not just average scores?)

Different retrievers produce incomparable scores (cosine vs BM25). RRF ignores raw scores and uses ranks:

[ \mathrm{RRF}(d) = \sum_{r \in R} \frac{1}{k + \mathrm{rank}_r(d)} ]

• (k) (commonly 60) damps the top ranks so a #1 on one list does not dominate forever
• Missing from a list ⇒ that list contributes 0
• No score calibration needed

def rrf(rank_lists: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = {}
    for ranks in rank_lists:
        for i, doc_id in enumerate(ranks):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + i + 1)
    return [d for d, _ in sorted(scores.items(), key=lambda x: x[1], reverse=True)]

Weighted RRF (multiply a list’s contribution by (w)) is fine once you have offline Hit@k data. Do not invent weights without a labeled set.

Explainer

Why ranks, not scores? Cosine 0.82 and BM25 12.4 are not comparable — calibrating them is a research project. Rank position is comparable: “this doc was #3 for dense and #1 for BM25.” RRF is a cheap agreement vote. If you min-max both score lists into ([0,1]) without offline labels, you are inventing a fusion that only looks scientific.

---

5. Reranking (second stage)

Cross-encoders score (query, passage) jointly. They are too slow for millions of docs, so run them on a small first-stage shortlist rather than the full corpus.

hybrid shortlist (N) → cross-encoder scores → keep top 5–8 → LLM

There's no universal right size for N. Cross-encoders are normally applied to a relatively small first-stage candidate set — commonly tens to hundreds of passages — sized by your p95 latency budget and the cross-encoder's own speed (a small local model tolerates a larger shortlist than a hosted rerank API call). Measure rerank latency at your actual shortlist size before picking a default; don't copy a number from a blog post that ran on different hardware and a different model.

def rerank(query: str, passages: list[tuple[str, str]], top_k: int = 5):
    """passages: list of (id, text). score_fn is a cross-encoder or API."""
    scored = [(pid, score_fn(query, text)) for pid, text in passages]
    scored.sort(key=lambda x: x[1], reverse=True)
    return scored[:top_k]

Options: sentence-transformers cross-encoders, Cohere/Jina/Voyage rerank APIs, or a small local model. Measure latency budget: rerank should fit inside your p95, not only the happy path.

Explainer

Bi-encoders (dense retrieval) embed query and doc independently so you can precompute doc vectors. Cross-encoders see both texts at once — higher quality, no ANN precompute. Classic IR cascade: cheap broad recall → expensive precise ranking. Skipping the cascade either explodes cost or tanks quality.

---

6. Hierarchical RAG

Index at multiple granularities:

1. Doc-level summaries — route which documents matter
2. Section-level chunks — main context for the LLM
3. Sentence / span — precise citations and table cells

query → retrieve summaries → open top documents
      → retrieve fine chunks *only inside* those docs
      → optional span extract for citations

Benefits:

• Less noise (global top-k from a huge corpus mixes unrelated domains)
• Better parent context (section headers survive)
• Cheaper fine retrieval when constrained to a doc set

Implementation tip: store parent_id / doc_id metadata on every chunk; never drop it in the vector payload.

---

7. Graph-oriented retrieval (when relationships matter)

Use entity/graph structure when questions are about edges, not bags of text:

• “Who owns service X and what depends on it?”
• “Which tickets share root cause entity Y?”

Pattern:

1. Entity-link the query (service:X)
2. Traverse 1–2 hops in a graph or join table
3. Pull text chunks for the resulting node set
4. Generate with those chunks

Start with an entity linking table + SQL/Cypher before a full GraphRAG product. Graphs help structure; vectors still help language.

---

8. Agentic RAG

When one retrieve is not enough, wrap retrieval in a bounded loop:

while not done and steps < limit:
  plan next info need
  retrieve / tool call
  critique: is evidence sufficient?
  answer or continue

def agentic_answer(question: str, retrieve, llm, max_steps: int = 4) -> str:
    notes: list[str] = []
    for step in range(max_steps):
        plan = llm(
            f"Goal: {question}\nNotes:\n{notes}\n"
            "Return JSON: {\"action\":\"search|answer\","
            "\"query\":str|null,\"draft\":str|null}"
        )
        # parse plan ...
        if action == "answer":
            return draft
        hits = retrieve(query, k=5)
        notes.append(f"Q: {query}\n" + "\n".join(hits))
    return llm(f"Answer with available notes only:\n{notes}\nQ: {question}")

Hard rules (same spirit as Module 11 agents):

• Cap steps and total retrieved tokens
• Log every query + hit IDs for offline eval
• Prefer “I don’t know” over another expensive hop when notes are empty

Agentic RAG multiplies cost. Gate it: only when a cheap single-shot retrieve scores low confidence or the query is classified multi-hop.

Raw dense-retrieval similarity scores are not calibrated confidence — a 0.72 cosine score doesn't mean "72% likely relevant," and the threshold that separates good from bad retrieval shifts per embedding model, index, and domain. Before wiring a similarity score into the escalation gate above, validate the threshold against a representative labeled evaluation set (queries with known-good/known-bad retrievals); skip agentic escalation only once that threshold rule is measured, not assumed.

Think about it

Question: Your agentic RAG loop has max_steps=8 and no memory of prior queries. On step 3–7 it re-embeds the same paraphrase of the original question and re-pulls the same empty shortlist. What two controls stop this class of burn?

Reveal a strong answer

(1) **Repeated-query abort** (normalize query text / embedding near-duplicates) so thrash ends after 1–2 identical retrieves. (2) **Early exit on empty evidence** — if notes stay empty, answer “I don’t know” or escalate instead of spending remaining steps. Bonus: cache retrieve(query)→ids, lower max_steps for single-fact classifiers, and log intermediate queries so offline eval can see the loop.

---

9. Packing context for the generator

Retrieval quality dies in packing if you:

• Dump 20 near-duplicate chunks
• Exceed the window so system instructions get truncated
• Omit source IDs the model cannot cite

Good packing:

1. Deduplicate near-identical text (hash / embedding similarity)
2. Enforce a token budget (Module 05) — e.g. 2–4k tokens of evidence
3. Preserve structure: title, section, chunk id
4. Instruct: answer only from sources; cite ids; refuse if insufficient

def pack(chunks: list[tuple[str, str]], budget_chars: int = 12_000) -> str:
    out, used = [], 0
    for cid, text in chunks:
        block = f"[{cid}]\n{text}\n"
        if used + len(block) > budget_chars:
            break
        out.append(block)
        used += len(block)
    return "\n".join(out)

---

10. Evaluation: measure the path, not the vibes

Split metrics:

Layer	Metric	Meaning
Retrieval	Hit@k / Recall@k	Gold doc id appears in top-k?
Retrieval	MRR	(1/\mathrm{rank}) of first relevant
Retrieval	nDCG@k	Graded relevance ranking quality
Context	Context precision	Fraction of packed chunks that are useful
Generation	Faithfulness	Claims supported by provided context?
Generation	Answer relevance	On-topic vs the question?

def hit_at_k(retrieved_ids: list[str], gold_ids: set[str], k: int) -> float:
    return float(bool(gold_ids & set(retrieved_ids[:k])))

def mrr(retrieved_ids: list[str], gold_ids: set[str]) -> float:
    for i, doc_id in enumerate(retrieved_ids, start=1):
        if doc_id in gold_ids:
            return 1.0 / i
    return 0.0

Build a labeled set: (question, must_have_doc_ids, optional gold_answer).
Run retrieval eval without calling the LLM when iterating on hybrid/RRF/rerank.
Use LLM-as-judge only for faithfulness/relevance with a fixed rubric (Module 04), and spot-check humans.

Ecosystem: RAGAS, promptfoo, DeepEval, Langfuse experiments.

Think about it

Question: Hit@5 improved from 0.55 → 0.78 after hybrid+RRF, but user-rated answer quality barely moved. What might still be broken?

Reveal a strong answer

Retrieval now finds the right docs, but generation may still fail: wrong span inside the doc (need rerank / smaller chunks), packing drops the gold chunk (budget / dedupe bug), prompt does not force citation grounding, or the gold answer requires multi-hop synthesis the single-shot prompt cannot do. Also check faithfulness: the model may ignore context. Fix by measuring context precision and faithfulness separately, then inspect a few failure traces end-to-end.

---

11. Multimodal and document structure notes

• PDFs: use layout-aware parsers; keep tables as tables, not garbled line soup
• Images: caption-then-embed or vision embeddings; store modality in metadata
• Code: chunk by symbol / AST, not fixed 500-char windows
• Never mix incompatible embedding spaces without an explicit routing plan

---

Failure modes (advanced RAG)

Symptom	Likely root cause	Fix
Hybrid worse than dense alone	Bad fusion / noisy BM25 corpus	Tune N; filter stopwordy fields; weight lists from eval
Reranker slow / timeouts	Scoring 200+ passages	Cap shortlist; batch; smaller cross-encoder
Agentic loop burns $	No step budget / thrashing queries	max_steps, repeated-query abort, cache
High Hit@k, low faithfulness	Generator ignores context	Stronger cite prompt; post-hoc claim check
Citations hallucinated	Free-form ids	Constrain to provided id set; validate like TinyRAG.validate_citations
Index stale after deploys	No ingestion versioning	Content hash, re-embed changed docs only

---

Lab

Goal: Prove hybrid + fusion beats dense-only on a small labeled set.

1. Take 5–10 of your own notes / READMEs (Module 07 corpus is fine).
2. Write 20 questions with must_have chunk or doc ids (include 5 keyword/ID questions and 5 multi-hop).
3. Implement or mock:
4. dense ranks
5. keyword/BM25 ranks
6. rrf from src.rag
7. Report Hit@5 and MRR for dense-only vs hybrid.
8. Optional: add a tiny rerank (even a lexical overlap score) and show delta.
9. For 5 multi-hop items, run a 2-step decompose → retrieve → answer; log intermediate queries.

# sanity: RRF unit behavior lives next to TinyRAG
poetry run pytest tests/test_rag.py -v

Stretch: hierarchical parent filter — retrieve doc summaries first, then only child chunks of top docs.

---

Quizzes

Quiz · +25 XP

Why is Reciprocal Rank Fusion preferred over averaging dense cosine scores with BM25 scores?

RRF always finds more documents than either list alone Raw scores from different retrievers are not on a shared scale; ranks are BM25 scores are probabilities and must be multiplied by cosine Cross-encoders require RRF as a mathematical precondition

Quiz · +25 XP

You change only the embedding model. Which metric should move first if the change is good?

BLEU on the final answer string Hit@k / MRR on a labeled retrieval set GPU utilization of the generator Average output tokens per response

Quiz · +25 XP

When is agentic (multi-step) RAG the wrong default?

When questions require comparing two policies When most queries are single-fact lookups and cost/latency budgets are tight When you already have a cross-encoder When Hit@k is already measured offline

---

Open source materials

Resource	Use it for
mlabonne/llm-course	RAG engineer path context
RAGAS	Faithfulness / context metrics
FAISS · Qdrant · Chroma	Vector indices
huggingface/agents-course	Agentic retrieval patterns
Sentence-transformers docs	Bi-encoders + cross-encoders
Course src/rag.py	rrf, TinyRAG teaching baseline

Also: Curated resources → RAG & embeddings.

---

Checkpoint

• You can explain hybrid search + RRF in one clear paragraph
• You rerank or fuse — not only single-vector top-k
• You measure retrieval (Hit@k / MRR) separately from generation
• Multi-hop path has a step budget and logged intermediate queries
• Citations are constrained to retrieved ids

---

Mark complete when you have run a Hit@k comparison (dense vs hybrid) on a small labeled set and can defend your fusion/rerank choices.

Complete module · +120 XP

Exercise

• Catalog: EX-09 — Hybrid retrieval
• Prove: Hybrid Hit@5 / MRR is reported against dense-only on a labeled slice.
• Test: pytest tests/test_rag.py -v

Next: Module 10 — Cost optimization