Module 09 — Advanced RAG & Knowledge Systems

Time: 7–10 days · Depends on: 07 Tools & RAG · Pairs with: 08 MCP if retrieval is exposed as a server · Corpus: Corpus engineering · Next: MCP

---

Fictional teaching scenario.

Ops bot answers “what does ERR_INV_88421 mean?” with a confident essay about inventory philosophy. The gold runbook title is ERR_INV_88421 — never retrieved. Dense-only search mapped the question to “inventory errors” prose and missed the rare token. Support escalates. Team “fixes quality” by switching to a larger generator. Bill goes up. Hit@5 stays flat. The crime scene was retrieval, not eloquence.

Case question: Which retrieval metric will reveal that the rare identifier never reached the generator, before anyone pays for a larger model?

Learning objectives

• Diagnose why naive top-k dense retrieval fails on real corpora
• Filter the candidate set with metadata before top-k
• Build hybrid retrieval (BM25 + dense) fused with Reciprocal Rank Fusion (RRF)
• Apply cross-encoder reranking after a cheap first-stage retrieve
• Design hierarchical indices (doc → section → span) and agentic multi-step retrieval
• Evaluate retrieval and generation separately (Hit@k, MRR, faithfulness, context precision)

---

Basic RAG is a vector nearest-neighbor lookup plus a prompt. Production RAG is closer to a search system: inverted indices, multi-stage ranking, query understanding, freshness, and offline metrics.

If you treat embeddings as magic:

• SKU codes, error IDs, and proper nouns miss because dense models blur rare tokens
• Multi-hop questions (“compare policy A with the exception in B”) need multiple retrieves
• You optimize the LLM when the gold document never entered the context

Your job is the information path: query → candidates → ranking → packing → generation → attribution. Generation quality is capped by what that path delivers.

---

Mental model

flowchart LR
  Q[User question] --> QU[Query understanding]
  QU --> D[Dense top-N]
  QU --> S[Sparse / BM25 top-N]
  D --> F[RRF fusion]
  S --> F
  F --> R[Rerank top-k]
  R --> P[Pack + cite]
  P --> LLM[Generator]
  LLM --> A[Answer + sources]
  QU -.->|multi-hop / agentic| Q2[Next sub-query]
  Q2 --> D

First stage optimizes recall (get the right docs in a shortlist).
Second stage optimizes precision (put the best spans in the window).
Generation should only compose what retrieval already supports.

Intuition lock

Sticky picture: Hybrid search is keyword cop + semantic cop on the same case — one chases exact IDs, the other chases paraphrase. RRF fuses rankings, not scores (you don’t average Fahrenheit and Celsius). Agentic RAG is a detective with a step budget, not an infinite coffee tab.

Kill this idea: “Better embeddings (or a bigger LLM) fix all RAG failures.” → Replace with: Diagnose the path — sparse miss, bad chunk, packing drop, multi-hop need, or generator ignoring context — and measure retrieval separately from generation.

---

Core tutorial

1. Failure modes of basic RAG (start here)

Symptom	Likely cause	Direction
Misses keyword SKUs / IDs	Dense-only; rare tokens	Hybrid BM25 + dense
Right doc, wrong span	Chunks too big or overlapping poorly	Smaller / structure-aware chunks + rerank
Multi-hop fails	One-shot query	Decomposition / agentic loop
Contradictory sources	No time/version filter	Metadata filter before rank, then a conflict-aware prompt
Stale answers	Index drift	Freshness TTL, re-embed policy
Fluent wrong answer	No faithfulness gate	Cite + verify against context

Naive pipeline from Module 07:

embed(query) → top-k cosine → stuff chunks → LLM

That is a good lab baseline and a bad production default.

Explainer

Embeddings recap (from Module 07): a dense retriever embeds the query and each chunk independently (bi-encoder) so you can precompute document vectors and search with nearest-neighbor. Nearby vectors mean semantic closeness, not shared keywords. BM25 scores term match with IDF weighting. Product codes (INV-88421) and stack traces are high-IDF; dense models often under-weight them. Hybrid search is not “enterprise theater” — it is covering two different failure modes with two rankers.

A cross-encoder (rerank stage) reads query and passage together. It is slower and cannot precompute the whole corpus, which is why it only sees a shortlist.

---

Metadata filtering

Similarity will rank a clause from the wrong tier, and it will rank a clause the reader is not allowed to see. Both failures happen before the model writes a word. Apply the filter in the retriever, then take top-k from what remains.

The refund page has two chunks. “Standard refund window is 30 days.” overlaps the query “refund window.” “Enterprise SKUs: 14 days.” shares almost none of those tokens. Top-1 on the unfiltered corpus returns the 30-day sentence. Filtering tier=enterprise first leaves the exception as the candidate, and the 30-day sentence never enters the prompt.

Metadata is a hard constraint. Department, year, document type, and access level live on the chunk. Every pair in where has to match. A missing key excludes the chunk. An empty where keeps the whole corpus, which is the right lab default and the wrong production call when audiences are mixed.

The enterprise row is the search space. The standard window and the payroll note are gone before similarity runs, so the model cannot cite them.

src/rag.py applies the filter inside TinyRAG.retrieve before the score sort. pytest tests/test_rag.py -v.

def metadata_matches(meta: dict[str, str] | None, where: dict[str, str]) -> bool:
    """True when every pair in where is present. A missing key does not match."""
    fields = meta or {}
    return all(fields.get(key) == value for key, value in where.items())


def filter_by_metadata(chunks: list[Chunk], where: dict[str, str]) -> list[Chunk]:
    """Keep chunks whose metadata matches every pair in where.

    An empty where keeps every chunk. Callers apply this before top-k.
    """
    if not where:
        return list(chunks)
    return [chunk for chunk in chunks if metadata_matches(chunk.meta, where)]

from src.rag import Chunk, TinyRAG

chunks = [
    Chunk(
        "refund-window",
        "Standard refund window is 30 days.",
        "policy",
        {"tier": "standard", "department": "support"},
    ),
    Chunk(
        "refund-window-enterprise",
        "Enterprise SKUs: 14 days.",
        "policy",
        {"tier": "enterprise", "department": "support"},
    ),
    Chunk(
        "payroll",
        "Payroll runs on Friday.",
        "hr",
        {"tier": "standard", "department": "hr"},
    ),
]
rag = TinyRAG(chunks)
assert rag.retrieve("refund window", k=1)[0].id == "refund-window"
hits = rag.retrieve(
    "refund window",
    k=1,
    where={"tier": "enterprise", "department": "support"},
)
assert [c.id for c in hits] == ["refund-window-enterprise"]
assert rag.retrieve("refund window", k=3, where={"department": "legal"}) == []

A support agent’s where includes department=support. The payroll chunk never reaches build_prompt. The same rule as the tool allowlist in Module 07: the runtime decides what is visible. Values are strings, so store the year as "2024". Live tier for this customer still comes from the billing tool. The filter uses that value. It does not ask the model to guess it.

---

2. Query understanding before you retrieve

Start with the raw query as your retrieval baseline and measure it with the eval harness from §10. Add rewriting, decomposition, expansion, or HyDE only when that measurement shows a retrieval problem the raw query can't solve — each of these techniques has a real cost: an exact identifier (order number, SKU, error code) can disappear during rewriting, expansion can hallucinate terms that pull in irrelevant documents, decomposition can drift from user intent, and every extra LLM call adds latency. Treat rewriting as a fix for a measured failure, not a default step.

Decompose multi-part questions — reach for this once a golden-set eval shows single-shot retrieval missing multi-hop questions:

import json

DECOMPOSE = """
Break the user question into independent search queries.
Return JSON only:
{{"queries": ["...", "..."], "needs_multi_hop": bool}}

Question: {q}
"""

def parse_queries(model_json: str) -> list[str]:
    data = json.loads(model_json)
    return list(data.get("queries") or [])

Each query is a separate search. Dedupe the hits before packing, or the generator spends its budget on four copies of the same paragraph.

Other useful rewrites:

Technique	Idea	When
HyDE	LLM writes a hypothetical answer; embed that	Vague questions, short queries
Step-back	Ask a more general question first	Policy / conceptual retrieval
Expand synonyms	Domain glossary expansion	Vertical jargon
Filter extract	Pull product=, date> from natural language	Structured metadata exists

Always log the rewritten queries. Eval failures often come from bad rewrites, not bad embeddings.

The rewrite is a search string. Keep the original question in the log, and skip the rewrite when the raw query already contains an identifier you cannot afford to lose.

Think about it

Question: A user asks: “Did the 2024 refund policy change the 30-day window that applied to enterprise SKUs last year?” How many retrievals do you need, and what goes wrong if you only retrieve once?

Reveal a strong answer

At least two conceptual hops: (1) 2024 refund policy text for the new window, (2) prior enterprise SKU policy / 30-day rule. A single embedding of the full sentence may land near one policy doc and miss the other. Decomposition into “2024 refund policy window” + “enterprise SKU return window 2023” (plus a filter on product tier if available) raises the chance both sources enter the context so the model can *compare* rather than invent.

---

3. Hybrid search: dense + sparse

One list chases meaning. The other chases exact tokens. Fusion uses rank position, because a cosine and a BM25 score are different units.

Dense path: embedding model → ANN index (FAISS, Qdrant, Pinecone, pgvector).
Sparse path: BM25 / Elasticsearch / OpenSearch / sparse vectors (SPLADE-style).

query → dense top-N  ─┐
                      ├→ fuse → shortlist
query → sparse top-N ─┘

Sketch (IDs only — swap in real scorers):

def dense_top(query: str, n: int = 50) -> list[str]:
    ...  # ANN over embeddings

def bm25_top(query: str, n: int = 50) -> list[str]:
    ...  # inverted index

def hybrid_candidates(query: str, n: int = 50) -> list[str]:
    return rrf([dense_top(query, n), bm25_top(query, n)])

This repo ships RRF in src.rag:

from src.rag import rrf

fused = rrf(
    [
        ["docA", "docB", "docC"],  # dense ranks
        ["docC", "docA", "docD"],  # bm25 ranks
    ],
    k=60,
)
# docs that rank well in *either* list rise; agreement boosts further

---

4. Reciprocal Rank Fusion (why not just average scores?)

Different retrievers produce incomparable scores (cosine vs BM25). RRF ignores raw scores and uses ranks:

[ \mathrm{RRF}(d) = \sum_{r \in R} \frac{1}{k + \mathrm{rank}_r(d)} ]

• (k) (commonly 60) damps the top ranks so a #1 on one list does not dominate forever
• Missing from a list ⇒ that list contributes 0
• No score calibration needed

def rrf(rank_lists: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = {}
    for ranks in rank_lists:
        for i, doc_id in enumerate(ranks):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + i + 1)
    return [d for d, _ in sorted(scores.items(), key=lambda x: x[1], reverse=True)]

Weighted RRF (multiply a list’s contribution by (w)) is fine once you have offline Hit@k data. Do not invent weights without a labeled set.

Explainer

Why ranks, not scores? Cosine 0.82 and BM25 12.4 are not comparable — calibrating them is a research project. Rank position is comparable: “this doc was #3 for dense and #1 for BM25.” RRF is a cheap agreement vote. If you min-max both score lists into ([0,1]) without offline labels, you are inventing a fusion that only looks scientific.

---

5. Reranking (second stage)

The first stage spends its budget on recall. The reranker reads the query and the passage together and keeps the few passages that fit the window.

Cross-encoders score (query, passage) jointly. They are too slow for millions of docs, so run them on a small first-stage shortlist rather than the full corpus.

hybrid shortlist (N) → cross-encoder scores → keep top 5–8 → LLM

There's no universal right size for N. Cross-encoders are normally applied to a relatively small first-stage candidate set — commonly tens to hundreds of passages — sized by your p95 latency budget and the cross-encoder's own speed (a small local model tolerates a larger shortlist than a hosted rerank API call). Measure rerank latency at your actual shortlist size before picking a default; don't copy a number from a blog post that ran on different hardware and a different model.

def rerank(query: str, passages: list[tuple[str, str]], top_k: int = 5):
    """passages: list of (id, text). score_fn is a cross-encoder or API."""
    scored = [(pid, score_fn(query, text)) for pid, text in passages]
    scored.sort(key=lambda x: x[1], reverse=True)
    return scored[:top_k]

Options: sentence-transformers cross-encoders, Cohere/Jina/Voyage rerank APIs, or a small local model. Measure latency budget: rerank should fit inside your p95, not only the happy path.

Explainer

Bi-encoders (dense retrieval) embed query and doc independently so you can precompute doc vectors. Cross-encoders see both texts at once — higher quality, no ANN precompute. Classic IR cascade: cheap broad recall → expensive precise ranking. Skipping the cascade either explodes cost or tanks quality.

---

6. Hierarchical RAG

The search hits the 14-day exception. Packing still includes the 30-day rule it modifies. The corpus lesson is the pipeline that writes those ids.

Index at multiple granularities:

1. Doc-level summaries — route which documents matter
2. Section-level chunks — main context for the LLM
3. Sentence / span — precise citations and table cells

query → retrieve summaries → open top documents
      → retrieve fine chunks *only inside* those docs
      → optional span extract for citations

Benefits:

• Less noise (global top-k from a huge corpus mixes unrelated domains)
• Better parent context (section headers survive)
• Cheaper fine retrieval when constrained to a doc set

Implementation tip: store parent_id / doc_id metadata on every chunk; never drop it in the vector payload. The corpus lesson is the pipeline that produces those ids: the exception stays stapled to the rule, and a content hash decides whether the vectors are still valid.

---

7. Graph-oriented retrieval (when relationships matter)

The question is about the edge. Walk one or two hops, then pull the text for the nodes you land on.

Use entity/graph structure when questions are about edges, not bags of text:

• “Who owns service X and what depends on it?”
• “Which tickets share root cause entity Y?”

Pattern:

1. Entity-link the query (service:X)
2. Traverse 1–2 hops in a graph or join table
3. Pull text chunks for the resulting node set
4. Generate with those chunks

Start with an entity linking table + SQL/Cypher before a full GraphRAG product. Graphs help structure; vectors still help language.

---

8. Agentic RAG

The model proposes the next lookup. Your runtime executes it. The loop ends at the step budget, or earlier when the notes are still empty.

When one retrieve is not enough, wrap retrieval in a bounded loop:

while not done and steps < limit:
  plan next info need
  retrieve / tool call
  critique: is evidence sufficient?
  answer or continue

def agentic_answer(question: str, retrieve, llm, max_steps: int = 4) -> str:
    notes: list[str] = []
    for step in range(max_steps):
        plan = llm(
            f"Goal: {question}\nNotes:\n{notes}\n"
            "Return JSON: {\"action\":\"search|answer\","
            "\"query\":str|null,\"draft\":str|null}"
        )
        # parse plan ...
        if action == "answer":
            return draft
        hits = retrieve(query, k=5)
        notes.append(f"Q: {query}\n" + "\n".join(hits))
    return llm(f"Answer with available notes only:\n{notes}\nQ: {question}")

Hard rules (same spirit as Module 11 agents):

• Cap steps and total retrieved tokens
• Log every query + hit IDs for offline eval
• Prefer “I don’t know” over another expensive hop when notes are empty

Agentic RAG multiplies cost. Gate it: only when a cheap single-shot retrieve scores low confidence or the query is classified multi-hop.

Raw dense-retrieval similarity scores are not calibrated confidence — a 0.72 cosine score doesn't mean "72% likely relevant," and the threshold that separates good from bad retrieval shifts per embedding model, index, and domain. Before wiring a similarity score into the escalation gate above, validate the threshold against a representative labeled evaluation set (queries with known-good/known-bad retrievals); skip agentic escalation only once that threshold rule is measured, not assumed.

Think about it

Question: Your agentic RAG loop has max_steps=8 and no memory of prior queries. On step 3–7 it re-embeds the same paraphrase of the original question and re-pulls the same empty shortlist. What two controls stop this class of burn?

Reveal a strong answer

(1) **Repeated-query abort** (normalize query text / embedding near-duplicates) so thrash ends after 1–2 identical retrieves. (2) **Early exit on empty evidence** — if notes stay empty, answer “I don’t know” or escalate instead of spending remaining steps. Bonus: cache retrieve(query)→ids, lower max_steps for single-fact classifiers, and log intermediate queries so offline eval can see the loop.

---

Corrective RAG

The judge sits after retrieval. A shortlist that contains the gold id proceeds. A shortlist that misses it is rewritten or expanded, still inside the step cap.

The branch in the figure is the evidence check in the agentic loop. A raw cosine score is not the judge. On a labeled set, a missing gold id means rewrite or expand and retrieve again. A present gold id means generate. When the second pass is still empty, the answer contract abstains.

---

Self-RAG

The checklist is split across pieces you already run. Routing decides whether to retrieve. The reranker and the packer decide which chunks are worth the window. The contract decides whether the evidence is enough. The critique is the last step of the bounded loop.

“Should I retrieve?” is the tools / RAG / weights route in Module 07. “Is this chunk relevant?” is the reranker and the packer’s dedupe. “Do I have enough?” is the answer contract: answer, clarify, abstain, or escalate. “Critique the draft” is the agentic loop’s last look, and max_steps still ends it.

---

9. Packing context for the generator

Retrieval quality dies in packing if you:

• Dump 20 near-duplicate chunks
• Exceed the window so system instructions get truncated
• Omit source IDs the model cannot cite

Good packing:

1. Deduplicate near-identical text (hash / embedding similarity)
2. Enforce a token budget (Module 05) — e.g. 2–4k tokens of evidence
3. Preserve structure: title, section, chunk id
4. Instruct: answer only from sources; cite ids; refuse if insufficient

def pack(chunks: list[tuple[str, str]], budget_chars: int = 12_000) -> str:
    out, used = [], 0
    for cid, text in chunks:
        block = f"[{cid}]\n{text}\n"
        if used + len(block) > budget_chars:
            break
        out.append(block)
        used += len(block)
    return "\n".join(out)

---

10. Evaluation: measure the path, not the vibes

Split metrics:

Layer	Metric	Meaning
Retrieval	Hit@k / Recall@k	Gold doc id appears in top-k?
Retrieval	MRR	(1/\mathrm{rank}) of first relevant
Retrieval	nDCG@k	Graded relevance ranking quality
Context	Context precision	Fraction of packed chunks that are useful
Generation	Faithfulness	Claims supported by provided context?
Generation	Answer relevance	On-topic vs the question?

def hit_at_k(retrieved_ids: list[str], gold_ids: set[str], k: int) -> float:
    return float(bool(gold_ids & set(retrieved_ids[:k])))

def mrr(retrieved_ids: list[str], gold_ids: set[str]) -> float:
    for i, doc_id in enumerate(retrieved_ids, start=1):
        if doc_id in gold_ids:
            return 1.0 / i
    return 0.0

Build a labeled set: (question, must_have_doc_ids, optional gold_answer).
Run retrieval eval without calling the LLM when iterating on hybrid/RRF/rerank.
Use LLM-as-judge only for faithfulness/relevance with a fixed rubric (Module 04), and spot-check humans.

Ecosystem: RAGAS, promptfoo, DeepEval, Langfuse experiments.

Think about it

Question: Hit@5 improved from 0.55 → 0.78 after hybrid+RRF, but user-rated answer quality barely moved. What might still be broken?

Reveal a strong answer

Retrieval now finds the right docs, but generation may still fail: wrong span inside the doc (need rerank / smaller chunks), packing drops the gold chunk (budget / dedupe bug), prompt does not force citation grounding, or the gold answer requires multi-hop synthesis the single-shot prompt cannot do. Also check faithfulness: the model may ignore context. Fix by measuring context precision and faithfulness separately, then inspect a few failure traces end-to-end.

---

11. Multimodal and document structure notes

• PDFs: use layout-aware parsers; keep tables as tables, not garbled line soup
• Images: caption-then-embed or vision embeddings; store modality in metadata
• Code: chunk by symbol / AST, not fixed 500-char windows
• Never mix incompatible embedding spaces without an explicit routing plan

Those four lines are the reminder. The corpus lesson is the worked pipeline: layout-aware parse, parent and child chunks, re-embed on content hash, and a reviewed relation when two cited clauses disagree.

---

Failure modes (advanced RAG)

Symptom	Likely root cause	Fix
Hybrid worse than dense alone	Bad fusion / noisy BM25 corpus	Tune N; filter stopwordy fields; weight lists from eval
Reranker slow / timeouts	Scoring 200+ passages	Cap shortlist; batch; smaller cross-encoder
Agentic loop burns $	No step budget / thrashing queries	max_steps, repeated-query abort, cache
High Hit@k, low faithfulness	Generator ignores context	Stronger cite prompt; post-hoc claim check
Citations hallucinated	Free-form ids	Constrain to provided id set; validate like TinyRAG.validate_citations
Index stale after deploys	No ingestion versioning	Content hash, re-embed changed docs only

---

Case checkpoint

Opening failure: Dense-only retrieval missed a rare error identifier, so changing the generator could not improve the answer.

What this lab demonstrates: The labeled queries compare dense-only with hybrid Hit@5 and MRR, making candidate-retrieval improvement visible before generation.

What it does not prove: A small corpus and lexical reranker do not prove production relevance, freshness, or grounded generation.

---

Lab

Goal: Prove hybrid + fusion beats dense-only on a small labeled set.

1. Take 5–10 of your own notes / READMEs (Module 07 corpus is fine).
2. Write 20 questions with must_have chunk or doc ids (include 5 keyword/ID questions and 5 multi-hop).
3. Implement or mock:
4. dense ranks
5. keyword/BM25 ranks
6. rrf from src.rag
7. Report Hit@5 and MRR for dense-only vs hybrid.
8. Optional: add a tiny rerank (even a lexical overlap score) and show delta.
9. For 5 multi-hop items, run a 2-step decompose → retrieve → answer; log intermediate queries.
10. Run the four-path comparison below. Keep retrieval rank, source support, and “does this answer the question?” in separate columns, and include one case where the citation is real and the answer is still wrong.
11. On one query, pass a metadata where before top-k. Show that the unfiltered top-1 was the wrong tier, and that a missing key returns nothing.

# sanity: RRF unit behavior lives next to TinyRAG
poetry run pytest tests/test_rag.py -v

Stretch: hierarchical parent filter — retrieve doc summaries first, then only child chunks of top docs.

---

Compare the paths on one corpus

A customer asks about order A17. The assistant returns a neatly cited paragraph about A71. Every citation validator passes. The document exists. The quotation is faithful. The customer still has the wrong answer.

Three questions have become tangled: did we retrieve useful evidence, does the answer follow from that evidence, and does it answer the question? This experiment gives each question its own number. Allow 60–90 minutes, plus optional model downloads.

Make four predictions

The fixture contains six policy/order documents and twelve labeled queries. Some use synonyms, two contain easily confused order IDs, and two have no answer.

1. Which path should preserve A17 better: a keyword score or a semantic vector?
2. What happens when “parcel” must find a document that says “shipping”?
3. Can a reranker recover a document that never entered its candidate pool?
4. Will a faithful quotation necessarily answer an unanswerable question?

Run the controlled experiment first:

python -m examples.retrieval.compare --output /tmp/retrieval-fixture.json

Path	What runs
Sparse	BM25 term matching
Dense	Cosine over three hand-authored semantic features
Hybrid	Reciprocal rank fusion of the two rankings
Reranked	Top four hybrid results ordered by annotated support

The hybrid step combines rank positions, because a BM25 score and a cosine similarity are different units:

def rrf(rank_lists: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = {}
    for ranking in rank_lists:
        for position, document_id in enumerate(ranking, start=1):
            scores[document_id] = scores.get(document_id, 0.0) + 1 / (k + position)
    return sorted(scores, key=scores.get, reverse=True)

hybrid = rrf([sparse_ranking, dense_ranking])
rerank_candidates = hybrid[:4]  # the reranker cannot recover rank 5

The default vectors and reranker are transparent test fixtures, not learned models. The reranker deliberately consults annotations; it tests candidate-pool and metric wiring. Its score is not evidence of a real model's quality. That distinction is why the report names its mode prominently.

Let real models surprise you

In a separate environment, install examples/retrieval/requirements.txt. The real path uses SentenceTransformer embeddings and a CrossEncoder reranker. The same corpus, query labels, candidate budget, and metrics stay fixed.

python -m examples.retrieval.compare --real \
  --embedding-model sentence-transformers/all-MiniLM-L6-v2 \
  --embedding-revision REPLACE_WITH_MODEL_COMMIT_SHA \
  --reranker-model cross-encoder/ms-marco-MiniLM-L6-v2 \
  --reranker-revision REPLACE_WITH_MODEL_COMMIT_SHA \
  --output /tmp/retrieval-real.json

Find each commit SHA in the revision history of the embedding model and reranker before running. The first real run downloads weights; CPU execution is supported. Save the package versions and hardware alongside the report. Query timing excludes model loading; the fixture is too small for capacity or production quality claims.

Now compare your predictions with the rankings. If fusion buries an exact identifier, inspect both lists before adding more machinery. If reranking fails, first ask whether the right document survived the top-four cut. A brilliant reviewer cannot choose a page that never reached their desk.

One answer, three checks

Metric	What it answers	What it cannot prove
Hit@1, Recall@2, MRR	Did relevant labeled sources rise to the top?	That the generated statement is supported
source_support	Does each annotated claim occur in its cited, retrieved document?	That it answers this question
answers_question	Does the first extractive answer contain the requested labeled fact?	General natural-language correctness

The answer in this experiment copies the first document. Source support can therefore be perfect while question relevance is poor. That is the point: a high grounding score can coexist with an unhelpful assistant.

The support checker uses a small, closed vocabulary of human-labeled atomic claims. Try citing refund-window while claiming unused_window_365_days; an existing ID must not rescue the invented claim. This is executable evidence beyond ID validity, not a universal entailment checker. For free-form answers, label claim/evidence pairs and evaluate a human-calibrated verifier separately.

def claims_supported(claims, documents, retrieved_ids):
    allowed = {
        doc["id"]: set(doc["claims"])
        for doc in documents
        if doc["id"] in retrieved_ids
    }
    return bool(claims) and all(
        claim["claim"] in allowed.get(claim["cite"], set())
        for claim in claims
    )

assert not claims_supported(
    [{"cite": "refund-window", "claim": "unused_window_365_days"}],
    documents,
    retrieved_ids=["refund-window"],
)

That assertion says something narrow and valuable: the cited document does not contain the annotated claim. It does not turn string membership into a general faithfulness metric.

Unanswerable queries have no positive document label, so Recall@k is undefined for them and is reported as null. They remain in the relevance results. The first run deliberately forces retrieval to expose the error; your job is to add abstention. Tune its threshold on development cases, then test on fresh answerable and unanswerable cases. Raw cosine is not a probability.

Artifact: one table comparing all four paths on a frozen query set, with candidate budgets, retrieval metrics, answer support, relevance, latency, and model revisions. Include an A17/A71 failure trace and an unanswerable query that abstains. Use new documents and at least 30 independent questions for your assessed version; the shipped twelve are a microscope, not a benchmark.

Check: pytest tests/test_retrieval_experiment.py -q.

Implementation references: SentenceTransformer and CrossEncoder. The latter scores query-document pairs jointly; its scores rank candidates rather than serving as calibrated confidence.

Quizzes

Quiz · +25 XP

Why is Reciprocal Rank Fusion preferred over averaging dense cosine scores with BM25 scores?

RRF always finds more documents than either list alone Raw scores from different retrievers are not on a shared scale; ranks are BM25 scores are probabilities and must be multiplied by cosine Cross-encoders require RRF as a mathematical precondition

Quiz · +25 XP

You change only the embedding model. Which metric should move first if the change is good?

BLEU on the final answer string Hit@k / MRR on a labeled retrieval set GPU utilization of the generator Average output tokens per response

Quiz · +25 XP

When is agentic (multi-step) RAG the wrong default?

When questions require comparing two policies When most queries are single-fact lookups and cost/latency budgets are tight When you already have a cross-encoder When Hit@k is already measured offline

---

Open source materials

Resource	Use it for
mlabonne/llm-course	RAG engineer path context
RAGAS	Faithfulness / context metrics
FAISS · Qdrant · Chroma	Vector indices
huggingface/agents-course	Agentic retrieval patterns
Sentence-transformers docs	Bi-encoders + cross-encoders
Course src/rag.py	rrf, TinyRAG teaching baseline

Also: Curated resources → RAG & embeddings.

---

Checkpoint

• You can explain hybrid search + RRF in one clear paragraph
• A metadata filter runs before top-k, and a missing key excludes the chunk
• You rerank or fuse — not only single-vector top-k
• You measure retrieval (Hit@k / MRR) separately from generation
• Multi-hop path has a step budget and logged intermediate queries
• Citations are constrained to retrieved ids
• You can show a cited answer that fails because it answers a different question

---

Mark complete when you have run a Hit@k comparison (dense vs hybrid) on a small labeled set and can defend your fusion/rerank choices.

Complete module · +120 XP

Exercise

• Catalog: EX-09 — Hybrid retrieval
• Prove: Hybrid Hit@5 / MRR is reported against dense-only on a labeled slice.
• Test: pytest tests/test_rag.py -v

Return to the case: Hybrid retrieval and reranking recover the rare identifier, and Hit@k shows whether the retrieval path improved before changing the generator. Success on this query does not guarantee coverage of the whole corpus.

Next: MCP. The corpus lesson is the parent/child and content-hash pipeline this module’s Hit@k assumes.