What you will be able to do

• Explain RAG in one sentence, then in one whiteboard diagram, to someone who has never heard of it.
• Build the naive pipeline end to end in under 80 lines, with no framework.
• Name the three things a raw model cannot know, and the reason it invents answers instead of refusing.
• Say precisely which step of the naive pipeline is hard, and why everything else in this course hangs off it.

Deep dive

A language model holds two kinds of knowledge. Parametric knowledge is whatever got compressed into the weights during training. Contextual knowledge is whatever sits in the prompt right now. RAG is the decision to stop relying on the first kind for facts and to supply them through the second.

A raw model cannot know three things: your private documents, anything after its training cutoff, and data specific to the person asking. Ask it about your enterprise refund policy and it has two options. It can decline, or it can produce the statistical average of every refund policy it has read. Training pushes it toward the second, because fluent completion is what next-token prediction rewards (module 16 covers the mechanics).

RAG in one sentence: before asking the model to answer, find the relevant documents and paste them into the prompt. Generation is the easy half. The engineering lives in "find the relevant documents".

With three documents you would paste all of them. With 300,000 you cannot: they exceed the context window, the cost per query is absurd, and irrelevant text measurably degrades answers. So retrieval is a search problem with a hard constraint: a few thousand tokens of budget, a second or two of latency, and the answer must be inside what you pick.

Why keyword search fails on its own. The user asks "Can a big client get their money back after two months?" The document says "Enterprise customers may request a refund within 90 days." There is no shared content word. This is the vocabulary mismatch problem, and it is as old as information retrieval. Embeddings (module 2) close it by comparing meaning rather than strings. Keep in mind that keyword search has not become obsolete; module 9 shows that it wins on a different class of query, which is why production systems run both.

The naive pipeline has two halves with different lifecycles:

The "same model" rule in step 5 is not a style preference. Two embedding models produce unrelated coordinate systems. A query vector from model A compared against chunk vectors from model B gives noise, and nothing errors. This is one of the first silent failures you will meet.

Build it

This is the whole thing, framework-free. embed and llm are the only two external calls; swap in any provider.

import numpy as np

def embed(texts: list[str]) -> np.ndarray:
    """Return an (n, d) array of L2-normalised vectors. Wrap your provider here."""
    raise NotImplementedError

def llm(system: str, user: str) -> str:
    raise NotImplementedError

class NaiveRAG:
    def __init__(self):
        self.chunks: list[str] = []
        self.vectors = np.empty((0, 0))

    def index(self, docs: list[str], size=500, overlap=50):
        for d in docs:
            words = d.split()
            for i in range(0, len(words), size - overlap):
                self.chunks.append(" ".join(words[i:i + size]))
        self.vectors = embed(self.chunks)            # (n, d), already normalised

    def retrieve(self, question: str, k=5):
        q = embed([question])[0]
        scores = self.vectors @ q                    # cosine == dot product when normalised
        top = np.argsort(-scores)[:k]
        return [(self.chunks[i], float(scores[i])) for i in top]

    def answer(self, question: str):
        hits = self.retrieve(question)
        context = "\n\n".join(f"[{i}] {c}" for i, (c, _) in enumerate(hits))
        system = ("Answer using only the context. If the context does not contain "
                  "the answer, say you don't have that information.")
        return llm(system, f"Context:\n{context}\n\nQuestion: {question}")

Read it as a list of decisions someone made without measuring: word-based splitting, size=500, k=5, brute-force search, no metadata, no permissions, no deletes, no logging of what was retrieved. Each later module replaces one of those guesses with a measured choice.

What breaks

• Mixed embedding models between index and query. No error, garbage results.
• Nothing relevant exists but top-k always returns k chunks. The model gets confident-looking context about the wrong thing. You need a score floor and an abstention path.
• No record of what was retrieved. When the answer is wrong you cannot tell whether retrieval or generation failed (module 11). Log chunk IDs and scores from day one.
• Index is the only copy of the text. Changing the chunker means re-parsing every source (module 6).

Lab

Create a repo called ragkit; every lab in this course extends it. Put 20 to 50 documents in data/ (your own notes, a product's docs, anything you know well enough to judge answers). Implement the class above. Write 10 questions by hand: five that reuse the documents' wording, five paraphrased the way a real user would ask. Record for each whether the right chunk appeared in the top 5. Keep this file; it becomes your eval set in module 12.

Done when: you can show a table of 10 questions with hit or miss, and you can explain at least one miss.

Interview drill

Teach it

Use the open-book exam. The model is a well-read student who has never seen your company's handbook. Closed book, the student writes something that sounds right. Open book, the student looks up the page and quotes it. RAG is handing over the right page. Then ask your listener the question that makes it stick: "What happens if we hand over the wrong page?" They will say "a wrong answer, stated confidently", and you have just motivated the rest of the course.

The misconception to correct early: people think the LLM "searches the documents". It does not. Your code searches; the model only reads what your code chose.

Check yourself

What you will be able to do

• Explain how an embedding model is trained and why that training causes its specific failure modes.
• Derive why cosine similarity equals the dot product for normalised vectors, and pick the right metric for a given index.
• Choose an embedding model on evidence: dimension, training length, query and document prefixes, domain fit.
• Predict which queries dense retrieval will get wrong before you run them.

Deep dive

What the vector is. An embedding model is a transformer encoder (or a decoder repurposed as one). Text goes in as tokens; each token gets a contextual vector; a pooling step collapses them into one fixed-size vector, usually the mean of the token vectors or the vector at a special [CLS] position. That single vector is then L2-normalised. One consequence follows directly from pooling: a long input is an average over many tokens, so distinct topics blur together. This is the mechanical reason chunking exists (module 8).

How it learns "similar". Training is contrastive. The model sees pairs that should match (a question and the passage that answers it, a title and its body, two paraphrases) and is trained so the matching pair scores higher than every other pairing in the batch. The standard loss is InfoNCE:

loss = -log( exp(sim(q, d+)/τ) / Σ_j exp(sim(q, d_j)/τ) )

where d+ is the correct passage, the sum runs over all passages in the batch (the "in-batch negatives"), and τ is a temperature. Better models add hard negatives: passages that look relevant but are not. Three interview-grade consequences:

1. Similarity is learned from data, so it reflects the training distribution. Internal codenames, error codes and SKUs were never in it. They tokenise into fragments and land somewhere arbitrary.
2. The model learned topical relatedness, because that is what most training pairs express. "Enterprise refund window" and "standard refund window" are near neighbours. Fine distinctions, negation ("not enterprise"), and numbers (30 vs 90) are weakly encoded.
3. Scores are not calibrated and not comparable across models. 0.8 means "strong" in one model and "mediocre" in another. Thresholds must be set per model on your own data.

Asymmetry. Questions and passages look different. Many models (E5, BGE, Cohere, and others) expect a prefix or an input_type flag such as query: vs passage:. Omitting it costs several points of recall and produces no error. Read the model card.

Measuring closeness. For vectors a and b:

• dot product: a·b = Σ aᵢbᵢ
• cosine: a·b / (|a||b|), the cosine of the angle, ignoring length
• Euclidean (L2): |a − b|

If both vectors have length 1, then |a − b|² = 2 − 2(a·b). All three give the same ranking. That is why people use the terms loosely, and why you should normalise once at write time and use inner product in the index, which is the cheapest to compute. If your vectors are not normalised, dot product rewards long vectors and the three metrics disagree; the index metric must match what the model was trained with.

Dimension and storage. One float32 vector of 1,536 dimensions is about 6 KB. Ten million chunks is about 61 GB before index overhead, and HNSW wants that in RAM. Options: smaller models, Matryoshka embeddings (trained so the first 256 or 512 dimensions are usable alone; truncate, then re-normalise), int8 or binary quantisation with full-precision rescoring of the top candidates.

Sequence length. Many models advertise 8K tokens but were trained mostly on 512. Quality falls beyond the training length regardless of the stated maximum. Test it; do not trust the spec sheet.

Choosing a model. Public leaderboards (MTEB) are a shortlist tool, not a decision. The decision is recall@k on your queries (module 12). Also weigh: multilingual need (Arabic plus English corpora need a genuinely multilingual model), latency of the query-side call, whether you can self-host, and the fact that switching later means re-embedding the whole corpus (module 14).

Build it

import numpy as np

def l2_normalise(x: np.ndarray) -> np.ndarray:
    return x / np.clip(np.linalg.norm(x, axis=-1, keepdims=True), 1e-12, None)

class Embedder:
    """One place that owns model id, prefixes, batching and normalisation."""
    def __init__(self, client, model_id: str, q_prefix="", d_prefix="", batch=64):
        self.client, self.model_id = client, model_id
        self.q_prefix, self.d_prefix, self.batch = q_prefix, d_prefix, batch

    def _run(self, texts):
        out = []
        for i in range(0, len(texts), self.batch):
            out.append(self.client.embed(self.model_id, texts[i:i + self.batch]))
        return l2_normalise(np.vstack(out).astype("float32"))

    def docs(self, texts):    return self._run([self.d_prefix + t for t in texts])
    def query(self, text):    return self._run([self.q_prefix + text])[0]

    def truncate(self, vecs, dims):          # only valid for Matryoshka-trained models
        return l2_normalise(vecs[..., :dims])

Store model_id next to every vector you write. At query time assert that the query embedder's model_id equals the index's. That single assertion removes the silent mixed-model failure from module 1.

What breaks

• Identifiers. ERR-4471 and ERR-4472 embed almost identically. Fix with hybrid search, not a better embedding model.
• Negation and numbers. "customers who are not enterprise" retrieves the enterprise document. Fix with a cross-encoder reranker (module 10).
• Missing prefixes or the wrong input_type. Silent recall loss.
• Un-normalised vectors in an inner-product index. Long chunks win for no semantic reason.
• Domain shift. A general model on legal or medical text under-performs. Measure first; consider a domain model or fine-tuning the embedder on your own (query, passage) pairs.
• Thresholds copied between models. 0.75 means nothing in isolation.

Lab

In ragkit, add the Embedder class. Then run three experiments on your 10 questions and write the numbers down:

1. With and without the correct query and passage prefixes.
2. Truncated to half the dimensions (if the model supports it) versus full.
3. Add five "identifier" questions (an ID, a version string, a surname) and compare their hit rate with the paraphrase questions.

Done when: you can state, with numbers, one case where embeddings clearly win and one where they clearly lose.

Interview drill

Teach it

Draw two clusters of dots on a page, refund sentences on the left, office-hours sentences on the right, and put a query dot near the left cluster. Say "real embeddings do this in a thousand dimensions". Then draw two dots almost on top of each other and label them "enterprise refund" and "standard refund". Ask "which one does the query want?" That second drawing is the more valuable one: it teaches that embeddings find the right neighbourhood and are unreliable about the right house.

Check yourself

What you will be able to do

• Explain prefill and decode, and use them to predict the latency and cost of any prompt.
• Explain the KV cache and why prompt caching requires stable content first.
• Describe what temperature, top-p and max_tokens do to the output distribution.
• Estimate tokens, cost and memory on a whiteboard without a calculator.

Deep dive

Tokens. Models read sub-word units produced by a tokenizer (byte-pair encoding or similar). In English a token is roughly 4 characters or 0.75 words; a dense page is about 500 tokens. Two warnings. Non-English text, code and identifiers tokenise less efficiently: Arabic can cost noticeably more tokens per word than English, which changes both price and how much fits in the window. And never count tokens with len(text)/4 in code that enforces a budget; use the provider's tokenizer or token-count endpoint.

Context window. The maximum tokens in one request, input plus output together. It is a hard limit with a soft problem inside it: quality degrades well before the limit (module 15).

One request, two phases.

• Prefill. The model processes the entire prompt in parallel and builds, for every layer, the key and value tensors for each token. This is compute-bound and scales with prompt length. Time to first token (TTFT) is essentially queueing plus prefill.
• Decode. The model generates one token at a time. Each step attends over all previous keys and values. This is memory-bandwidth-bound: the GPU spends its time moving weights and cache, not computing. Decode speed is quoted as tokens per second.

So: latency ≈ queue + prefill(input_tokens) + output_tokens / decode_rate. Long prompts hurt TTFT. Long answers hurt total time. Streaming does not change either number; it changes what the user perceives, because they start reading at TTFT.

KV cache. Those key and value tensors are kept so decode does not recompute them. Size per token is 2 × layers × kv_heads × head_dim × bytes. For a large model this reaches hundreds of KB per token, so a 100K-token conversation holds tens of GB. This is why long contexts are expensive to serve and why serving engines manage the cache in pages (module 37).

Prompt caching is the provider keeping the KV cache of a prompt prefix and reusing it on a later request that starts with the identical tokens. Properties that drive design:

• It matches on an exact prefix. One changed character early in the prompt invalidates everything after it. A timestamp in the system prompt gives a 0% hit rate.
• Order must be: stable instructions, tool definitions, few-shot examples, then dynamic content.
• Entries expire after minutes of disuse, and there is usually a minimum cacheable length. Some providers cache automatically; others need explicit cache markers, and cache writes can cost slightly more than normal input. Check your provider's current rules and log cached-token counts so you can see the hit rate.

Sampling. Each decode step yields logits over the vocabulary. Temperature divides the logits before softmax: p = softmax(logits / T). T→0 approaches greedy argmax; T>1 flattens the distribution. Top-p keeps the smallest set of tokens whose probability sums to p and samples within it. Temperature 0 is "mostly deterministic", not bit-exact: batching and floating-point non-associativity on GPUs can change results between runs. Build evals that tolerate this. For RAG and extraction use 0 to 0.3. Remember that temperature controls variance, not correctness: a model that is confidently wrong stays wrong at T=0.

Stopping. max_tokens caps output. Inspect finish_reason: stop is normal, length means truncated (dangerous for JSON), tool_use means the model wants a tool, content_filter means blocked. A rising length rate is an alert-worthy metric.

Roles. System, user, assistant, tool. Roles are a training convention that makes the model weight system instructions more heavily. They are not a security boundary (module 33).

Whiteboard arithmetic. 4,000 input tokens at $3 per million is $0.012; 300 output tokens at $15 per million is $0.0045. In RAG, input dominates. At 50,000 requests a day that is about $825 a day, and trimming context from 4,000 to 1,500 tokens saves roughly $375 a day. Being able to do that sum aloud is a seniority signal.

Build it

A thin client wrapper that every later module builds on. It records what you need for cost, latency and debugging.

import time, dataclasses

@dataclasses.dataclass
class LLMResult:
    text: str
    finish_reason: str
    input_tokens: int
    cached_tokens: int
    output_tokens: int
    ttft_ms: float
    total_ms: float
    cost_usd: float

PRICES = {"big":   {"in": 3.00, "cached": 0.30, "out": 15.00},   # per million tokens; load from config
          "small": {"in": 0.25, "cached": 0.03, "out": 1.25}}

def call_llm(provider, tier, stable_prefix, dynamic, max_tokens=600, temperature=0.0):
    t0 = time.perf_counter(); first = None; parts = []
    stream = provider.stream(system=stable_prefix, user=dynamic,
                             max_tokens=max_tokens, temperature=temperature)
    for ev in stream:
        if first is None and ev.text:
            first = time.perf_counter()
        parts.append(ev.text or "")
    u, p = stream.usage, PRICES[tier]
    cost = ((u.input - u.cached) * p["in"] + u.cached * p["cached"] + u.output * p["out"]) / 1e6
    return LLMResult("".join(parts), stream.finish_reason, u.input, u.cached, u.output,
                     (first - t0) * 1000 if first else float("nan"),
                     (time.perf_counter() - t0) * 1000, cost)

The signature forces callers to separate stable_prefix from dynamic. That is a small API decision that makes caching work by construction instead of by discipline.

What breaks

• A date, request ID or user name interpolated into the system prompt: cache hit rate is zero and nobody notices because nothing fails.
• Tool definitions serialised from a dict with unstable key order: same effect.
• Budgeting with character counts, then hitting a hard context error in production on non-English input.
• Ignoring finish_reason == "length" and parsing a truncated JSON object.
• Treating temperature 0 as reproducible and writing exact-match tests against free text.

Lab

Add call_llm to ragkit. Send the same 3,000-token stable prefix 20 times with different questions and log cached_tokens, TTFT and cost. Then put a timestamp at the top of the prefix and repeat. Plot or tabulate both runs.

Done when: you can show the hit rate collapsing and state the cost difference per 10,000 requests.

Interview drill

Teach it

Compare it to reading and then writing an essay. Prefill is reading the whole brief once, quickly, because you can skim in parallel. Decode is writing the essay one word at a time, glancing back at your notes (the KV cache) before each word. Prompt caching is a colleague who already read the first ten pages of the brief and hands you their notes, which only helps if the first ten pages are exactly the same ones.

Check yourself

What you will be able to do

• Explain HNSW construction and search, and tune M, ef_construction and ef_search against measured recall.
• Explain IVF and product quantisation, including the arithmetic of the compression.
• Decide between brute force, pgvector, and a dedicated vector database for a given scale.
• Measure ANN recall yourself instead of trusting defaults.

Deep dive

Brute force is a real option. Exact search is one matrix-vector product. 100,000 vectors of 1,024 float32 dimensions is 400 MB and about 10⁸ multiply-adds per query, which a CPU with BLAS does in a few milliseconds. It is exact, has no index to rebuild, handles filters trivially and deletes are free. Below roughly 100K vectors per searchable partition it is often the right answer. Note "per partition": a multi-tenant system with 5,000 tenants of 20K chunks each is a brute-force problem if you partition by tenant.

Why approximate. At 10⁷ to 10⁹ vectors, exact search costs too much per query. ANN gives up a little recall (finding 98 of the true top 100) for a 100× or greater speedup. For retrieval feeding a reranker, that trade is nearly always right.

HNSW. Build a graph where each vector links to near neighbours. Search is greedy: from an entry point, move to whichever neighbour is closer to the query, repeat until no neighbour improves. Greedy walks on one flat graph get stuck or take long routes, so HNSW stacks layers.

• Insertion. Each new node draws a top level l = floor(−ln(U) · mL) with U uniform in (0,1) and mL = 1/ln(M). Most nodes live only on layer 0; exponentially fewer reach higher layers. This is the skip-list idea applied to a proximity graph.
• From the top layer down to l+1, do a greedy walk to find a good entry point. From layer l down to 0, run a beam search of width ef_construction, pick up to M neighbours with a heuristic that prefers diverse directions (not just the closest, which would all sit in one cluster), and link both ways, pruning neighbours that exceed the cap (M, and 2M on layer 0).
• Search. Greedy descent through the upper layers, then a beam search of width ef_search on layer 0 that maintains a candidate heap and a result heap, stopping when the best unexplored candidate is worse than the worst kept result.

Knobs: M (typically 16 to 64) raises recall and memory; ef_construction (100 to 400) raises graph quality and build time; ef_search is the runtime dial and must be at least k. Cost is roughly logarithmic in N.

Memory: vectors plus links. Links cost about M × 2 × 4 bytes per node on layer 0. For 10M vectors at 1,024 dims that is ~41 GB of vectors plus ~1 to 3 GB of graph, all wanting RAM. Disk-based variants (DiskANN-style) exist for when it does not fit.

Deletes: removing a node can disconnect the graph, so engines tombstone. Dead nodes still occupy memory and are still traversed. Recall and latency drift until you compact or rebuild (module 14).

IVF. Run k-means to get nlist centroids (rule of thumb: near √N up to a few times that). Each vector is assigned to its nearest centroid's list. At query time, find the nprobe closest centroids and scan only those lists. nprobe is the recall dial. The failure is the boundary case: the true neighbour sits in a cell you did not probe. Centroids also go stale as the corpus shifts, so retrain periodically. IVF builds faster and uses less memory than HNSW; at equal latency it usually has lower recall.

Product quantisation. Split each d-dimensional vector into m sub-vectors. For each sub-space learn a codebook of 256 centroids by k-means. Store each sub-vector as the 1-byte index of its nearest centroid. A 1,024-dim float32 vector (4,096 bytes) with m = 32 becomes 32 bytes: 128× smaller. Distances are computed by asymmetric distance computation: for a query, precompute an m × 256 table of distances from each query sub-vector to each codeword, then a stored vector's distance is m table lookups added together. Recall drops; you recover it by over-fetching (say the top 200 by PQ distance) and rescoring with full-precision vectors kept on disk.

Simpler compression is often enough: int8 scalar quantisation (4× smaller, tiny loss) or binary quantisation (32× smaller, needs rescoring, works best with high-dimension models).

Filtering and ANN. Pre-filtering to a tiny subset shatters graph connectivity; post-filtering can return nothing. Good engines evaluate the predicate during traversal, and fall back to brute force on the filtered subset when it is small. Module 13 depends on this.

Picking infrastructure.

Build it

Measure ANN recall against exact search. This harness is the only honest way to set ef_search.

import numpy as np, hnswlib, time

def exact_topk(vecs, queries, k):
    return np.argsort(-(queries @ vecs.T), axis=1)[:, :k]

def ann_recall(vecs, queries, k=10, M=32, efc=200, ef_values=(16, 32, 64, 128, 256)):
    index = hnswlib.Index(space="ip", dim=vecs.shape[1])        # vectors are normalised
    index.init_index(max_elements=len(vecs), M=M, ef_construction=efc)
    index.add_items(vecs, np.arange(len(vecs)))
    truth = exact_topk(vecs, queries, k)
    rows = []
    for ef in ef_values:
        index.set_ef(ef)
        t0 = time.perf_counter()
        labels, _ = index.knn_query(queries, k=k)
        ms = (time.perf_counter() - t0) * 1000 / len(queries)
        recall = np.mean([len(set(a) & set(b)) / k for a, b in zip(labels, truth)])
        rows.append((ef, round(float(recall), 4), round(ms, 3)))
    return rows        # pick the smallest ef that clears your recall target

Two different recalls exist in this course and interviewers like candidates who keep them apart. ANN recall is approximate versus exact nearest neighbours, a property of the index. Retrieval recall@k is whether the relevant chunk was returned, a property of the whole system. An index with 99% ANN recall can sit under a system with 60% retrieval recall.

What breaks

• Defaults left untouched, so nobody knows the ANN recall.
• ef_search < k, which silently truncates results in some engines.
• Heavy delete or update churn with no compaction: rising latency, falling recall, no errors.
• A highly selective filter on a graph index: recall collapse or empty results.
• Index metric not matching vector normalisation.
• Sizing RAM for vectors and forgetting graph overhead, replicas and the rebuild-in-parallel copy.

Lab

Embed at least 20,000 chunks (duplicate your corpus with public text if needed). Run ann_recall for M ∈ {8, 32}. Produce a recall-versus-latency table. Then delete 30% of items with mark_deleted, re-measure, rebuild, measure again.

Done when: you can point to the ef_search you would ship and justify it with numbers, and you have seen the effect of tombstones yourself.

Interview drill

Teach it

Use travel. To reach a village in another country you take a flight between major hubs (top layer), a regional train (middle layer), then a local bus that stops everywhere (layer 0). Checking every village on earth is brute force. IVF is sorting villages into provinces and only searching the two or three nearest provinces, which fails when your village sits just across a provincial border. PQ is replacing each address with a short postcode: far smaller, slightly lossy, so you confirm the final few with the full address.

Check yourself

What you will be able to do

• Write the BM25 formula, explain each term, and say what k1 and b control.
• Build an inverted-index scorer and know which analyser choices matter more than the formula.
• Explain learned sparse retrieval (SPLADE-style) and where it fits.
• Decide tokenisation rules for identifiers, code and non-English text.

Deep dive

The representation. A sparse retriever treats a document as a bag of terms. As a vector over the vocabulary it is almost entirely zeros, hence "sparse". The data structure is an inverted index: for each term, a postings list of (document, term frequency). A query touches only the postings of its own terms, which is why keyword search is fast and cheap at very large scale.

The formula. For query Q and document D:

score(D,Q) = Σ_t IDF(t) · f(t,D)·(k1+1) / ( f(t,D) + k1·(1 − b + b·|D|/avgdl) )

IDF(t) = ln( (N − n_t + 0.5)/(n_t + 0.5) + 1 )

• f(t,D) is how often the term occurs in the document. The fraction saturates: the first occurrence matters a lot, the tenth barely at all. k1 (usually 1.2 to 2.0) sets how fast. This stops keyword stuffing from winning, which is the core improvement over raw TF-IDF.
• IDF rewards rare terms. N is the corpus size, n_t the number of documents containing the term. "refund" in a corpus about refunds is worth little; XR-4471 is worth a great deal.
• |D|/avgdl with b (usually 0.75) is length normalisation. Long documents contain more terms by chance, so they are penalised. For uniform chunks, lower b (0.3 to 0.5) is often better because length no longer signals padding.

The analyser matters more than the formula. BM25 only sees what the tokenizer emits.

• Lowercasing and Unicode normalisation: almost always.
• Stemming or lemmatisation helps recall in English ("refunds" → "refund"). For Arabic and other morphologically rich languages use a language-specific analyser; whitespace splitting alone performs badly because clitics attach to words.
• Identifiers. A standard tokenizer splits XR-4471 into xr and 4471, so the query matches any document containing either. Index a second field with a whitespace or pattern analyser that keeps the whole token, and boost it. The same applies to version strings, SKUs, file paths and email addresses.
• Stop words: removing them saves space and occasionally destroys meaning ("to be or not to be", "IT department"). With BM25's IDF weighting you can usually keep them.
• Fields: title and headings deserve their own field with a boost (BM25F generalises this).

Where sparse wins and loses. It wins on exact identifiers, names, rare jargon, quoted phrases, and as an explainable baseline (you can show which term matched). It loses completely on paraphrase: "how do I get my money back" against a document titled "Refund Request Procedure" scores zero. This is the mirror image of the dense failure from module 2, and the reason hybrid search works (module 9).

Learned sparse retrieval. Models such as SPLADE use a transformer to output term weights over the vocabulary, including terms that do not appear in the text ("refund" gets weight for a passage that says "money back"). You keep an inverted index and exact matching while gaining some semantic expansion. Costs: a model at index time and query time, larger postings. Worth knowing as the middle ground; BM25 plus dense remains the pragmatic default.

Where to run it. Elasticsearch or OpenSearch if you have them. Postgres full-text search is usable, though its built-in ranking is not BM25 (extensions add it). Several vector databases now include a sparse or BM25 index so both searches live in one system. For a small corpus, an in-process library is fine.

Build it

A compact, correct BM25 over an inverted index, with a second analyser that preserves identifiers.

import math, re
from collections import Counter, defaultdict

WORD = re.compile(r"[a-z0-9]+")
IDENT = re.compile(r"[A-Za-z]+[-_./]?\d[\w\-./]*")       # XR-4471, v2.14.3, SKU_9912

def analyse(text: str) -> list[str]:
    toks = WORD.findall(text.lower())
    toks += ["id:" + m.lower() for m in IDENT.findall(text)]   # whole identifier kept as one term
    return toks

class BM25:
    def __init__(self, k1=1.5, b=0.75):
        self.k1, self.b = k1, b
        self.postings = defaultdict(dict)       # term -> {doc_id: tf}
        self.length = {}

    def add(self, doc_id, text):
        toks = analyse(text)
        self.length[doc_id] = len(toks)
        for t, tf in Counter(toks).items():
            self.postings[t][doc_id] = tf

    def remove(self, doc_id):
        self.length.pop(doc_id, None)
        for plist in self.postings.values():
            plist.pop(doc_id, None)

    def search(self, query, k=50):
        N = len(self.length)
        avgdl = sum(self.length.values()) / max(N, 1)
        scores = defaultdict(float)
        for t in set(analyse(query)):
            plist = self.postings.get(t)
            if not plist: continue
            idf = math.log((N - len(plist) + 0.5) / (len(plist) + 0.5) + 1)
            for d, tf in plist.items():
                norm = tf + self.k1 * (1 - self.b + self.b * self.length[d] / avgdl)
                scores[d] += idf * tf * (self.k1 + 1) / norm
        return sorted(scores.items(), key=lambda x: -x[1])[:k]

Note that the query is run through the same analyser as the documents. An index-time and query-time analyser mismatch is the sparse equivalent of mixing embedding models.

What breaks

• Identifier splitting, as above. The most common real defect.
• Different analysers at index and query time.
• Chunk-level BM25 with a contextual prefix on every chunk (module 8): the prefix words appear everywhere and IDF correctly ignores them, but a shared boilerplate footer of rare words will not be ignored. Strip boilerplate before indexing.
• Forgetting deletes in the keyword index when you delete from the vector index. There are two derived stores; both must be updated.
• Assuming BM25 scores are comparable with cosine scores. They are unbounded and corpus-dependent (module 9).

Lab

Add BM25 to ragkit, indexing the same chunks as the vector store under the same chunk IDs. Run your identifier questions and your paraphrase questions through both retrievers and fill a 2×2 table: which retriever found which question type. Then remove the IDENT rule and watch the identifier results degrade.

Done when: your table shows the complementary failure pattern with your own data.

Interview drill

Teach it

Open a textbook's back-of-book index. Each term lists the pages it appears on; that is an inverted index. Now ask "which page is best for refund deadline?" Pages that mention both terms beat pages with one; a term that appears on nearly every page tells you nothing; a 40-page chapter that mentions "refund" once is a worse bet than a half-page note that mentions it twice. Those three intuitions are IDF, saturation and length normalisation.

Check yourself

What you will be able to do

• Draw the two systems, ingestion and query, and justify why they are separate deployables.
• Explain the three-store design and why the vector index is a materialised view.
• Build an ingestion pipeline with per-document isolation, idempotency and a dead-letter queue.
• Present the prototype-versus-production comparison from memory.

Deep dive

Two systems with different operational profiles.

Merging them is the common structural mistake. An ingestion bug then takes search down, or a query-side optimisation breaks re-indexing. Separate services, a shared library for the things that must match on both sides (the embedder wrapper, the text analyser, the chunk ID scheme).

Ingestion stages. sources → change detection → parse → chunk → enrich → embed → index

• Sources and connectors. One connector per system (SharePoint, Drive, Confluence, S3, ticketing, a database). A connector's contract: list documents with stable IDs, modified times and ACLs; fetch content; ideally expose a change feed.
• Change detection. Hash content; process only new and changed documents; detect deletes. Detail in module 14.
• Parse, chunk, enrich. Modules 7 and 8.
• Embed. Batched, rate-limited, checkpointed. Record the embedding model ID on every vector.
• Index. Write to all derived stores under one chunk ID.

Query stages. query → rewrite → [dense ∥ sparse] → fuse → filter → rerank → assemble → generate → verify

Rewrite turns a follow-up ("what about for enterprise?") into a standalone question using conversation history. Fuse merges the two ranked lists. Filter applies metadata and permissions. Rerank rescores candidates with a cross-encoder. Assemble orders and formats context. Verify checks that citations refer to chunks that were really retrieved.

Three stores.

1. Document store: parsed text, structure, metadata, ACL snapshot, content hash. Postgres or object storage. Durable and boring. This is the source of truth for the RAG system.
2. Vector index: embeddings keyed by chunk ID.
3. Keyword index: BM25 postings keyed by the same chunk ID.

Stores 2 and 3 are derived. You will rebuild them whenever the embedding model, chunker or enrichment logic changes. If the only copy of parsed text is inside the vector database, every rebuild means re-parsing thousands of PDFs, which is slow, costs money (OCR, layout models), and may be impossible if the source has moved on.

Say it: The vector index is a materialised view. The document store is the source of truth.

Go one level finer and store parsed output and chunks as separate tables. A chunker change then needs no re-parse, and an embedder change needs no re-chunk. Each stage's output is cached by a key that includes the stage's version.

Idempotency and identity. Chunk IDs should be deterministic: hash(doc_id, chunker_version, chunk_ordinal) or a content hash. Then a retried batch overwrites instead of duplicating, and you can diff two index versions.

Per-document isolation. One malformed PDF must not end a ten-hour run. Wrap each document in its own try block, send failures to a dead-letter queue with the error and stage, continue, and alert on DLQ growth rate.

What "production" means.

Build it

The ingestion worker's core. Each stage is a pure function from the previous stage's stored output, so any stage can be re-run alone.

import hashlib, traceback

def sha(b: bytes) -> str: return hashlib.sha256(b).hexdigest()

def ingest_document(src_doc, stores, cfg):
    """Idempotent. Safe to retry. Returns 'skipped' | 'indexed'."""
    raw = src_doc.fetch()
    h = sha(raw)
    rec = stores.docs.get(src_doc.id)
    if rec and rec.content_hash == h and rec.pipeline_version == cfg.version:
        return "skipped"

    parsed = parse(raw, src_doc.mime)                 # module 7
    validate_parse(parsed, len(raw))                  # raises on suspicious output
    chunks = chunk(parsed, cfg.chunker)               # module 8
    for i, c in enumerate(chunks):
        c.id = sha(f"{src_doc.id}|{cfg.version}|{i}".encode())[:24]
        c.meta = {"doc_id": src_doc.id, "title": parsed.title, "section": c.section,
                  "modified_at": src_doc.modified_at, "acl": src_doc.acl}
    vectors = cfg.embedder.docs([c.embed_text for c in chunks])

    stores.docs.put(src_doc.id, parsed, content_hash=h, pipeline_version=cfg.version)
    old_ids = stores.docs.chunk_ids(src_doc.id)
    stores.vector.delete(old_ids);  stores.keyword.delete(old_ids)      # delete then insert
    stores.vector.upsert(chunks, vectors, model_id=cfg.embedder.model_id)
    stores.keyword.upsert(chunks)
    stores.docs.set_chunk_ids(src_doc.id, [c.id for c in chunks])
    return "indexed"

def run(connector, stores, cfg, dlq):
    seen = set()
    for d in connector.list_documents():
        seen.add(d.id)
        try:
            ingest_document(d, stores, cfg)
        except Exception as e:
            dlq.put({"doc_id": d.id, "error": repr(e), "trace": traceback.format_exc()})
    for gone in stores.docs.all_ids(source=connector.name) - seen:       # the forgotten case
        ids = stores.docs.chunk_ids(gone)
        stores.vector.delete(ids); stores.keyword.delete(ids); stores.docs.delete(gone)

One safety rule for the delete sweep: if the connector's listing fails or returns suspiciously few documents, abort the sweep. Otherwise a source outage reads as "everything was deleted" and you wipe the index.

What breaks

• A listing API outage interpreted as mass deletion.
• Non-deterministic chunk IDs, so retries create duplicates that dominate search results.
• The shared analyser or embedder diverging between the two services.
• A DLQ nobody reads. Alert on its growth and review it weekly.
• Writes to the vector index succeed and the keyword index fails, leaving the two out of step. Make the write a retried unit and reconcile periodically by comparing ID sets.

Lab

Restructure ragkit into ingest/ and query/ packages with a shared common/. Add a SQLite document store holding parsed text, hash and chunk IDs. Make ingestion idempotent and prove it: run it twice and assert zero writes the second time. Delete a source file, run again, and assert its chunks are gone from both indexes. Feed it a corrupt file and confirm the run completes with one DLQ entry.

Interview drill

Teach it

A library has a stacks room with the actual books and a card catalogue. When you reorganise the catalogue you do not reprint the books. The catalogue can be rebuilt from the stacks at any time, and the reverse is not true. Then add the second idea: cataloguing happens in the back office on its own schedule; the reference desk answers patrons in seconds. If the same clerk does both, a delivery of new books stops the desk.

Check yourself

What you will be able to do

• Explain why parsing quality is the ceiling on RAG quality, with concrete examples.
• Route documents to the right parser by type and detect when a parser has failed silently.
• Preserve structure (headings, tables, lists, reading order) through to the chunker.
• Build automated parse validation and a sampling review process.

Deep dive

Everything downstream sees only the parser's output. If the text is scrambled, embeddings are noise, BM25 indexes nonsense, and the LLM reads nonsense. No retrieval tuning recovers it. Teams routinely spend weeks on prompts for a problem that lives here.

Why PDFs are hard. A PDF is a list of drawing instructions: place these glyphs at these coordinates. It has no paragraphs, no reading order, no table cells. Extractors reconstruct structure from geometry, and that reconstruction fails in recognisable ways:

• Two-column layout read straight across. "Enterprise refund window is Standard refund window / 90 days from purchase. is 30 days from purchase." Fluent-looking garbage.
• Tables flattened. "Tier Window Processing Enterprise 90 days 10 days Standard 30 days 15 days". A model may pair Standard with 10 days.
• Scanned pages. No text layer. A naive extractor returns an empty string, nothing errors, and that region of the corpus silently does not exist.
• Headers, footers and page numbers interleaved mid-sentence and repeated on every page, polluting BM25 and embeddings.
• Hyphenation and ligatures: "refund-\nable", "finance" as a single odd glyph.
• Multi-page tables that lose their header row after page one.

Routing. Use the cheapest parser that works for each document class, and detect the class rather than assuming it.

Decide per page, not per file: a contract may be digital with two scanned signature pages.

Target format. Parse into a structured intermediate, not a string: an ordered list of blocks, each with a type (heading with level, paragraph, table, list, code, figure caption), text, page number and bounding box. Serialise tables as Markdown or HTML so cell relationships survive; for wide tables, consider emitting one natural-language row per record ("Tier: Enterprise; Window: 90 days; Processing: 10 days") because it both embeds and reads well. This intermediate is what you keep in the document store.

Validation. Build the check instead of finding out by accident.

• Characters extracted per page and per megabyte. A 2 MB PDF that yields 40 characters is a failure.
• Ratio of alphanumeric to other characters; share of tokens found in a dictionary for the expected language.
• Repeated-line detection across pages to strip headers and footers.
• Empty-page rate per source.
• For OCR, mean confidence.
• A weekly human sample per source type, comparing the rendered page with the parsed output side by side. Twenty documents is enough to find systematic faults.

Failures route to a more capable parser or to the dead-letter queue. They never enter the index quietly.

Build it

import re, fitz                      # PyMuPDF
from collections import Counter

class ParseError(Exception): ...

def page_needs_ocr(page) -> bool:
    text = page.get_text("text").strip()
    has_big_image = any(img for img in page.get_images(full=True))
    return len(text) < 40 and has_big_image

def parse_pdf(data: bytes, layout_parser, ocr):
    doc = fitz.open(stream=data, filetype="pdf")
    blocks = []
    for page in doc:
        if page_needs_ocr(page):
            blocks += ocr(page)                               # scanned page
        elif looks_complex(page):
            blocks += layout_parser(page)                     # tables / multi-column
        else:
            blocks += [{"type": "paragraph", "text": b[4], "page": page.number, "bbox": b[:4]}
                       for b in page.get_text("blocks", sort=True) if b[4].strip()]
    return strip_repeated_lines(blocks, n_pages=len(doc))

def looks_complex(page) -> bool:
    xs = sorted(b[0] for b in page.get_text("blocks"))
    two_columns = len(xs) > 6 and (xs[-1] - xs[0]) > page.rect.width * 0.35
    return two_columns or bool(page.find_tables().tables)

def strip_repeated_lines(blocks, n_pages, share=0.6):
    norm = lambda t: re.sub(r"\d+", "#", t.strip().lower())
    counts = Counter(norm(b["text"]) for b in blocks if len(b["text"]) < 120)
    boiler = {t for t, c in counts.items() if n_pages > 3 and c >= share * n_pages}
    return [b for b in blocks if norm(b["text"]) not in boiler]

def validate_parse(blocks, n_bytes, n_pages):
    text = " ".join(b["text"] for b in blocks)
    if n_pages and len(text) / n_pages < 80:
        raise ParseError("too little text per page")
    alnum = sum(ch.isalnum() or ch.isspace() for ch in text) / max(len(text), 1)
    if alnum < 0.8:
        raise ParseError(f"junk ratio too high ({alnum:.2f})")

looks_complex is a heuristic; its job is to be cheap and to err toward the better parser. Tune it per source using the DLQ and your sample reviews.

What breaks

• Silent empty output from scans. The single most common hidden hole in a corpus.
• Reading-order scrambling that still looks like English, so nobody notices.
• Tables losing header association; numbers attached to the wrong label.
• Boilerplate repeated on every page dominating keyword search.
• Vision-model OCR inventing or "correcting" digits in amounts and IDs. Cross-check critical fields with a second method.
• Treating parsing as done. New sources and templates arrive; keep the validation metrics on a dashboard per source.

Lab

Collect six hard documents: a two-column paper, a PDF with a table, a scanned page, a slide deck exported to PDF, an HTML page with navigation chrome, and an email thread. Parse each with your fast path and print the output next to the original. Record what broke. Add routing and validate_parse, and confirm the scan is caught. Add a table-to-Markdown path and ask the model a question whose answer requires reading a table cell correctly, before and after.

Interview drill

Teach it

Give a colleague a newspaper page and ask them to read it aloud by moving a ruler straight across both columns. The result sounds like language and means nothing. That is what a naive PDF extractor hands your embedding model. The misconception to correct: "the LLM is smart, it will figure it out". It cannot recover a table association that was destroyed before it saw the text, and it will not tell you it is guessing.

Check yourself

What you will be able to do

• Explain the precision-versus-completeness tension from the mechanics of pooling.
• Implement structure-aware chunking with a recursive fallback and atomic tables and code blocks.
• Implement contextual retrieval with prompt caching, and parent-document retrieval.
• Choose chunk size by experiment against a retrieval eval set.

Deep dive

The tension. An embedding is one fixed-size vector, mean-pooled over the input. A long passage covering refunds, shipping and privacy produces a vector that sits between all three and close to none. A very short passage ("It must be requested within 90 days.") has no topic signal and no standalone meaning. You need chunks small enough to embed precisely and large enough to be a complete thought.

Strategies, weakest to strongest.

1. Fixed size with overlap (500 tokens, 50 overlap). Overlap means a sentence cut at a boundary survives intact in one of the two chunks. It ignores structure and will cut tables and functions. A baseline only.
2. Recursive splitting. Try paragraph breaks; if a piece is still too large, try line breaks, then sentences, then words. A sensible default for unstructured text.
3. Structure-aware. Split on the document's own hierarchy: Markdown and HTML headings, PDF outline entries, function and class boundaries from tree-sitter. The author already marked where topics change; that signal is free and accurate. Carry the heading path ("Refund Policy > Enterprise > Eligibility") into each chunk.
4. Semantic chunking. Embed sentences and split where consecutive similarity drops. Attractive in theory; in practice slow, threshold-sensitive and often no better than structure-aware. State that you would measure before adopting it.

Combine 3 and 2: split by structure, then recursively split any section that is still too large, then merge tiny adjacent sections under the same parent.

Rules that matter more than the size number.

• Tables, code blocks and lists are atomic. A half table looks complete and is wrong. If a table is too large, split by rows and repeat the header row in each part.
• Different document types get different configurations.
• Keep provenance: document ID, page, heading path, character offsets. You will need to point at the source.
• Distinguish embed_text (what gets embedded and keyword-indexed) from display_text (what the model and user see). This one separation enables the two techniques below.
• Put metadata used for filtering in fields, not in the text, or a metadata change forces re-embedding (module 14). A short title and heading prefix in embed_text is worth it; ACLs and dates in the text are not.

Contextual retrieval. A chunk out of its document loses its referents: "The window was extended to 90 days following customer feedback in Q2." Whose window? Before embedding, ask an LLM for one or two sentences situating the chunk within the whole document, and prepend them to embed_text for both the vector and BM25 indexes. Anthropic's published results: about 35% fewer retrieval failures from contextual embeddings alone, about 49% with contextual BM25 added, about 67% with reranking on top. The cost is one LLM call per chunk with the full document in the prompt. Because the document is identical across all its chunks, put it first and mark it cacheable: you pay full price once per document and the cached rate for the rest. Use a small model. For documents larger than the context window, use the enclosing section plus the document's outline.

Small-to-big (parent document retrieval). Match on small units, read large ones. Index 150 to 250-token children, each with a parent_id pointing at its 1,000 to 2,000-token section. After retrieval, map children to parents, deduplicate, and send parents to the reranker or the model. A variant, sentence-window, returns the matched sentence plus its neighbours. You get precise matching and complete context, at the price of more tokens per hit; control it with a parent token cap.

Late chunking is worth naming: run a long-context embedding model over the whole document, then pool token vectors per chunk, so each chunk vector already carries document context. It needs a model that supports it and is less proven than contextual retrieval.

Choosing a size. Start at 400 to 800 tokens with 10 to 20% overlap. Then run the experiment: three or four configurations, same eval set, compare recall@k and downstream faithfulness. Narrow factual questions favour smaller chunks; questions needing reasoning across a section favour larger ones or small-to-big. Check the embedding model's real training length (module 2).

Build it

import re
from dataclasses import dataclass, field

@dataclass
class Chunk:
    display_text: str
    heading_path: list[str]
    page: int
    parent_id: str | None = None
    context: str = ""                      # filled by contextualise()
    meta: dict = field(default_factory=dict)
    @property
    def embed_text(self):
        head = " > ".join(self.heading_path)
        return "\n".join(p for p in (self.context, head, self.display_text) if p)

SEPS = ["\n\n", "\n", ". ", " "]
def recursive_split(text, max_tok, count, seps=SEPS):
    if count(text) <= max_tok or not seps:
        return [text]
    out, buf = [], ""
    for p in text.split(seps[0]):
        cand = buf + seps[0] + p if buf else p
        if count(cand) <= max_tok:
            buf = cand
            continue
        if buf:
            out.append(buf); buf = ""
        if count(p) <= max_tok:
            buf = p
        else:
            out += recursive_split(p, max_tok, count, seps[1:])
    if buf:
        out.append(buf)
    return out

def chunk_blocks(blocks, count, max_tok=600):
    path, chunks, buf = [], [], []
    def flush(page):
        if buf:
            for piece in recursive_split("\n\n".join(buf), max_tok, count):
                chunks.append(Chunk(piece, list(path), page))
            buf.clear()
    for b in blocks:
        if b["type"] == "heading":
            flush(b["page"]); path[:] = path[: b["level"] - 1] + [b["text"]]
        elif b["type"] in ("table", "code"):
            flush(b["page"]); chunks.append(Chunk(b["text"], list(path), b["page"]))   # atomic
        else:
            buf.append(b["text"])
            if count("\n\n".join(buf)) > max_tok: flush(b["page"])
    flush(blocks[-1]["page"] if blocks else 0)
    return chunks

CTX_PROMPT = ("Here is a chunk from the document above:\n<chunk>\n{chunk}\n</chunk>\n"
              "Write one or two sentences situating this chunk within the document, to improve "
              "search retrieval. Name the document, the entity and the topic. Answer with only that.")

def contextualise(doc_text, chunks, small_llm):
    for c in chunks:                                   # doc_text is the cached prefix
        c.context = small_llm(cached_prefix=f"<document>\n{doc_text}\n</document>",
                              dynamic=CTX_PROMPT.format(chunk=c.display_text)).strip()

For small-to-big, run chunk_blocks twice: once at the parent size, then split each parent into children that inherit parent_id. Only children go into the indexes; parents live in the document store.

What breaks

• Tables and code split in the middle.
• Overlap so large that near-duplicate chunks fill the top-k (fix with deduplication and MMR, module 10).
• Context prefixes generated by a model that invents facts. Keep them short and descriptive; review a sample.
• Parents so large that three hits exceed the context budget.
• One configuration for every document type.
• Changing the chunker without re-running the retrieval eval, so a recall regression ships silently.

Lab

Implement structure-aware chunking and contextual prefixes in ragkit. Run a four-way experiment on your eval questions: fixed 500/50, structure-aware, structure-aware with context, and small-to-big. Report recall@5 and recall@20 for each. Read ten chunks from each configuration with your own eyes; you will learn more from that than from the numbers.

Interview drill

Teach it

Cut a policy handbook into index cards. Cards that hold a whole page are hard to file, because each covers three topics. Cards that hold one sentence are easy to file and useless to read ("within 90 days": of what?). Contextual retrieval is writing "Acme 2025 Refund Policy, enterprise section" along the top of each card. Small-to-big is filing by the card and, when one matches, fetching the whole page it was cut from.

Check yourself

What you will be able to do

• Explain why combining two retrievers helps only when their failures do not overlap, and show that they do not.
• Compute Reciprocal Rank Fusion by hand and implement it for any number of retrievers.
• Explain weighted score fusion, when to prefer it, and why it is fragile.
• Build conversation-aware query rewriting and know when it hurts.

Deep dive

Complementary failure. Dense retrieval fails on tokens the embedding model never learned: XR-4471, v2.14.3, surnames, codenames, statute numbers, drug codes. Sparse retrieval fails on paraphrase: "how do I get my money back" against "Refund Request Procedure". An ensemble gains only when members err on different inputs, and these two barely overlap. You showed this with your own data in module 5; say so in interviews, because "I measured it" beats "people say so".

The fusion problem. BM25 scores are unbounded and corpus-dependent (14.7, 9.2, 3.1). Cosine scores are bounded and model-dependent (0.87, 0.81, 0.62). Adding them is meaningless.

Reciprocal Rank Fusion discards scores and uses ranks:

RRF(d) = Σ_r 1 / (k + rank_r(d)), with k ≈ 60

Worked example. Dense: A, B, C. Sparse: C, A, D.

A is high in both lists and wins. Items in only one list score about half. The constant k flattens the curve: with k = 60, rank 1 versus rank 2 differ by under 2%, so one retriever's top pick cannot dominate; with k = 1 the top rank would be worth far more than everything else. RRF needs no normalisation, no tuning, survives score-distribution shift, and extends to any number of lists (several query rewrites, a title-field search, a learned-sparse retriever).

Weighted score fusion. Normalise each retriever's scores (min-max or z-score over the returned list), then α·dense + (1−α)·sparse. It can beat RRF by a little when tuned per domain, and it lets you shift weight per query type (raise sparse weight when the query contains an identifier). The costs: α must be calibrated on an eval set, and min-max over a returned list is unstable, since one outlier compresses everything else. A weighted RRF (w_r / (k + rank)) is a sturdier way to get the same control.

Over-retrieve. Pull 50 to 100 from each retriever. Fusion and the reranker need candidates to work with, and ANN search cost barely changes between k=10 and k=50.

Query rewriting. Retrieval sees one string. In a conversation that string is often "what about for enterprise?", which retrieves nothing useful. A small model rewrites the latest message into a standalone question using recent turns. Related techniques to know:

• Decomposition: split "compare the enterprise and standard refund windows" into two lookups and retrieve for each.
• Multi-query: generate a few paraphrases, retrieve for each, fuse with RRF. Helps recall, costs latency.
• HyDE: have the model write a hypothetical answer and embed that, since an answer looks more like a passage than a question does. Helps on some corpora; risky when the model's guess pulls retrieval toward a wrong topic.

Rewriting can also do harm: it may drop the identifier the user typed, or "improve" a precise query into a vague one. Send the original and the rewrite to retrieval and fuse; skip rewriting on the first turn; log both strings.

Build it

import asyncio
from collections import defaultdict

def rrf(ranked_lists: list[list[str]], k: int = 60, weights=None) -> list[tuple[str, float]]:
    weights = weights or [1.0] * len(ranked_lists)
    score = defaultdict(float)
    for w, lst in zip(weights, ranked_lists):
        for rank, doc_id in enumerate(lst, start=1):
            score[doc_id] += w / (k + rank)
    return sorted(score.items(), key=lambda x: -x[1])

REWRITE = """Rewrite the user's last message as a standalone search query.
Keep every identifier, number, name and product term exactly as written.
If it is already standalone, return it unchanged. Output only the query.

Conversation:
{history}

Last message: {message}"""

async def hybrid_retrieve(message, history, stores, embedder, small_llm, k_each=50):
    queries = [message]
    if history:
        rewritten = (await small_llm(REWRITE.format(history=history[-6:], message=message))).strip()
        if rewritten and rewritten.lower() != message.lower():
            queries.append(rewritten)

    async def one(q):
        dense, sparse = await asyncio.gather(
            stores.vector.search(embedder.query(q), k=k_each),
            stores.keyword.search(q, k=k_each))
        return [ids(dense), ids(sparse)]

    lists = [l for pair in await asyncio.gather(*(one(q) for q in queries)) for l in pair]
    return rrf(lists), {"queries": queries, "lists": lists}      # second value goes to the trace

def ids(hits): return [h.chunk_id for h in hits]

Dense and sparse run concurrently, as do the searches for each query variant. The function returns the per-retriever lists for the trace; without them you cannot later say which retriever found (or missed) the right chunk.

What breaks

• Adding raw scores from two retrievers.
• Chunk IDs that differ between the vector and keyword stores, so fusion never sees agreement.
• Retrieving only 5 to 10 per side, leaving fusion nothing to work with.
• A rewriter that strips identifiers or changes intent, with no fallback to the original query.
• HyDE on a domain the model knows nothing about.
• Different filters applied to the two retrievers, which makes fused results inconsistent with permissions.

Lab

Add rrf and hybrid_retrieve to ragkit. On your eval questions, report recall@10 for dense only, sparse only and fused, broken down by question type (paraphrase, identifier, other). Then sweep k ∈ {1, 10, 60, 200} and see how little it matters at 60 and above. Add three follow-up questions that only make sense with history and confirm rewriting rescues them.

Interview drill

Teach it

Two friends recommend restaurants. One knows the city's food by feel ("you want somewhere cosy and Lebanese"); the other knows every restaurant's exact name and address. Ask both for a ranked list. You do not compare their enthusiasm, since one says "9/10" about everything; you only look at order. A place both put near the top is your best bet. That is RRF.

Check yourself

What you will be able to do

• Explain bi-encoder versus cross-encoder at the level of attention, and why that dictates a two-stage design.
• Add a reranker and prove with recall numbers whether it will help before you deploy it.
• Implement MMR diversification and deliberate context ordering.
• State the full retrieval stack in order, with the measurable output of each stage.

Deep dive

Bi-encoder. Query and document pass through the encoder separately. Each is compressed to one vector; all interaction between them is a single dot product. Document vectors are computed once at index time, which is what makes search over millions of items possible. The price is that the model never sees the query while reading the document. It cannot notice that the query says "enterprise" and the passage says "standard", or that the query contains "not".

Cross-encoder. Query and document are concatenated into one input: [CLS] query [SEP] document. Every query token attends to every document token in every layer, and a classification head outputs a relevance score. It can model negation, qualifiers, numbers and entity match. Cost: one forward pass per (query, document) pair, nothing precomputable. A million documents would take minutes per query.

So the architecture is forced:

Late interaction (ColBERT-style) sits between them: store one vector per token, score with MaxSim (for each query token, its best-matching document token, summed). More accurate than single-vector, cheaper than a cross-encoder, much larger index.

What reranking buys. Typical gains are 10 to 30% in NDCG@10 over the first stage. The less obvious benefit: you can over-retrieve safely. Without a reranker, k=50 floods the prompt. With one, stage 1 casts a wide net for recall and the model still receives 8 clean chunks. Fewer chunks means lower cost, lower TTFT and less distraction, so the reranker often pays for itself.

When it will not help. A reranker only reorders what it is given.

Options. Hosted APIs (Cohere Rerank, Voyage, Jina), open models (BGE-reranker, mxbai-rerank) on a small GPU, or an LLM prompted to score or order passages (most capable, slowest, priciest; use listwise prompts and watch position bias). Practicalities: rerankers have input limits, often 512 tokens, so long chunks are truncated, which interacts with small-to-big (rerank children or a trimmed parent); batch the pairs into one call; set a timeout and fall back to fused order if the reranker is slow or down; scores are model-specific, so any relevance floor must be tuned on your data.

Diversity. After reranking, the top five may be near-duplicates: overlapping chunks, the same paragraph in three policy versions. Maximal Marginal Relevance selects iteratively:

next = argmax_d [ λ·rel(d) − (1−λ)·max_{s ∈ selected} sim(d, s) ]

λ near 0.7 keeps relevance dominant while penalising redundancy. Also deduplicate exactly (same parent, same text hash) and prefer the newest version when metadata says two chunks are revisions of one document.

Context assembly. Order and format are part of retrieval quality.

• Models use the beginning and end of the context better than the middle. Place the strongest chunks at the edges.
• Give each chunk a header with its citation ID, title, section and date, so the model can cite and can reason about recency.
• Enforce a token budget by dropping the weakest chunks, never by truncating mid-chunk.
• If the best reranker score is under your floor, send no context and take the abstention path rather than feeding weak evidence.

The stack, in order: rewrite, dense ∥ sparse (k=50 each), RRF, metadata and permission filter, cross-encoder rerank (50 → 8), dedupe and MMR, assembly.

Say it: Each stage is separately measurable, which is what makes the pipeline debuggable. When quality drops I can tell you which stage regressed.

Build it

import numpy as np

def rerank(query, cands, cross_encoder, top_n=8, timeout_s=0.4, floor=None):
    """cands: list of objects with .text, .vector, .fused_rank. Falls back to fused order."""
    try:
        scores = cross_encoder.score([(query, c.text[:2000]) for c in cands], timeout=timeout_s)
    except Exception:
        return cands[:top_n], {"reranker": "fallback"}
    order = np.argsort(-np.asarray(scores))
    ranked = [cands[i] for i in order]
    for c, i in zip(ranked, order): c.rel = float(scores[i])
    if floor is not None and ranked and ranked[0].rel < floor:
        return [], {"reranker": "below_floor", "top": ranked[0].rel}
    return ranked[: top_n * 2], {"reranker": "ok"}           # leave headroom for MMR

def mmr(cands, top_n=8, lam=0.7):
    if not cands: return []
    rel = np.array([c.rel for c in cands]); rel = (rel - rel.min()) / (np.ptp(rel) + 1e-9)
    V = np.stack([c.vector for c in cands]); sim = V @ V.T
    chosen, rest = [int(np.argmax(rel))], set(range(len(cands))) - {int(np.argmax(rel))}
    while rest and len(chosen) < top_n:
        best = max(rest, key=lambda i: lam * rel[i] - (1 - lam) * sim[i, chosen].max())
        chosen.append(best); rest.remove(best)
    return [cands[i] for i in chosen]

def edge_order(chunks):
    """Best at the start, second best at the end, weakest in the middle."""
    front, back = [], []
    for i, c in enumerate(chunks):
        (front if i % 2 == 0 else back).append(c)
    return front + back[::-1]

def assemble(chunks, budget_tokens, count):
    kept, used = [], 0
    for c in chunks:                                          # already in relevance order
        t = count(c.text) + 30
        if used + t > budget_tokens: break
        kept.append(c); used += t
    return "\n\n".join(
        f'<source id="{c.cite_id}" title="{c.title}" section="{c.section}" date="{c.date}">\n{c.text}\n</source>'
        for c in edge_order(kept))

What breaks

• Adding a reranker to fix a recall@50 problem.
• No timeout or fallback, so a reranker outage takes the whole query path down.
• Silent truncation of long chunks at the reranker's input limit, hiding the relevant sentence.
• Top-k filled with duplicates from overlap or versioned documents.
• A relevance floor borrowed from another model.
• Budget enforcement that cuts a chunk in half.

Lab

Add a cross-encoder to ragkit (an open model on CPU is fine for 50 pairs). Before enabling it, compute recall@5 and recall@50 and write down your prediction. Then measure recall@5 and MRR with the reranker. Add two negation or qualifier queries ("not enterprise", "after 2024") and compare. Finally, place the known-correct chunk at position 1, in the middle, and last among 20 chunks and record answer correctness for each placement.

Interview drill

Teach it

Hiring. Stage one is screening CVs by keyword and gist: fast, thousands per hour, easy to fool. Stage two is an interview: slow, one candidate at a time, and you can ask exactly the question that matters for this role. Nobody interviews every applicant and nobody hires from the CV pile alone. MMR is making sure the shortlist is not five people with identical CVs.

Check yourself

What you will be able to do

• Run a fixed diagnostic procedure that localises any wrong answer to a stage in minutes.
• Distinguish retrieval failure from generation failure, and ranking failure from retrieval failure, with tests.
• Name the four generation-side causes and the experiment that confirms each.
• Turn every diagnosed failure into a regression test and a category count.

Deep dive

This is the question most likely to be asked. The answer is a decision tree. Practise saying it without notes.

Step 0: get the trace. You need, for the failing request, the rewritten query, per-retriever results, fused and reranked lists with scores, the exact rendered prompt, the completion, and prompt, model and index versions. If you do not have them, the first fix is observability (module 35).

Step 1: the split. Was the correct information in the context sent to the model?

• No: retrieval failure. No prompt change can help. The model never had a chance.
• Yes: generation failure. The model had it and still got it wrong.

Teams skip this and tune prompts for weeks against a retrieval problem.

Retrieval branch, walking up the stack.

1. Does the content exist in the corpus? Search the sources directly. Often it does not: the feature was never documented. The fix is writing the document. The list of low-scoring queries, the content gap report, is frequently the most valuable early artefact of a RAG project.
2. Did parsing destroy it? Read the parsed text, not the PDF (module 7).
3. Did chunking split or bury it? Find the chunk(s) holding the answer. Is the key sentence cut at a boundary, or sitting inside a huge multi-topic chunk whose embedding is mush?
4. Is the query lexical? Identifiers, names, version strings. Check whether BM25 alone finds it. If so, the fix is hybrid search or analyser changes.
5. Ranking or retrieval? Compute where the correct chunk ranks. Found at 12 of 50: a ranking problem, so rerank. Not in 50 at all: a retrieval problem, so fix the earlier items.
6. Was it found and then removed? Filters (permissions, dates, tenant), MMR, the token budget and the relevance floor all drop chunks. Check the list before and after each.
7. Did rewriting change the question? Compare the raw and rewritten queries.

Generation branch: four causes.

1. Lost in the middle. Twenty chunks, the right one at position 11. Test: rerun with only the correct chunk. If the answer becomes right, it is a position or volume problem. Fix: fewer chunks, edge ordering.
2. Distraction. Similar-but-wrong chunks (the standard policy beside the enterprise one) get blended. Test: remove the distractors and rerun. Fix: tighter reranking, metadata in chunk headers so the model can tell them apart.
3. Conflicting sources. The 2023 and 2025 policies are both retrieved. Test: check document versions among the retrieved set. Fix: filter to current versions at retrieval, carry dates in headers, instruct the model to prefer the latest and state the conflict.
4. Context ignored. Your policy says 90 days; the model's prior says 30; it answers 30. Test: ask it to quote the supporting sentence. Fix: firmer grounding instruction, required citations verified in code (module 16), and a stronger model if the behaviour persists.

Two more to keep in your pocket: the answer was truncated (finish_reason = length), or the question was ambiguous and the system should have asked a clarifying question.

Close the loop. Add the query to the eval set with its correct chunk IDs and a failure-category label. Count failures by category each week. The counts tell you what to build next, which is more persuasive to a team than any single fix.

Say it: Whatever the cause, that query becomes a permanent regression test. That is how the eval set compounds; it ends up encoding everything we have ever got wrong.

Build it

A triage function that runs the tree automatically over a trace, given the known-correct chunk IDs.

def triage(trace, gold_ids: set[str], stores, llm_answer, is_correct) -> str:
    sent = {c.id for c in trace.context_chunks}
    if gold_ids & sent:
        # --- generation branch ---
        only_gold = [c for c in trace.context_chunks if c.id in gold_ids]
        if is_correct(llm_answer(trace.question, only_gold)):
            pos = [i for i, c in enumerate(trace.context_chunks) if c.id in gold_ids][0]
            n = len(trace.context_chunks)
            return "gen:lost_in_middle" if 0.25 * n < pos < 0.75 * n else "gen:distraction"
        if len({c.doc_version for c in trace.context_chunks if c.doc_family == only_gold[0].doc_family}) > 1:
            return "gen:conflicting_sources"
        if trace.finish_reason == "length":
            return "gen:truncated"
        return "gen:context_ignored"

    # --- retrieval branch ---
    if not gold_ids:
        return "ret:content_gap"
    for stage in ("reranked", "filtered", "fused"):          # dropped after being found?
        if gold_ids & set(trace.stage_ids[stage + "_in"]) and not gold_ids & set(trace.stage_ids[stage + "_out"]):
            return f"ret:dropped_at_{stage}"
    in_dense  = gold_ids & set(trace.stage_ids["dense"])
    in_sparse = gold_ids & set(trace.stage_ids["sparse"])
    if in_dense or in_sparse:
        return "ret:ranking"                                  # found in top-50, ranked too low
    if gold_ids & set(ids_of(stores.keyword.search(trace.raw_question, k=200))):
        return "ret:lexical_or_rewrite"                       # raw keywords find it; pipeline did not
    gold_text = stores.docs.chunk_text(next(iter(gold_ids)))
    return "ret:parse_or_chunk" if looks_damaged(gold_text) else "ret:embedding_miss"

def ids_of(hits): return [h.chunk_id for h in hits]
def looks_damaged(t): return len(t) < 80 or sum(ch.isalnum() or ch.isspace() for ch in t) / len(t) < 0.8

Run it nightly over all failing eval cases and chart the categories. The chart is your roadmap.

What breaks

• Skipping the split and tuning prompts.
• No logged retrieval results, so the split cannot be made.
• Fixing the single query instead of the category it represents.
• Fixing it and not adding the regression test, so it returns after the next reindex.
• Trusting a user's "wrong answer" report without checking the corpus: sometimes the document is what is wrong, and the right action is to notify the content owner.

Lab

Break ragkit on purpose, five ways: remove a document the eval set depends on; replace a document's parsed text with scrambled text; set chunk size to 3,000 tokens; disable BM25; stuff 25 chunks into the context with the right one in the middle. For each, run triage and confirm the category is right. Then practise the spoken answer below against a two-minute timer.

Interview drill

Teach it

Frame it as a doctor's differential diagnosis. Symptom: wrong answer. First question: did the patient receive the medicine (was the chunk in context)? If no, trace the supply chain backwards: was it ever manufactured (exists), was it damaged in transit (parsing), was it split across two boxes (chunking), was it filed under a name nobody searched (lexical), was it in the warehouse on a back shelf (ranking). If yes, ask why it did not work: buried among twenty other pills, mixed with a similar-looking one, two contradictory prescriptions, or the patient ignored it.

Check yourself

What you will be able to do

• Compute recall@k, precision@k, MRR and NDCG by hand and in code.
• Implement faithfulness and answer-relevance scoring with an LLM judge, plus programmatic checks.
• Build an eval set from real traffic, failures and synthetic data, and explain the bias of synthetic questions.
• Report stratified results and define production monitoring signals.

Deep dive

Two layers. One end-to-end score tells you something is wrong and never what. Measure retrieval and generation separately.

Layer 1: retrieval. Labelled data is (query, relevant_chunk_ids).

• Recall@k: fraction of relevant chunks that appear in the top k, averaged over queries. With one relevant chunk per query this is the hit rate. It is the primary metric because retrieval is a hard gate: everything downstream is bounded by it.
• Precision@k: fraction of the top k that are relevant. Matters for cost and distraction.
• MRR: mean of 1/rank of the first relevant result. 1st → 1.0, 2nd → 0.5, 4th → 0.25, missing → 0.
• NDCG@k: for graded relevance. DCG = Σ_i (2^rel_i − 1) / log2(i + 1), divided by the DCG of the ideal ordering. Rewards placing the most relevant items highest.

Track recall at k = 5, 10, 20, 50 so the ranking-versus-retrieval diagnostic is always available. A practical detail: chunk IDs change when you re-chunk. Label at a level that survives, such as (doc_id, character span) or the answer text, and map to chunk IDs at eval time by overlap.

Layer 2: generation.

• Faithfulness: split the answer into atomic claims; for each, ask a judge whether the context supports it. Score = supported / total. Context "90 days"; answer "90 days, processed within 5 business days": claim 2 is unsupported, faithfulness 0.5. The answer looks fine, which is why you need the metric.
• Answer relevance: does the response address the question asked? Catches faithful but beside-the-point answers.
• Correctness against a reference answer, where you have one.
• Abstention accuracy: on unanswerable questions, did it decline? On answerable ones, did it answer? Report both, since each can be gamed by always doing the other.
• Programmatic checks, free and deterministic: cited IDs exist in the retrieved set; quoted text appears in the cited chunk; output parses against the schema.

Faithfulness and correctness are different. An answer can be faithful to an outdated chunk and wrong in the world. That is a retrieval or content problem, and the two-layer split shows it.

Where the eval set comes from.

1. Real queries with human annotation. Best source. 100 queries, someone finds the correct chunks. A day or two of work.
2. Failures from bug reports. Each becomes a case. This is how the set compounds.
3. Synthetic. Give an LLM a chunk, ask for a question it answers, record the chunk as gold. Fast, free labels.

Say it: Synthetic questions are cleaner and more literal than real ones. They reuse the document's own vocabulary, which flatters dense retrieval and makes numbers look better than production. I use them to supplement, not as the core.

A synthetic question reads "What is the refund window for enterprise customers?" A real one reads "hey can a big client still get money back, it's been like 2 months?" To narrow the gap, prompt for paraphrase without the chunk's key terms, generate multi-chunk questions, filter questions answerable without the chunk, and always report synthetic and real subsets separately.

Stratify. An aggregate is uninformative; a breakdown is a work plan.

Statistical honesty. With 50 cases, one case is 2 points and the 95% interval on 80% is roughly ±11 points. Do not celebrate +2. Use paired comparison on the same cases (which cases flipped, in which direction) and grow the set toward a few hundred for decisions that matter.

Production monitoring. No labels, so use proxies: zero-result rate (spikes mean ingestion or index trouble); mean top-1 score over time (downward drift means user questions are moving away from the corpus, or something broke); the low-confidence query log, which is the content gap report; the retrieved-to-cited ratio, which shows how much noise you send; thumbs, regenerations and escalations linked to trace IDs.

Build it

import math, json

def recall_at_k(ranked, gold, k):   return len(set(ranked[:k]) & gold) / len(gold)
def precision_at_k(ranked, gold, k): return len(set(ranked[:k]) & gold) / k
def rr(ranked, gold):
    return next((1 / (i + 1) for i, d in enumerate(ranked) if d in gold), 0.0)
def ndcg_at_k(ranked, grades: dict, k):
    dcg  = sum((2 ** grades.get(d, 0) - 1) / math.log2(i + 2) for i, d in enumerate(ranked[:k]))
    best = sorted(grades.values(), reverse=True)[:k]
    idcg = sum((2 ** g - 1) / math.log2(i + 2) for i, g in enumerate(best))
    return dcg / idcg if idcg else 0.0

def eval_retrieval(cases, retrieve, ks=(5, 10, 20, 50)):
    rows = []
    for c in cases:
        ranked = retrieve(c["query"], k=max(ks)); gold = set(c["gold_ids"])
        rows.append({"id": c["id"], "cat": c["category"], "source": c["source"], "rr": rr(ranked, gold),
                     **{f"r@{k}": recall_at_k(ranked, gold, k) for k in ks}})
    return rows

def by_category(rows, key="r@10"):
    cats = {}
    for r in rows: cats.setdefault(r["cat"], []).append(r[key])
    return {c: (round(sum(v) / len(v), 3), len(v)) for c, v in sorted(cats.items())}

CLAIMS = "List the atomic factual claims in this answer as a JSON array of strings.\n\nAnswer: {a}"
VERIFY = ("Context:\n{ctx}\n\nClaim: {claim}\n\nIs the claim fully supported by the context? "
          'Think briefly, then answer with JSON: {{"reason": "...", "supported": true|false}}')

def faithfulness(answer, context, judge):
    claims = json.loads(judge(CLAIMS.format(a=answer)))
    if not claims: return 1.0, []
    verdicts = [json.loads(judge(VERIFY.format(ctx=context, claim=c))) for c in claims]
    return sum(v["supported"] for v in verdicts) / len(claims), list(zip(claims, verdicts))

def citations_valid(answer_citations, retrieved_ids): return set(answer_citations) <= set(retrieved_ids)

Validate the judge before trusting it: hand-label 50 claim-context pairs and report agreement (module 36).

What breaks

• One end-to-end number, so regressions cannot be localised.
• An all-synthetic eval set reporting 95% while users struggle.
• Gold labels as chunk IDs that die on the next re-chunk.
• Reading a 2-point change on 50 cases as signal.
• An unvalidated judge.
• No should-abstain cases, so a system that always answers looks perfect.
• Overfitting to the working set. Keep a held-out set for release decisions.

Lab

Turn your question file into evals/retrieval.jsonl with fields id, query, gold (doc_id + span), category, source. Reach 50 cases: at least 25 written as a real user would, 10 synthetic, 5 identifier queries, 5 multi-chunk, 5 unanswerable. Build make eval to print the stratified table for recall@5/10/50 and MRR, and faithfulness on a 20-case subset. Save each run's results with the git commit and index version. From here on, no change to ragkit lands without this table.

Interview drill

Teach it

Test a relay team by timing each runner, not only the team. If you only time the team you know you lost and not whose leg to train. Retrieval is the first runner; if they drop the baton, the anchor's speed is irrelevant. Then show the stratified table and ask "where would you spend next week?" Everyone points at the 61% row. That is the entire argument for stratification.

Check yourself

What you will be able to do

• Explain why authorisation must happen before retrieval returns, never in the prompt.
• Implement metadata filtering, per-tenant indexes and late-binding checks, and choose among them.
• Explain how filters interact with ANN indexes and what good engines do about it.
• List the five places tenant data leaks that are not the retriever.

Deep dive

Say it: Filter at retrieval, never in the prompt.

Why the prompt approach is not access control. "Only use documents the user is authorised to see", followed by chunks from two customers' contracts, fails because the data is already in the context. The model may quote it, paraphrase it, blend the two contracts, or be talked into revealing it. The data left your trust boundary the moment it entered the prompt. An instruction is a request; a filter is a guarantee.

Pattern 1: metadata filtering. Each chunk carries tenant_id, visibility, allowed_groups. At query time, resolve the caller's groups from the identity provider and pass a filter to both retrievers. Two hard parts:

ANN interaction. Pre-filtering (restrict, then search) is exact, but a filter that leaves 0.1% of nodes destroys the graph's connectivity and recall collapses. Post-filtering (search, then filter) keeps the graph and may leave one survivor out of fifty, or none. Mature engines filter during traversal, visiting non-matching nodes as stepping stones and only collecting matching ones, and switch to brute force over the matching subset when it is small. Know which strategy your engine uses, and test recall under selective filters specifically.

ACL modelling. Real permissions are inherited (folder → file), group-based with nesting, and include denies. Flatten to a list of principal IDs at ingestion, cap the list length, and for very large group sets consider indexing a permission "bucket" ID per distinct ACL so that chunks store one small value and the query passes the buckets the user can see.

Pattern 2: index per tenant. index_acme, index_globex. A filter bug cannot leak across tenants because the data is not in the index being searched. It also gives per-tenant deletion, per-tenant encryption keys and brute-force-sized partitions. Costs: fixed overhead per index, operational sprawl, poor fit for hundreds of thousands of tiny tenants. Good for B2B with tens to thousands of tenants.

Pattern 3: hybrid. Large tenants get dedicated indexes; the long tail shares a filtered one. Common at scale. Route by a tenant → index map.

Freshness of permissions. The index holds permissions as of ingestion. Someone leaves the legal group at 10:00 and the index does not know.

• Scheduled ACL sync: simple, with a staleness window. Acceptable for low-sensitivity content.
• Late binding: retrieve with coarse filtering, then check each candidate's source document against the live permission service before use. Always fresh. Costs latency and load; batch the checks and cache decisions for 30 to 60 seconds.
• Both: coarse filter at retrieval (tenant, visibility class) to shrink candidates and provide defence in depth, then a live fine-grained check on the survivors. This is what mature systems do. Over-retrieve to allow for candidates the live check removes.

Leaks that are not the retriever.

1. Caches. key = hash(query) serves Acme's answer to Globex. Key on (tenant, permission_hash, query). This bug ships often.
2. Traces hold retrieved content from every tenant. They need the same access control, retention and deletion handling as the primary store.
3. Eval sets built from production traffic contain customer data.
4. Right to deletion has many destinations: source, document store, vector index, keyword index, caches, traces, eval sets, fine-tuning data. Map it before someone asks.
5. Conversation memory and summaries carried across sessions or users.

Also: citations and "related documents" UI must respect the same filter, since a title alone can be sensitive.

Test it. Access-control failures are silent: nothing errors, the answer simply contains what it should not. Put cross-tenant assertions in CI.

Build it

import hashlib

def principal_filter(user):
    return {"must": [{"key": "tenant_id", "match": user.tenant_id}],
            "should_any": [{"key": "allowed_principals", "any": user.principal_ids()},
                           {"key": "visibility", "match": "tenant_public"}]}

async def authorised_retrieve(query, user, stores, authz, k_final=8, overfetch=6):
    flt = principal_filter(user)                                   # coarse, in-engine
    cands = await hybrid_search(query, stores, flt, k=k_final * overfetch)
    doc_ids = list({c.doc_id for c in cands})
    allowed = await authz.batch_can_read(user.id, doc_ids)         # live, fine-grained, short-TTL cache
    out = [c for c in cands if allowed.get(c.doc_id)]
    audit.log(user=user.id, query_hash=h(query), returned=[c.id for c in out],
              denied=len(cands) - len(out))
    return out[:k_final]

def cache_key(user, query):
    perm = hashlib.sha256(",".join(sorted(user.principal_ids())).encode()).hexdigest()[:16]
    return f"{user.tenant_id}:{perm}:{h(query)}"

# tests/test_isolation.py, runs in CI against a seeded index
def test_cross_tenant_isolation(seeded):
    for query in seeded.queries_with_known_acme_answers:
        hits = run(authorised_retrieve(query, seeded.globex_user, seeded.stores, seeded.authz))
        assert all(c.meta["tenant_id"] == "globex" for c in hits)

def test_revocation_takes_effect(seeded):
    seeded.authz.revoke(seeded.alice, "legal-team")
    hits = run(authorised_retrieve("payment terms", seeded.alice, seeded.stores, seeded.authz))
    assert not any("legal-team" in c.meta["allowed_principals"] for c in hits)

def h(s): return hashlib.sha256(s.encode()).hexdigest()[:16]

A rising denied count in the audit log is a useful signal: it means the index's ACL snapshot is drifting from the live system.

What breaks

• Authorisation by prompt instruction.
• Filter applied to the vector search and forgotten on BM25, or the reverse.
• Highly selective pre-filters wrecking ANN recall, noticed only as "search is bad for some users".
• Caches keyed without tenant and permissions.
• A service account for retrieval that can read everything, combined with a bug in filter construction. Fail closed: no user identity, no results.
• Deletion requests that miss traces and eval sets.

Lab

Add tenant_id and allowed_principals to ragkit chunks; seed two tenants with near-identical documents. Implement the coarse filter plus a fake live authz service. Write both tests above. Then introduce the cache-key bug deliberately, write a test that catches it, and fix it. Finally measure recall for a user who can see 1% of the corpus versus 100%.

Interview drill

Teach it

A bank vault versus a sign. The prompt instruction is a sign on an open vault reading "please take only your own money". Retrieval filtering is the teller fetching only your deposit box. Late binding is the teller checking your ID against today's revocation list before handing it over, because the list taped to the vault door was printed last night.

Check yourself

What you will be able to do

• Implement change detection that handles new, updated, deleted and unchanged documents correctly.
• Prefer change feeds to polling and know the mechanism for each common source.
• Monitor index health and know when to rebuild.
• Run a zero-downtime reindex behind an alias, gated on an eval set.

Deep dive

State per document: doc_id, content_hash, source_modified_at, indexed_at, pipeline_version, chunk_ids[].

Delete then insert. A new version may produce 9 chunks where the old made 12. Inserting first and then removing "the old ones" requires knowing exactly which were old. Deleting everything for the doc_id and inserting is unambiguous. With the doc_id → chunk_ids map the delete is one filtered operation. If a brief gap matters, write new chunks under a new doc_version, switch a per-document pointer, then delete the old version; queries filter on the current version.

Deletes are the forgotten case and the one with consequences. A document removed from SharePoint for legal reasons that your assistant still quotes is a compliance problem. Also handle moves and renames (key on the source's stable ID, not the path) and permission changes (an ACL edit often does not change the content hash; track an ACL hash too and update metadata without re-embedding).

Change feeds beat polling. SharePoint and OneDrive delta queries; Google Drive Changes API; databases through CDC (Debezium reading the WAL or binlog); S3 event notifications; Confluence and Notion webhooks. Feeds lose events, so keep a periodic reconciliation pass that compares full listings. When polling large corpora, poll in partitions and use modified-time to avoid hashing everything.

The metric: ingestion lag. Time from a source change to searchability. Measure it per document (indexed_at − source_modified_at), publish p50 and p95 per source, and give it an SLO. "The assistant quoted the old policy for six hours" then has a measurable cause.

Index health. HNSW tombstones consume memory, are still traversed and weaken connectivity. Monitor tombstone ratio, latency trend and recall on a fixed probe set of queries with known neighbours. Compact or rebuild at 20 to 30% tombstones or on schedule. IVF centroids go stale as the corpus shifts; retrain them periodically.

Reindexing is routine. New embedding model, new chunker, added enrichment, parsing fix, engine upgrade. The pattern:

app → alias "docs-current" → physical index "docs-v7"

1. Build docs-v8 beside v7. Budget double storage.
2. Dual-write live changes to both while v8 builds, or replay the change log from the build's start time, so v8 is not stale at cut-over.
3. Run the retrieval eval set on v8 and compare with v7. This is the gate. A reindex that loses 8% recall is otherwise invisible: nothing errors and answers get slightly worse.
4. Optionally shadow live queries against v8 and compare result overlap and score distributions.
5. Flip the alias atomically. The query-side embedder must flip at the same moment if the embedding model changed; bind index name and embedder ID in one config object.
6. Keep v7 for a rollback window, then delete.

Re-embedding a large corpus costs real money and time. That is a hidden tax on changing embedding models and a reason to choose carefully. Metadata backfills should not require re-embedding: keep filter metadata in fields, out of the embedded text.

Build it

from dataclasses import dataclass

@dataclass(frozen=True)
class IndexBinding:            # one atomic unit of config; never flip these separately
    physical_index: str
    embedder_id: str
    chunker_version: str

def plan_sync(source_listing: dict, state: dict, pipeline_version: str, min_expected: int):
    if len(source_listing) < min_expected:
        raise RuntimeError("listing suspiciously small; refusing to plan deletes")
    new     = [d for d in source_listing if d not in state]
    deleted = [d for d in state if d not in source_listing]
    updated = [d for d, s in source_listing.items() if d in state and
               (s.content_hash != state[d].content_hash or state[d].pipeline_version != pipeline_version)]
    acl_only = [d for d, s in source_listing.items() if d in state and d not in updated
                and s.acl_hash != state[d].acl_hash]
    return new, updated, deleted, acl_only          # acl_only → metadata update, no re-embed

def promote(candidate: IndexBinding, current: IndexBinding, evalset, cfg_store, tolerance=0.01):
    base, cand = run_eval(evalset, current), run_eval(evalset, candidate)
    regressions = [c for c in evalset.ids if base.hit[c] and not cand.hit[c]]
    if cand.recall10 + tolerance < base.recall10 or len(regressions) > cfg_store.max_regressions:
        raise RuntimeError(f"gate failed: {base.recall10:.3f} → {cand.recall10:.3f}, "
                           f"{len(regressions)} regressions")
    cfg_store.set_atomic("docs-current", candidate, previous=current)   # rollback = set it back

def lag_seconds(rec): return (rec.indexed_at - rec.source_modified_at).total_seconds()

What breaks

• Deletes never propagating. Test by deleting a source file and querying for it.
• A flaky listing read as mass deletion.
• Flipping the index without flipping the query embedder, or the reverse.
• A new index missing the writes that happened during its multi-hour build.
• No eval gate, so recall regressions ship silently.
• Filter metadata baked into the embedded text, making every metadata change a re-embed.
• Rollback window too short to notice a slow-burn problem.

Lab

Implement plan_sync with all five outcomes and unit-test each. Add an indexed_at − modified_at histogram. Then perform a real reindex in ragkit: change the chunk size, build v2 beside v1, run the eval on both, flip through IndexBinding, and roll back. Deliberately build a worse v3 (huge chunks) and confirm the gate refuses it.

Interview drill

Teach it

A shop's website listing versus its stockroom. New stock must appear, changed prices must update, and above all sold-out items must disappear, or you sell something you do not have. Reindexing is refitting the shop: you build the new shopfront next door, check that customers can find things in it, swing the sign across in one motion, and keep the old shop unlocked for a week in case you need to go back.

Check yourself

What you will be able to do

• Give the balanced answer with numbers: where long context wins, where retrieval wins, and how they combine.
• Do the cost arithmetic live, including the effect of prompt caching.
• Explain why "supports 1M tokens" differs from "uses 1M tokens well".
• Describe cache-augmented generation, hierarchical summarisation and agentic retrieval.

Deep dive

Say it: No. It changed the operating point rather than replacing it. Long context made RAG more forgiving, not unnecessary.

The case for long context. No chunking decisions, no embedding model to version, no vector store to run, no retrieval step to get wrong. For one 200-page contract or a mid-sized codebase, pasting the whole thing is simpler and often better, because the model can reason across the whole document instead of eight fragments.

Four arguments for retrieval.

1. Cost. You pay for every input token on every request. At an illustrative $3 per million: 200,000 tokens is $0.60 a query; 2,000 tokens is $0.006; a 100× gap. At 10,000 queries a day that is $6,000 against $60. Prompt caching narrows this for a stable corpus queried repeatedly: cached input is heavily discounted and prefill is much faster. It does not help when the context differs per user or changes often, and cache entries expire.
2. Capacity. A company wiki is gigabytes. Ten years of tickets is larger. No window holds "all our documents".
3. Quality. Accuracy is strongest at the start and end of a long context and dips in the middle. Needle-in-a-haystack tests measure locating one planted sentence; they say little about multi-hop reasoning, which degrades much faster with length ("compare the refund clauses across these 40 contracts"). Irrelevant text measurably distracts. Latency matters too: prefill over hundreds of thousands of tokens takes many seconds.
4. Attribution and access control. Retrieval naturally yields "document X, page 4" and filters per user. With long context you choose what to include up front, which means over-including (leak risk) or assembling context per user, which is retrieval under another name.

The synthesis. Bigger windows change how you tune retrieval. You can pass 20 to 50 chunks, or whole parent sections, instead of agonising over top-3 precision. Retrieval errors cost less. Retrieval still does the work of choosing.

Combined patterns.

• Retrieve documents, read them whole. Use retrieval to pick the three relevant documents, then give the model their full text. Often the best of both for document-centric corpora.
• Cache-augmented generation. Prefill a fixed corpus once and reuse the cached state across queries. Effectively cheap long context for a stable document set that fits.
• Hierarchical summarisation. Summarise sections, retrieve over summaries, pull full text for matches. Also the answer to "summarise this 2,000-page corpus" questions, which top-k retrieval handles badly (map-reduce over sections).
• Agentic retrieval. The model issues targeted searches over several turns rather than receiving one fixed context (module 23).

Decide by measurement. Run the same eval set through both designs and compare quality, cost per successful answer and p95 latency.

Build it

A router that chooses the strategy per request, with the arithmetic explicit.

def choose_strategy(corpus_tokens, qpd, needs_citations, per_user_acl, corpus_changes_daily,
                    price_in=3.0, price_cached=0.30, window=200_000):
    if corpus_tokens > 0.5 * window or per_user_acl:
        return "rag"                                   # capacity / authorisation force the choice
    rag_cost  = qpd * 3_000 * price_in / 1e6
    lc_cost   = qpd * corpus_tokens * (price_in if corpus_changes_daily else price_cached) / 1e6
    if needs_citations or lc_cost > 5 * rag_cost:
        return "rag_with_whole_document_reads"
    return "long_context_cached"

def whole_document_context(hits, docs_store, budget_tokens, count):
    """Retrieval picks documents; the model reads them in full."""
    out, used, seen = [], 0, set()
    for h in hits:                                     # hits are reranked chunks
        if h.doc_id in seen: continue
        text = docs_store.full_text(h.doc_id); t = count(text)
        if used + t > budget_tokens: continue
        out.append((h.doc_id, text)); used += t; seen.add(h.doc_id)
    return out

What breaks

• Assuming a vendor's context length implies reasoning quality at that length. Test with your own multi-fact questions at your own lengths.
• Forgetting that cached long context still pays full decode cost and that cache entries expire between bursts of traffic.
• Per-user data pasted into a shared cached prefix.
• Using top-k RAG for "summarise everything" tasks, which need map-reduce or hierarchical summaries.
• Declaring a winner without running both on the same eval set.

Lab

Pick a document set under 100K tokens. Answer your eval questions three ways: standard RAG, full corpus in context, and retrieve-then-read-whole-documents. Record correctness, input tokens, cost and latency per question. Add three questions that require combining facts far apart in the corpus and note which approach handles them.

Interview drill

Teach it

An open-book exam where you may bring the entire library into the hall. You are allowed to. You will also spend the whole exam walking the aisles, you pay by the kilogram to bring the books in, and you still cannot point to the page when asked where an answer came from. A good index lets you bring the right three books. Bigger desks mean you can bring ten books instead of three and worry less about picking perfectly.

Check yourself

What you will be able to do

• Explain hallucination as a structural consequence of training and decoding, not a glitch.
• Apply the mitigation stack in order of effectiveness and say what each costs.
• Build a grounded-answer prompt with abstention and citations, and verify citations in code.
• State an honest goal: detectable and bounded, not eliminated.

Deep dive

Why it happens. The training objective is next-token prediction, which optimises for plausibility. A fabricated citation has the right shape: a plausible author, journal and year. Nothing in the objective separates a true statement from a well-formed false one. Three compounding factors:

1. No "I don't know" token. Every forward pass yields a distribution and something must be sampled. Sampling from a flat, uncertain distribution still yields confident prose; uncertainty does not show in the words.
2. Incentives reward guessing. When preference data and benchmarks score confident answers above hedged ones, the model learns to guess, like a student on a test with no penalty for wrong answers.
3. Autoregressive commitment. After one wrong token the model conditions on it and builds a coherent justification, because coherent text is more probable than self-contradiction.

Why RAG helps and does not solve it. Facts are in the context, so recall is not required. The model can still misread, over-generalise ("within 10 business days" → "usually about a week"), blend context with its priors, or answer confidently when the context lacks the answer. Faithfulness must be measured.

Mitigation stack, most effective first.

1. Retrieve the right content. Most RAG "hallucinations" are retrieval failures in disguise: the model was not given the answer and made one up.
2. Permit abstention explicitly, with an exact sentence to use and a few-shot example of a correct refusal. Models follow a template better than a concept. Measure both false answers and false refusals.
3. Require citations, verify in code. Ask for [source_id] after each claim and check: is every cited ID in the retrieved set; does quoted text appear in the cited chunk? This converts a soft instruction into a hard check, and verification failures are excellent automatic eval signals. A variant: ask for supporting quotes first, then an answer built from them.
4. Groundedness check. A second call judging each claim against the context. Roughly doubles cost and latency. Use on high-stakes paths, or asynchronously as a monitor.
5. Do not over-stuff the context. Irrelevant chunks increase fabrication because the model tries to use what it is given.
6. Self-consistency for high stakes: sample several times at non-zero temperature; real knowledge is stable, fabrications vary. Agreement is a confidence signal. Expensive.
7. Tools over recall. Arithmetic to a calculator, current data to an API, lookups to a query.
8. Lower temperature. Reduces variance, not bias. A confidently wrong model stays wrong at 0, consistently.

What to do on a failed check. Decide it in advance: regenerate once with the failure fed back; drop unsupported sentences; fall back to the abstention message; or route to human review. Log which path was taken.

Say it: I would frame the goal as making hallucination detectable and bounded rather than eliminated. A system that claims zero hallucination is a system that is not measuring.

Build it

import re

GROUNDED_SYSTEM = """You answer questions for Acme staff using only the sources provided.

Rules
1. Use only information inside <source> tags. Do not use outside knowledge.
2. After every sentence that states a fact, cite the source id like [S3]. Use only ids you were given.
3. If sources conflict, prefer the most recent date and say that they conflict.
4. If the sources do not contain the answer, reply with exactly:
   I don't have information about that in the available documents.
   Then add one sentence saying what is missing.

Example of a correct refusal
Question: What is the refund window for government customers?
Sources mention enterprise and standard customers only.
Answer: I don't have information about that in the available documents. The sources cover enterprise and standard tiers but not government customers."""

ABSTAIN = "I don't have information about that in the available documents."
CITE = re.compile(r"\[(S\d+)\]")
QUOTE = re.compile(r"“([^”]{12,})”|\"([^\"]{12,})\"")

def verify_answer(answer: str, sources: dict[str, str]):
    if answer.strip().startswith(ABSTAIN):
        return {"ok": True, "abstained": True}
    cited = set(CITE.findall(answer))
    unknown = cited - set(sources)
    sentences = [s for s in re.split(r"(?<=[.!?])\s+", answer) if len(s.split()) > 4]
    uncited = [s for s in sentences if not CITE.search(s)]
    norm = lambda t: re.sub(r"\s+", " ", t).lower()
    bad_quotes = [q for m in QUOTE.findall(answer) for q in m if q
                  and not any(norm(q) in norm(src) for src in sources.values())]
    return {"ok": not unknown and not bad_quotes and len(uncited) <= 1,
            "abstained": False, "unknown_citations": sorted(unknown),
            "uncited_sentences": uncited, "fabricated_quotes": bad_quotes}

def answer_with_guard(question, sources, llm):
    ans = llm(GROUNDED_SYSTEM, render(sources, question))
    v = verify_answer(ans, sources)
    if not v["ok"]:
        ans = llm(GROUNDED_SYSTEM, render(sources, question) +
                  f"\n\nYour previous draft failed verification: {v}. Rewrite it so every fact is cited to a given source.")
        v = verify_answer(ans, sources)
    return (ans if v["ok"] else ABSTAIN + " I could not produce a fully sourced answer."), v

A valid citation proves the source exists, not that it supports the claim. That second property needs the claim-level judge from module 12, run inline on high-stakes paths and as a sampled monitor elsewhere.

What breaks

• Tuning the prompt when retrieval is the cause.
• An abstention rule with no example, leading to either constant refusals or none.
• Treating citation presence as proof of support.
• Abstention measured on one side only. A system that always refuses is perfectly "safe" and useless.
• Large contexts of marginal chunks.
• Temperature 0 presented as a hallucination fix.

Lab

Add the grounded prompt and verify_answer to ragkit. On your unanswerable eval questions, measure the false-answer rate with and without the abstention rule and example. On answerable questions, measure the false-refusal rate. Then pad the context with ten irrelevant chunks and measure faithfulness again.

Interview drill

Teach it

An improv actor's rule is "never say no; always continue the scene". That is a language model by default: asked anything, it continues plausibly. You cannot lecture the actor into being an encyclopaedia. You can hand them a script (context), tell them the exact line to use when the script is silent (abstention template), require them to point at the line they are reading from (citations), and have a stage manager check the pointing (verification).

Check yourself

What you will be able to do

• Produce a latency budget and a cost breakdown for a RAG request from memory.
• Apply the latency techniques in order of payoff: parallelise, stream, cache the prefix, trim context, skip retrieval.
• Implement semantic caching with the safeguards that prevent wrong answers.
• Compute cost per successful task and use it to compare models.

Deep dive

Where time goes.

Prefill dominates and scales with prompt length (module 3). Context size is therefore the main lever for both latency and cost.

Latency techniques.

1. Parallelise. Dense and sparse search are independent; so are query embedding, the permission lookup, and multiple query variants. Use asyncio.gather. Frequently missed.
2. Stream. Total time is unchanged; perceived latency is transformed. A 4-second answer that starts at 800 ms feels fast; a 3-second answer that arrives whole feels slow. After streaming, optimise TTFT. Streaming and post-hoc verification conflict: either stream and append a verification status at the end, or buffer on high-stakes paths.
3. Prompt caching. Stable instructions, format and few-shots first; chunks, history and question last. A 3,000-token prefix at 50,000 queries a day is 150M tokens a day billed at the cached rate and prefetched faster. Wrong order means zero hits.
4. Trim context. Going from 20 chunks to 5 cuts prefill time and input cost roughly fourfold and usually improves answers. This is why reranking pays for itself twice.
5. Skip retrieval when unnecessary. "Hello", "thanks", "rephrase that". A cheap classifier, or exposing retrieval as a tool the model may choose not to call, saves the whole path on a real share of traffic.

Also: keep connections warm, co-locate services in one region, give every stage a timeout with a defined fallback, and cap the rewrite step's latency since it sits on the critical path before anything else can start. A useful trick: start retrieval on the raw query immediately, run the rewrite in parallel, then retrieve on the rewrite and fuse.

Cost structure per request. Query embedding about $0.00001; vector search about $0.0001 amortised; reranking about $0.001 per 50 documents; generation $0.003 to $0.05. Within generation, input dominates output in RAG: 4,000 tokens in, 300 out.

Cost levers, by impact. (1) Less context through better reranking, which also improves quality. (2) Prompt caching. (3) Model routing: simple lookups from clean context do not need the largest model (module 32). (4) Cap max_tokens; runaway lists are a real line item. (5) Semantic caching on repetitive workloads.

Semantic caching. Key responses by query embedding and serve a stored answer when a new query is similar enough. Real savings where twenty questions make up 60% of traffic. The failure mode: "What's the refund window?" cached as "30 days", then "What's the enterprise refund window?" arrives at similarity 0.94. Serve from cache and you are wrong. Safeguards: a high threshold (0.95+) tuned on labelled near-miss pairs; tenant and permission hash in the key; short TTL; invalidate on index version change; exclude time-sensitive and personalised questions; cache only answers that passed verification; optionally confirm equivalence with a cheap model before serving. An exact-match cache on the normalised query is a safer first step and catches a surprising share.

The tying metric.

Say it: I would track cost per successful task, not cost per request or per token.

Model A: $0.02 per request, 95% success → $0.021 per success. Model B: $0.008 per request, 70% success; failures retry on A, and 10% escalate to a human at $4: 0.008 + 0.30 × 0.02 + 0.10 × 4 = $0.414. B is 2.5× cheaper per request and about 20× more expensive per outcome.

Build it

import asyncio, time

async def answer(req, deps):
    t0 = time.perf_counter(); spans = {}
    if await deps.needs_retrieval(req.message) is False:            # cheap classifier, <30 ms
        async for token in deps.chitchat(req): yield token
        return

    raw_task     = asyncio.create_task(deps.hybrid(req.message, req.user))     # start immediately
    rewrite_task = asyncio.create_task(asyncio.wait_for(deps.rewrite(req), timeout=0.6))
    lists = [await raw_task]
    try:
        rewritten = await rewrite_task
        if rewritten != req.message:
            lists.append(await deps.hybrid(rewritten, req.user))
    except asyncio.TimeoutError:
        spans["rewrite"] = "timeout"                                # degrade, do not fail
    cands  = deps.fuse(lists)
    chunks = await deps.rerank_or_fallback(req.message, cands, top_n=6, timeout=0.4)
    spans["retrieval_ms"] = (time.perf_counter() - t0) * 1000

    async for token in deps.llm_stream(stable_prefix=deps.prompt.stable,       # cacheable
                                       dynamic=deps.prompt.render(chunks, req), max_tokens=500):
        yield token

class SemanticCache:
    def __init__(self, embedder, store, threshold=0.96, ttl_s=3600):
        self.e, self.s, self.th, self.ttl = embedder, store, threshold, ttl_s
    def _scope(self, user, index_version): return f"{user.tenant_id}:{user.permission_hash}:{index_version}"
    def get(self, query, user, index_version):
        hit = self.s.nearest(self._scope(user, index_version), self.e.query(query))
        if hit and hit.score >= self.th and hit.age_s < self.ttl:
            return hit.answer
    def put(self, query, user, index_version, answer, verified: bool, time_sensitive: bool):
        if verified and not time_sensitive:
            self.s.add(self._scope(user, index_version), self.e.query(query), answer)

def cost_per_success(cost_req, p_success, retry_cost=0.0, p_escalate=0.0, human_cost=0.0):
    return cost_req + (1 - p_success) * retry_cost + p_escalate * human_cost

What breaks

• Sequential awaits where gather would do.
• Optimising total latency while users are waiting on TTFT.
• A semantic cache threshold set by feel, serving near-miss answers.
• Cache not scoped by tenant, permissions and index version.
• A cascade that improves average cost while p95 latency worsens, unnoticed.
• Reporting cost per token to finance when the business question is cost per resolved task.

Lab

Instrument every stage of ragkit's query path. Produce a p50/p95 table. Apply in order: parallel retrieval, streaming, stable-prefix caching, context reduced from 20 chunks to 6. Record TTFT, total time, cost and eval score after each step. Build a labelled set of 30 near-miss query pairs (same answer or different answer) and choose the semantic cache threshold from it.