Pre-launch · Training and the free Foundations exam are open now. Levels 1–5 open soon. Get notified on any level page.
20 min read
Retrieval-augmented generation (RAG) gives a model access to documents it wasn't trained on by finding relevant passages and placing them in context before it answers. It's the standard way to ground answers in your own policies, manuals, and records. It also introduces new failure points: the wrong passage, a passage the user shouldn't see, or an answer that sounds grounded but isn't. This module covers how to design, secure, and evaluate a retrieval pipeline end to end.
Chunk along structure. Documents are split into passages for search. Fixed character counts cut rules away from their headings and exceptions. Split on sections, keep headings with each chunk, keep tables whole, and add some overlap so a rule ending one section isn't separated from the exception that starts the next. Test chunk sizes and overlap on real questions, because overlap adds duplication and cost.
Carry provenance. Every chunk keeps its document ID, version, effective date, section, and page. Answers cite these. Without provenance, no one can check the answer or tell if it came from a superseded version.
Retrieve in two passes. Keyword search finds exact terms like part numbers and form codes. Semantic search finds passages that mean the same thing in different words. Hybrid retrieval combines both, and a reranker reorders the combined results by relevance to the specific question.
Check access before retrieval, not after. Filter the index by the requester's permissions before searching. If restricted passages reach the model, a later instruction to "not reveal them" is not a control.
Relevant isn't supporting. A passage about the right topic may not support the specific claim. "Overtime policy" is relevant to "Is overtime paid at 1.5x on holidays?" but may say nothing about holidays. Require each claim to be supported by a specific cited passage, and let the system abstain when none is.
Bound searching. Agents that search iteratively need a call budget, a history of queries already tried, and a rule to stop when a new query is unlikely to add evidence. Repeating near-identical queries that return the same page isn't corroboration.
Evaluate two things separately. Retrieval: did the right passages come back, ranked high enough? Faithfulness: is every claim in the answer supported by what was retrieved? A good retriever can feed an unfaithful answer, and a faithful answer can rest on poor retrieval.
Test dates hard. Dates are where models fabricate most convincingly. Build test cases where the effective date is missing, where two versions conflict, and where the only date is in an appendix. The right answer is often "not established" or "conflicting sources."
Nadia Haddad runs knowledge systems at a pharmaceutical manufacturer with 6,000 employees. A plant-safety assistant answers questions from procedures. Some procedures are restricted to certified technicians. An operator asked when the revised solvent-handling rule took effect and got "March 1" with a confident citation. The rule had two versions, and the cited section had no date.
Before:
Search the safety documents and answer the user's question.
Cite your sources.
After:
Pipeline (enforced in application):
- Filter index to documents the requester's role may access.
- Hybrid search (keyword + semantic), top 30, rerank to top 6.
- Chunks: by section with headings; tables whole; 15% overlap;
each carries doc_id, version, effective_date, section, page.
- Max 3 searches per question; log queries; stop if a new query
returns no new doc_ids.
Model instruction:
Answer only from <retrieved_passages>. For each claim, cite
doc_id, version, and section. If no passage states the fact,
write "not established in available procedures." If versions
conflict, report both with effective dates. Never infer a date.
Nadia's eval set has 120 questions, including 15 date questions with missing or conflicting dates and 10 questions whose answers depend on appendix exceptions. She scores retrieval recall (was the correct section in the top 6?) and faithfulness (was every claim supported?) separately.
Why this is stronger. Access is enforced before search. Chunks keep their structure and provenance, so the appendix exception arrives with its rule. Search is bounded and logged. The model has a defined abstention path. And the eval targets exactly the failure that occurred, fabricated dates, with cases built to provoke it.
Setup. Take a policy document of 5 to 10 pages that has at least one exception in an appendix or footnote. Write 8 questions: 5 answerable from the main text, 2 whose answer depends on the exception, and 1 the document doesn't answer.
Task. Paste the document into a chat tool and ask each question with an instruction to cite sections and say "not established" if unsupported. Score each answer twice: was the right section cited, and was every claim supported?
What a good result looks like. You have two scores per question, not one. Any answer that cites a relevant section but makes an unsupported claim is marked as a faithfulness failure. The unanswerable question gets "not established."