Generative AI / explainer
RAG gives a model evidence. Not certainty.
Separate retrieval from generation to understand where grounded answers succeed—and where they still fail.
explainerGenerative AIexplainerWhat if the answer is in your documents?
A model is asked about a procedure updated yesterday. Its parameters may not contain that update. Retrieval-augmented generation introduces another source of information: material retrieved at request time.
Two pipelines, two sets of decisions
The preparation pipeline collects documents, extracts text, preserves metadata, splits material into useful units, and builds an index.
The request pipeline interprets a question, retrieves candidates, optionally reranks them, selects context, and asks a model to produce an answer. The answer can include source references for inspection.
In a document assistant, enforce the caller’s access rules before a passage enters the model context. Searching an index and supplying context do not train the model or grant permission to execute a tool. When no permitted passage supports the question, make that missing evidence visible.
Retrieval brings the evidence. Generation still has to use it well.
Walk through a failure
The generation step did its local job, yet the answer is wrong for the current policy. The missing requirement was effective-date handling in the retrieval pipeline. Rewriting the prompt alone will not repair that index.
Grounding is a measurable behavior
Keep separate questions in your evaluation:
- Did retrieval find the relevant, authorized material?
- Did the selected context include the necessary evidence?
- Does each important answer claim follow from that evidence?
- Do citations point to passages that support the claim?
- Does the system abstain when the evidence is inadequate?
A citation can be real but irrelevant. A fluent answer can combine a correct statement with an unsupported conclusion.
A practical trade-off table
| Decision | Benefit | Cost or risk |
|---|---|---|
| Smaller chunks | More precise retrieval units | Context can become fragmented |
| More retrieved context | Potentially broader coverage | Higher latency and irrelevant material |
| Reranking | A second relevance check | Another stage to operate and evaluate |
| Frequent indexing | Fresher evidence | More ingestion and versioning work |
Treat these as design hypotheses. Measure their effects on your own questions.
What comes next?
Start with a versioned document set and an explicit evaluation set. Log which document versions were selected for each answer without collecting unnecessary personal data.
Next question: how should identity, permission checks, caching, and source updates fit around this pipeline? Those are architecture decisions, and they are part of answer quality.
Primary sources
- Lewis et al. (2020): Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks ↗
The original retrieval-augmented sequence-to-sequence formulations and their reported task evaluations; not the production architecture proposed here.
Primary sources support descriptions of the original research. TheJobPoolConsultants explanations, diagrams, examples, and engineering interpretations are identified separately.
Part 1 of 1 published · Generative AI Systems
Explore this series →