Inside a production RAG pipeline.

Problem and requirements

Design a document assistant for an internal knowledge collection. Different readers can access different documents. Policies change, and some questions have no supported answer.

Define success before selecting infrastructure: authorized evidence, traceable document versions, understandable failure states, and a latency budget appropriate to the workflow.

High-level architecture

Keep ingestion separate from serving. An ingestion job should extract text, validate metadata, record the source version, and write searchable chunks. Promote an index version only after quality checks pass.

Request flow

Resolve identity in the application, then apply access controls before material can enter the model context. Retrieve candidates, rerank if justified by evaluation, and select evidence within a context budget.

Generate an answer with source identifiers. Resolve those identifiers to actual documents when rendering citations. Return an explicit insufficient-evidence response when the system cannot support an answer.

The original RAG research motivates combining retrieval with generation. The permissions, deployment boundaries, and operational checks here are engineering recommendations for this example.

Data flow and caching

Store source identifier, version, chunk location, and access metadata with every chunk. Avoid logging raw sensitive document text merely to make debugging convenient.

A cache key for an answer may need to include the query, permitted scope, index version, prompt version, and model version. Sharing a cached answer across permission boundaries can expose material even when the retriever itself is correctly scoped.

Security and failure modes

Retrieved text is untrusted input. A document may contain language that tries to redirect the assistant. It should remain evidence, not acquire authority over the application.

Observability and evaluation

Measure each stage independently: retrieval relevance, answer support, citation correctness, refusal quality, latency, and cost. Record version identifiers so a regression can be reproduced.

Keep a fixed evaluation set for comparisons, but add examples from newly discovered failures. Track both, because improving on an old test does not prove coverage of new situations.

Scaling and alternatives

A single lexical index may be enough for a small collection with precise terminology. Hybrid search or a reranker adds operational cost and should earn its place.

Scale ingestion, retrieval, and generation independently when measurements show a bottleneck. Queue background indexing work, but avoid turning every online step into another network hop without a reason.

Production checklist

• Demonstrate that forbidden documents never enter unauthorized context.
• Verify how deletions and permission changes propagate.
• Test stale, contradictory, missing, and malicious documents.
• Exercise timeout and partial-failure responses.
• Establish a rollback path for index, prompt, and model changes.

Next question: which of these guarantees can you test automatically before every release? Start with the ones whose failure would be hardest to notice from the final answer alone.