Before generation, there was representation.

A search problem before a language problem

A reader asks for “ways to reduce inference expense.” Your document is titled “Lowering model serving costs.” An exact word match may miss a useful result.

Before a system can compare a query and a document, it needs a representation of each. Different representations preserve different information.

Start with counts

A bag-of-words representation records terms and their frequencies. It can be sparse and interpretable: one dimension corresponds to one vocabulary term. But word order is mostly lost.

Term frequency–inverse document frequency (TF-IDF) combines term frequency with inverse document frequency, reducing the weight of terms that occur in many documents. The Stanford information retrieval text explains term weighting. These lexical methods remain useful when exact identifiers, names, or specialist terms matter.

Dense vectors change the comparison

An embedding maps an input into a vector. The training objective influences which inputs end up close under a chosen similarity measure. Related phrases may then match even without sharing the same words.

That similarity is not human understanding. An embedding can confuse related but contradictory statements, miss a critical number, or behave poorly in a domain unlike its training data.

Context changes the task

Document retrieval adds further choices: whether to represent a sentence, paragraph, section, or entire document; how to preserve headings; and what metadata to retain. A vector detached from its source is difficult to inspect or cite.

Evaluate with questions from the actual task

Create a small set of realistic queries and label relevant documents. Include paraphrases, abbreviations, rare identifiers, and queries for which the collection has no answer.

Compare lexical retrieval, vector retrieval, and a hybrid approach. Do not infer quality from a few satisfying demos. Examine missing results as well as high-ranked irrelevant ones.

Try the contrast

Write two short documents that use different words to express the same idea. Then write two that share many words but disagree. Predict how a term-based and an embedding-based retriever might rank each.

Next question: once you retrieve the right material, how do you get a language model to use it faithfully? Retrieval-augmented generation connects those steps, but introduces its own failure modes.