Why RAG

1 min read
On trails:AGRAGstep 1/8

An LLM’s knowledge is cut off at its training data. Retrieval-augmented generation (RAG) is the technique that fixes this inside context engineeringContext engineeringDeciding what the model sees on every call, because the window is finite.Opens in a new tab: fetch the relevant data when the question arrives, and put it in the prompt.

The gap

Two things are missing from a trained model. No new data: anything that happened or was written after the cutoff. No private data: the company’s documents, tickets and code were never in the training set. LangChain’s retrieval docs name the same two limits, static knowledge (training data is frozen at a point in time) and finite context (a model can’t take in a whole corpus at once).

Context injection

Asking the model about something it never saw gets a guess. So the answer has to be carried in the context. The user query still goes to the LLM, and the retrieved pieces are injected next to it.

The user query goes to the LLM, which answers. Retrieved context, searched from the data the model never saw in training, is injected into the LLM alongside the query. User query the question LLM frozen at training Answer grounded in context Retrieved context relevant pieces injected Your data new, private search
The model answers from what was injected, not from memory.

In the vocabulary of the anatomy of contextAnatomy of contextFive kinds of context, told apart by where they come from and how long they live.Opens in a new tab, this is retrieved context: knowledge fetched just in time from an external source.

The catch

RAG adds a second system whose job is to find the right pieces. If retrieval fetches the wrong ones, the answer is built on the wrong ones. The next notes cover how it works, starting with embeddingsEmbeddingsData turned into an array of numbers, placed so that similar meaning sits close together.Opens in a new tab. Whether it is needed at all is a separate question (see RAG versus long contextRAG versus long contextPassing the whole file avoids retrieval, until the cost, the attention and the window limit show up.Opens in a new tab).

Sources