RAG at query time

1 min read
On trails:AGRAGstep 5/8

When the user asks something, RAG runs a one-way chain: Query, Embed, Search, Stuff, Generate. The index it searches was built earlier by ingestionRAG ingestionThe offline half of RAG: chunk the data, embed the chunks, save them in a vector database.Opens in a new tab.

At query time: the query is embedded, the vector database is searched for the closest chunks, the chunks are stuffed into the prompt with the query, and the LLM generates the answer. The vector database was built offline. AT QUERY TIME Query user asks Embed query vector Search top chunks Stuff into prompt Generate LLM answers Vector DB built offline
Five steps, once, in a straight line.

The chain

The query is embedded with the same model used at ingestion. A semantic search returns the chunks closest to it. Then comes the stuffing: the user query and the retrieved chunks go into the prompt together, which must fit the limited context window. The LLM generates from that.

Python
from langchain.chat_models import init_chat_modelfrom langchain_core.embeddings import DeterministicFakeEmbeddingfrom langchain_core.prompts import ChatPromptTemplatefrom langchain_core.runnables import RunnableLambdafrom langchain_core.vectorstores import InMemoryVectorStore # Built earlier by the ingestion step (placeholder embeddings).store = InMemoryVectorStore.from_texts(    ["Refunds are issued within 14 days.", "Support is open on weekdays."],    DeterministicFakeEmbedding(size=256),)model = init_chat_model("openai:gpt-5.4-mini")  # any chat model prompt = ChatPromptTemplate.from_template(    "Answer from the context only.\n<context>\n{context}\n</context>\nQuestion: {question}") chain = (    {        "context": RunnableLambda(lambda q: store.similarity_search(q, k=2))        | RunnableLambda(lambda docs: "\n\n".join(d.page_content for d in docs)),        "question": RunnableLambda(lambda q: q),    }    | prompt    | model) chain.invoke("How long do refunds take?")

The retrieval step is search, then join into one string, and the prompt template stuffs it beside the question. The store here uses placeholder embeddings, so what it returns is not meaningful. LangChain calls this shape 2-step RAG: retrieval always comes before generation, so the latency is predictable.

What the chain cannot do

It runs once. If the first search misses, nothing notices, and the model answers from whatever came back. Reordering the results helps (see rerankingContext rerankingModels read the middle of a long context worst, so put what matters at the edges.Opens in a new tab), and so does letting the model retry (see agentic RAGAgentic RAGRetrieval as a loop: the agent decides whether to search, checks what came back, and tries again.Opens in a new tab).

Retrieved text is also untrusted. The LangGraph tutorial’s prompts tell the model to treat it as data only, because a chunk can carry instructions (see prompt injectionProbabilistic risksThe model is probabilistic, so its failures are too: three threats, four things at stake.Opens in a new tab).

Sources