Retrieval-Augmented Generation (RAG)

6.3.1Definition

RAG is a pattern that retrieves relevant documents or data at query time and inserts them into the model's context before generating a response, rather than relying solely on knowledge baked into the model's training. It combines a search step (5.12) with a generation step, grounding the model's answer in specific, current, retrievable source material.

6.3.2Why It Exists

A model's training data has a fixed cutoff and cannot include a company's private documents, a product's current data, or anything created after training ended. RAG exists to bridge that gap — letting a model answer accurately about proprietary or current information — without the cost and latency of retraining the model itself every time the underlying information changes.

6.3.3The RAG Pipeline

• Chunking — source documents split into retrievable pieces small enough to be relevant, large enough to retain meaning.
• Embedding & indexing — each chunk converted to a vector representation (6.4) and stored in a searchable index.
• Retrieval — at query time, the most relevant chunks are found via similarity search and inserted into context (6.2).
• Generation — the model produces its answer grounded in the retrieved chunks, ideally citing which chunk supported which claim.

6.3.4Common Mistakes

• Chunking documents naively by fixed character count, splitting related content mid-sentence or mid-table and losing the context needed to make the chunk useful on its own.
• Retrieving too many or too few chunks, either diluting the model's context with marginally relevant material or omitting the one chunk that actually contained the answer.
• No re-indexing pipeline when source documents change, causing RAG to confidently answer from stale, outdated chunks.
• Treating RAG as a hallucination cure-all, when a model can still misinterpret or contradict retrieved content if the prompt (6.1) doesn't explicitly instruct it to ground its answer in the provided sources.

6.3.5Best Practices

• Chunk along natural document boundaries (sections, paragraphs) rather than fixed character counts.
• Explicitly instruct the model to answer only from retrieved content and to say so when the answer isn't present.
• Build a monitored re-indexing pipeline so retrieval stays current with source data (5.12).