Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation (RAG) is an architecture pattern that enhances a large language model’s responses by first retrieving relevant information from external knowledge sources—such as documents, databases, or APIs—and then feeding that retrieved context into the model alongside the user’s query. Rather than relying solely on what the model memorized during pre-training, RAG lets it look things up before answering.

Why RAG exists

Large language models have a fundamental limitation: their knowledge is frozen at the time of training. They cannot access information published after their training cutoff, they do not have access to your company’s internal documents, and they sometimes generate plausible-sounding but incorrect information (hallucinations) when their training data is insufficient. RAG addresses all three problems by giving the model access to external, up-to-date, and proprietary information at query time—without requiring the model to be retrained.

How RAG works: the three-stage pipeline

A standard RAG system has three components:

1. Knowledge base (indexing): Documents are split into chunks (typically a few hundred words each), converted into numerical representations called embeddings using an embedding model, and stored in a vector database. This indexing step happens once (and is updated as new documents arrive). Common vector databases include Pinecone, Weaviate, Qdrant, and Chroma.

2. Retriever (query time): When a user asks a question, the system converts the query into an embedding and searches the vector database for the chunks most similar to the query. The retriever returns the top-k most relevant chunks (typically 3-10).

3. Generator (response): The retrieved chunks are injected into the LLM’s context window alongside the user’s original question. The model uses this context to compose its answer—ideally grounded in the retrieved information rather than hallucinated from its training data.

RAG vs fine-tuning

RAG and fine-tuning solve overlapping but distinct problems:

Use RAG when:

• Your data changes frequently and real-time relevance matters
• You need transparent sourcing and citations for answers
• You need access to private or proprietary data not in the model’s training set
• You want a fast, cost-effective solution without retraining
• You have limited ML resources or budget for model training

Use fine-tuning when:

• You need to change the model’s behavior, tone, or output style (not just its knowledge)
• Your task involves consistent, repetitive patterns (classification, extraction)
• You need domain-specific terminology deeply encoded in the model
• You have a static dataset that rarely changes

Many production systems use both: fine-tune the model for domain style and structured output, then layer RAG on top for real-time knowledge retrieval and citation-backed answers. The two approaches are complementary, not competing.

RAG limitations

RAG is powerful but not a silver bullet. Key limitations include:

• Garbage in, garbage out: Retrieval quality directly determines answer quality. If the knowledge base contains outdated, incorrect, or poorly structured documents, the model will retrieve and use bad information.
• Context window limits: Even models with large context windows (100K+ tokens) cannot fit an entire knowledge base into a single query. The retriever must select the most relevant chunks, and if it misses the right one, the answer suffers.
• Infrastructure complexity: A production RAG system requires pipelines for document ingestion, chunking, embedding, vector storage, retrieval, and monitoring. This is non-trivial engineering.
• Conflicting sources: If the knowledge base contains documents with contradictory information, the model may surface both without resolving the conflict.
• Still can hallucinate: When retrieved context is insufficient or ambiguous, the model may fill in gaps from its training data—introducing hallucinations despite having access to external information.
• Security risks: Corpus poisoning attacks—where an adversary inserts malicious content into the knowledge base—can manipulate RAG outputs. Defenses like RAGPart and RAGMask address this but add complexity.

Advances in 2025-2026

RAG has evolved significantly beyond simple vector search:

• Agentic RAG: Instead of a single retrieve-then-generate step, the model plans multi-step retrieval strategies—deciding what to search for, evaluating whether the retrieved information is sufficient, and requesting additional retrieval if needed. This turns RAG from a pipeline into a reasoning process.
• GraphRAG: Combines vector search with knowledge graphs, allowing the model to follow structured relationships between entities rather than relying only on text similarity.
• Evidence verification: Systems like SURE-RAG evaluate whether retrieved evidence actually supports, refutes, or is insufficient for a candidate answer—adding a fact-checking layer before the model responds.
• Multimodal RAG: Extends retrieval beyond text to tables, images, video, and even robotics applications.
• Streaming re-indexing: Rather than batch-reprocessing the entire knowledge base when documents change, modern systems update only the changed portions—keeping the index fresh with lower compute cost.

Worked example: RAG-powered customer support

A company deploys a RAG-based AI assistant for its support team. Here is how it works in practice:

1. Knowledge base setup: The company ingests 5,000 product documentation pages, 10,000 past support tickets, and 200 policy documents into a vector database.
2. Customer question arrives: “Can I get a refund on my annual subscription if I cancel after 6 months?”
3. Retrieval: The system converts the question to an embedding, searches the vector database, and retrieves three relevant chunks: the refund policy document, a past ticket about a similar refund request, and the subscription terms page.
4. Generation: The model receives the question plus the three retrieved chunks. It composes an answer grounded in the refund policy.
5. Citation: The answer includes a link to the refund policy document and notes which section it references.

Without RAG, the model might have guessed at the refund policy, hallucinated specific terms, or given a generic answer. With RAG, the answer is grounded in the company’s actual documents.

Reader Questions

Does RAG eliminate hallucinations?

No. RAG reduces hallucinations by grounding the model’s response in retrieved documents, but it does not eliminate them. If the retrieved context is insufficient, ambiguous, or contradictory, the model may still fill in gaps from its training data.

What is the difference between RAG and a search engine?

A search engine returns a list of documents that match a query. RAG goes further: it retrieves relevant document passages, reads them, and synthesizes a direct answer to the user’s question.

How much does it cost to set up RAG?

The cost varies widely. A basic prototype using open-source tools can be set up in a day for minimal cost. A production-grade system can require significant engineering investment.

Can I use RAG with any LLM?

Yes, in principle. RAG works with any LLM that can process text input. However, models with larger context windows can accept more retrieved context.

Sources

1. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (2020-05-22)
2. Gao et al., “Retrieval-Augmented Generation for Large Language Models: A Survey” (2023-12-18)
3. Microsoft, “GraphRAG: Unlocking LLM discovery on narrative private datasets” (2024-07-02)
4. Edge et al., “From Local to Global: A Graph RAG Approach to Query-Focused Summarization” (2024-04-24)
5. Amazon Web Services, “What is RAG?” (2024)