Retrieval-Augmented Generation (RAG) is a technique that gives an LLM access to specific knowledge at query time — without retraining the model or stuffing everything into the context window. Instead of asking a model to answer from its training data alone, RAG first retrieves the most relevant documents from a knowledge base, then passes those documents to the model alongside the question. The core idea: find first, then answer. If Cartara flagged this in your diff, you're likely setting up a vector database, writing an embedding pipeline, or building a retrieval step into an LLM call.

Why RAG Exists

LLMs have two fundamental limitations:

1. Training cutoff — the model doesn't know about events or documents created after it was trained
2. Finite context window — you can't fit a 10,000-page knowledge base into every prompt

RAG solves both. You store your knowledge externally and retrieve only the relevant pieces on demand.

Fine-tuning teaches the model how to behave. RAG gives the model what to know. They're complementary, not competing — and RAG is almost always the right first tool for knowledge problems, since it's cheaper, faster to iterate, and keeps your knowledge base current without retraining.

How RAG Works

Phase 1: Indexing (done once, or when data changes)

1. Collect your documents (PDFs, web pages, databases, transcripts)
2. Split them into chunks (paragraphs, sections, fixed token windows)
3. Convert each chunk into an embedding — a numerical vector representing its meaning
4. Store the embeddings in a vector database

Phase 2: Retrieval + Generation (done at query time)

1. User asks a question
2. Convert the question into an embedding using the same model
3. Search the vector database for chunks most semantically similar to the question
4. Insert the top-k retrieved chunks into the prompt as context
5. LLM answers using those chunks as grounding

```
User Question
↓
[Embed question]
↓
[Search vector DB] → Top 3-5 relevant chunks
↓
[Prompt = question + chunks]
↓
LLM generates answer grounded in retrieved context
```

Key Concepts

Embeddings — a list of numbers (a vector) that represents the meaning of text. Texts with similar meanings have vectors close together in space, which enables semantic search. You need the same embedding model for both indexing and querying.

Chunking — how you split documents significantly affects retrieval quality. Common starting point: 256–512 tokens per chunk with 10–20% overlap between adjacent chunks. Too small = chunks lack context; too large = chunks contain too much irrelevant content.

Vector databases — store embeddings and support fast similarity search:

For small projects, pgvector via Supabase is the path of least resistance — no new infrastructure required.

What You'll See in Your Code

A basic RAG prompt template:

```python
system_prompt = """You are a helpful assistant. Answer the user's question
using only the provided context. If the context doesn't contain the answer,
say so — do not make up information."""

context_block = "\n---\n".join(retrieved_chunks)

user_message = f"""Context:

{context_block}

Question: {user_question}"""
```

Indexing documents with OpenAI embeddings and pgvector (Supabase):

```python
from openai import OpenAI

client = OpenAI()

def embed_chunk(text: str) -> list[float]:
response = client.embeddings.create(
model="text-embedding-3-small",
input=text
)
return response.data[0].embedding

Store in Supabase with pgvector

supabase.table("documents").insert({
"content": chunk_text,
"embedding": embed_chunk(chunk_text)
}).execute()
```

Retrieval Strategies

Dense retrieval (standard) — embedding-based similarity search. Finds chunks that mean the same thing as the query even if they don't share the same words.

Sparse retrieval (keyword-based) — traditional keyword search (BM25). Misses semantic matches but excels at exact terms, names, and identifiers.

Hybrid retrieval — combines dense and sparse. Best of both worlds for most production use cases.

Re-ranking — after retrieving top-k candidates, run a cross-encoder model to reorder them by true relevance. Significantly improves precision at modest added cost.

Common Failure Modes

Retrieval misses — the right chunk exists but isn't retrieved. Usually caused by poor chunking or a mismatch between how documents are written and how users ask questions. Fix: improve chunking, add hybrid retrieval.

Context dilution — too many chunks retrieved, burying the relevant information in noise. Fix: reduce k (number of retrieved chunks), add re-ranking, use metadata filters.

Lost in the middle — LLMs attend more to the start and end of context. Relevant chunks placed in the middle get underweighted. Fix: put the most relevant chunks first.

Hallucination despite retrieval — the model ignores retrieved context and answers from training data anyway. Fix: strengthen the system prompt instruction ("answer only from provided context"); use a model with strong instruction following.

Stale index — vector database isn't updated when source documents change. Fix: implement incremental indexing — update the index when documents are added, modified, or deleted.

Getting Started

1. Start with Supabase Vector or Chroma — minimal new infrastructure
2. Use OpenAI text-embedding-3-small — cheap, fast, very good
3. Chunk at ~512 tokens with 10% overlap
4. Retrieve top 3–5 chunks — start conservative
5. Instruct the model explicitly to stay grounded in provided context
6. Measure retrieval quality on a small test set before scaling

The most common mistake is over-engineering the retrieval pipeline before validating that the basic setup works. Start simple, measure, then optimize.

Related Concepts

• Prompting Best Practices
• Structured Outputs
• The MCP Ecosystem
• LoRA (Low-Rank Adaptation)