Part 9 of the Agentic Engineering Series
By: David Estrada | 
Date Published: 2026/06/19
In Part 8, we went deep on Markdown as the invisible substrate holding the entire agentic stack together. Now we zoom out to tackle the question underneath the question — how do AI systems actually find and use your information? The answer is RAG, and it’s not what most people think it is.
RAG Explained: How AI Systems Search Your Documents — And Why “Smarter AI” Won’t Fix a Broken Pipeline
The retrieval layer is where most AI systems fail. Here’s how it actually works.
"You can't fix a retrieval problem with a better LLM. The model only sees what you hand it. If you hand it garbage, it produces garbage — confidently."
If you’ve used an AI chatbot that “knows about your documents,” you’ve seen RAG in action. Maybe it worked brilliantly. Maybe it gave you a confidently wrong answer about something clearly in the document you uploaded.
That second outcome isn’t a model failure. It’s a pipeline failure. And you can’t fix it by upgrading the model.
RAG — Retrieval-Augmented Generation — is the architecture that lets AI systems search a knowledge base and use what they find to answer questions. It’s behind every “chat with your docs” product, every enterprise knowledge assistant, every AI that claims to know about your internal systems.
Understanding it deeply is not optional for agentic engineers. It’s table stakes.
Part 1: The Core Problem RAG Solves 🧩
Large language models are trained on massive datasets — but that training has a cutoff. They don’t know about your internal docs. They don’t know about the meeting notes from last Tuesday. They don’t know about the spec that changed three months ago.
The naive fix is to paste everything into the prompt. But there’s a hard limit: context windows cap out. You can’t stuff a 50,000-document knowledge base into a prompt. Even if you could, the model’s attention degrades when the context is too long.
RAG solves this by flipping the problem. Instead of giving the model everything upfront, you:
- Index your documents into a searchable vector database
- Retrieve only the most relevant chunks at query time
- Inject those chunks into the prompt
The model never sees the full knowledge base. It sees a curated, relevant slice — and answers from that.
Simple in theory. Surprisingly easy to break in practice.
Part 2: The Indexing Phase — Building the Searchable Foundation 🏗️
Before you can retrieve anything, you have to build the index. This is the phase most people rush through. It’s where most RAG systems quietly break.
Step 1: Documents
Raw input. Text files, PDFs, Obsidian notes, web pages, Confluence pages — anything with content you want to make searchable. At this stage it’s unstructured text. Qdrant (or any vector database) doesn’t understand it yet.
Step 2: Chunking
You can’t embed an entire document as one unit effectively.
A 10,000-word document embedded as a single vector loses all specificity. The embedding becomes an average of everything, which means it matches nothing well. Ask about the billing section of a long spec — a whole-document embedding returns the whole spec, ranked below documents that are more narrowly about billing.
Chunking solves this by splitting documents into smaller pieces — typically 200–500 words per chunk, with overlap between adjacent chunks.
The overlap is not optional. Here’s why:
If chunk 3 ends mid-sentence and chunk 4 starts mid-sentence, retrieval might return chunk 3 without its conclusion, or chunk 4 without its setup. Either way, the context the LLM receives is broken. With overlap — say 50 words shared between adjacent chunks — each chunk carries enough surrounding context to stand alone as a coherent piece of text.
Get chunking wrong and everything downstream suffers. This is the most common root cause of RAG failures.
Document (10,000 words)
→ Chunk 1: words 1–400
→ Chunk 2: words 350–750 ← 50-word overlap with chunk 1
→ Chunk 3: words 700–1100 ← 50-word overlap with chunk 2
→ ...
Get chunking wrong and everything downstream suffers. This is the most common root cause of RAG failures.
Step 3: Embedding
An embedding model — for example, all-MiniLM-L6-v2 — converts each text chunk into a vector: a list of floating point numbers.
"The server is down" → [0.12, -0.83, 0.44, 0.07, -0.31, ...]
← 384 dimensions in MiniLM's case →
What do those numbers mean? Semantic meaning, encoded as a position in high-dimensional space.
Chunks that mean similar things end up geometrically close to each other. “The server is down” and “the machine stopped responding” land near each other. “The server is down” and “I like pasta” do not.
This is the translation step — from human language to math that Qdrant can reason about. The quality of your embedding model sets the ceiling on how well semantic search can work.
Step 4: Store in Qdrant
Each chunk gets stored as a point in Qdrant with three parts:
| Field | What it is |
|---|---|
| ID | Unique identifier for this chunk |
| Vector | The embedding — the 384 (or 768, or 1536, depending on model) floating point numbers |
| Payload | Metadata: source file, original text, section heading, date, whatever you attach |
Qdrant builds an HNSW graph (Hierarchical Navigable Small World) over all stored vectors. This index structure lets Qdrant find nearest neighbors in milliseconds — without comparing your query against every single point in the database.
At scale, that index is the difference between a 20ms query and a 20-second one.
Part 3: The Retrieval + Generation Phase — From Question to Answer 🔍
With the index built, you’re ready for the live pipeline. This is what runs every time a user asks a question.
Step 1: Query
The user’s question. Plain text. No special syntax required — this is the whole point.
Step 2: Embed the Query
The query goes through the exact same embedding model you used during indexing. This produces a vector in the same dimensional space as your stored chunks.
This constraint is non-negotiable. If you indexed with all-MiniLM-L6-v2 (384 dimensions) and query with text-embedding-3-large (1536 dimensions), the vectors live in completely different spaces. The geometry doesn’t align. You’ll get garbage results with no error message to tell you why.
"Why is the server slow?" → [0.09, -0.71, 0.51, ...] (384 dimensions)
Step 3: Search Qdrant
Qdrant computes the cosine similarity (or dot product, depending on your config) between your query vector and every stored vector, then returns the top-k closest matches.
“Closest” = most semantically similar to your question.
This is not keyword search. It doesn’t look for matching words. It finds chunks whose meaning is geometrically near your query’s meaning. That’s the power:
- “server slow” matches “high CPU utilization causing degraded response times” — even with zero word overlap
- “billing issue” matches “payment processing error in the invoicing module” — same idea
It’s also the failure mode: near in meaning isn’t always relevant to the question. Semantic similarity is a proxy for relevance, not a guarantee of it.
Step 4: Inject Top-k Chunks into the LLM Prompt
You take the retrieved chunks and inject them into your prompt as context:
Here is relevant context:
[chunk 1 text]
[chunk 2 text]
[chunk 3 text]
Answer this question using the context above: {user question}
The LLM never searched Qdrant. It never saw your documents. It only sees what you paste into the prompt.
The retrieval step is entirely your responsibility. You chose which chunks to retrieve. You chose how many (top-k). You chose how to format them. The LLM is just reading what you hand it.
This is the critical point most people miss: the LLM is not doing RAG — you are. The model is pattern-matching over whatever context it receives. If you hand it the wrong chunks, it either hallucinates or says “I don’t know.” Upgrading the model won’t fix a retrieval problem.
Step 5: Generate the Answer
The LLM reads the injected context and answers based on it. A good model grounds its response in the chunks. If the chunks are relevant but the model is weak, it may still drift. If the chunks are irrelevant, even the best model can’t save you.
Part 4: The Core Insight — Every Layer Caps the Next ⚠️
The entire RAG system is a pipeline where each step’s output quality sets a ceiling on the next step:
Bad chunks → bad embeddings → bad retrieval → bad context → bad answer
You can’t fix a retrieval problem by swapping in a better LLM. You can’t fix a chunking problem by upgrading your embedding model. Each layer has to be right independently.
Most RAG failures are chunking or retrieval failures misdiagnosed as LLM failures.
The debugging order matters:
- Are the chunks coherent and appropriately sized?
- Is the embedding model a good fit for your content domain?
- Is the top-k actually returning relevant chunks for your test queries?
- Is the retrieved context being injected cleanly into the prompt?
- Only then: Is the LLM response quality the problem?
Teams that skip straight to step 5 spend weeks swapping models and see no improvement. Because the model was never the problem.
Part 5: Common Failure Modes 🚨
| Failure | Symptom | Root Cause |
|---|---|---|
| Chunks too large | Retrieval returns vaguely relevant but unfocused context | Embedding averages too much meaning |
| Chunks too small | Retrieved chunks lack enough context to answer the question | Sentences without surrounding setup/conclusion |
| No chunk overlap | Mid-sentence boundaries break context | Adjacent chunks don’t share enough overlap |
| Model mismatch | Garbage retrieval results, no error | Indexing and querying used different embedding models |
| Top-k too low | Correct answer exists but wasn’t retrieved | Not enough candidates surfaced |
| Top-k too high | LLM context overloaded with irrelevant chunks | Too many candidates dilute signal |
| Weak embedding model | Domain-specific terminology not captured | General model on specialized content |
Part 6: RAG in the Agentic Engineering Stack 🤖
RAG doesn’t exist in isolation in an agentic system. It’s a retrieval tool that agents invoke — and the quality of what the agent can reason about is bounded by how well that tool is built.
In the Agent, for example, the FAQ knowledge base is indexed nightly. When a visitor asks a question, the agent doesn’t search the raw Confluence pages — it queries the Qdrant index, retrieves the top-k FAQ chunks, and generates a response grounded in that content.
The agent is only as good as the index. Stale chunks = stale answers. Bad chunking = broken context. No overlap = incomplete answers. The model — even a strong one — can’t compensate.
This is why RAG setup is an engineering task, not a configuration task. Chunk size, overlap, embedding model selection, index refresh cadence — these are decisions with real quality consequences. They deserve the same rigor as any other infrastructure choice.
RAG Quick Reference
| Decision | Recommendation |
|---|---|
| Chunk size | 200–500 words to start; tune based on your content |
| Chunk overlap | 10–15% of chunk size (e.g., 50 words on a 400-word chunk) |
| Embedding model | Match domain — general content: MiniLM; code/technical: look at domain-specific models |
| Top-k | Start at 3–5; increase if answers are incomplete, decrease if too noisy |
| Index refresh | Match your content update cadence — nightly for most KB use cases |
| Debugging order | Chunks → embeddings → retrieval → injection → generation |
| Model mismatch check | Always verify indexing and query embedding models are identical |
| Metadata in payload | Always store source + original text; add section/date if available |
"The teams that get RAG right aren't using better models than everyone else. They're building better pipelines."