Inside a Complete RAG Pipeline
See a complete RAG system end to end, and learn to trace a bad answer back to the stage that caused it.
Time to step back and look at the whole machine. A real RAG system has two halves that run at completely different times.
Indexing happens ahead of time, and again whenever documents change. Collect the documents, parse them into clean text, split them into chunks, embed each chunk, and store vectors, text and metadata in the vector database. It can take hours for a big collection, but no user is waiting.
Answering happens for every question, in a second or two. Embed the question, search the database (with filters), rerank the shortlist, build the prompt with the best chunks and instructions, generate the answer, and show it with citations.
Every stage can fail, and each failure looks different from the outside. That makes debugging a detective game. When an answer is wrong, the first question is always: did the right chunk reach the model?
- If not, it’s a retrieval problem. Was the document never collected, or out of date? Garbled by parsing? Cut in half by chunking? Missed by search because it hinges on an exact code?
- If it did, it’s a generation problem. Unclear instructions, too many distracting chunks, or a model misreading the text.
Because of that split, teams usually measure the two halves separately. For a set of test questions with known answers, they check whether retrieval found the right chunk at all, and then whether the final answer was correct and faithful to the chunks it was given. Fix the stage the numbers point to, rerun the tests, repeat.
A RAG system is never really finished. Documents change, so the index must keep up; new kinds of questions appear, so the tests must grow.
Key takeaways
- Indexing (collect, parse, chunk, embed, store) runs ahead of time and whenever documents change.
- Answering (embed, search, rerank, prompt, generate, cite) runs for every question.
- Debug by asking first: did the right chunk reach the model? That separates retrieval from generation problems.
- Measure retrieval and answer quality separately on test questions, fix the weakest stage, repeat.
Check your understanding
Go deeper (more technical)
Common retrieval metrics are recall@k (was a correct chunk among the top k?) and MRR (how high was the first correct one?). Answer quality is often judged for faithfulness (supported by the retrieved chunks) and relevance, frequently by another model acting as a grader, a technique Track 9 looks at more critically.
Production pipelines add caching of frequent questions, query rewriting (turning “and the price?” into a full question using the chat history), monitoring of “I don’t know” rates, and incremental re-indexing triggered by document changes.