RAG and vector databases

Why retrieval beats a bigger prompt

A language model answers from patterns learned during training, which ended at some cutoff and never included your private files. Ask it about your biology notes and it will produce something plausible, which is worse than producing nothing.

You could paste every note into the prompt, but that runs into three walls. Context windows have limits, every extra token costs money and time, and models can overlook a detail buried in a very long context. Retrieval-augmented generation, or RAG, solves this by searching first and generating second.

The pattern has two halves. Offline, you index your documents once. Online, for each question, you find the most relevant passages, put them into the prompt with clear rules, and let the model write an answer based on them. Updating knowledge then means updating the index, not retraining anything.

RAG is usually the first tool to reach for when the problem is missing knowledge. Fine-tuning, covered in lesson 19, changes how a model behaves, but it is a poor way to teach facts that change every week.

Indexing: chunk, embed, store

Chunking splits long documents into pieces small enough to retrieve precisely and large enough to make sense alone. Splitting by headings or paragraphs keeps ideas together better than cutting every fixed number of characters. A small overlap between neighbouring chunks stops a sentence that crosses a boundary from being lost.

Clean the text before you split it. Text pulled out of PDFs and web pages often carries page headers, footers, menus and broken line breaks, and that noise ends up inside chunks and their vectors.

Keep metadata with each chunk: the file name, the heading, a page number or a date. Metadata lets you filter results, for example to one course, and it is what you show the learner as the source.

Each chunk then goes through an embedding model, which turns text into a vector of numbers where similar meanings sit close together. Use the same embedding model for documents and questions, because vectors from different models are not comparable. If you switch models later, you must re-embed everything.

• Too small: chunks lose context, and the model gets fragments that cannot answer anything.
• Too large: one chunk mixes several topics, so its vector matches nothing strongly.
• A sensible start: a few hundred words per chunk with a short overlap, then tune on your own questions.

Retrieval: finding the right passages

At question time you embed the question and compare it with every stored vector. Cosine similarity, the angle between two vectors, is the usual score; dot product and Euclidean distance are common alternatives. You keep the top few chunks, often three to five.

Vector search understands paraphrases, so a question about how plants make food finds a note titled photosynthesis. It is weaker with exact strings such as error codes, names or formula labels, where classic keyword search shines. Hybrid search runs both and merges the results, and most serious RAG systems end up there.

Two upgrades help when results look noisy. A re-ranker takes the top twenty candidates and scores each one against the question more carefully, keeping the best few. A minimum similarity threshold stops the app from passing weak matches to the model when nothing relevant exists.

In a chat, follow-up questions are often too short to search with. A message like "and why does it need light?" means nothing on its own, so ask the model to rewrite it into a standalone question using the conversation history, then embed the rewritten version.

Grounding and citations

Retrieval only helps if the prompt tells the model how to use it. Number the passages, tell the model to answer only from them, ask it to cite the numbers it used, and give it a clear way out when the passages do not contain the answer.

Show the cited sources in the interface, with the file name and the passage text. Learners can then check an answer in seconds, and you can spot a wrong citation while testing.

Evaluate the two halves separately. Retrieval quality asks whether the right chunk appeared in the top results at all. Answer quality asks whether the reply is faithful to the passages and actually answers the question. When an answer is wrong, the log of retrieved chunks tells you which half failed.

Retrieved text is data, never instructions. A shared document can contain a line telling the model to ignore its rules, which is the indirect prompt injection from lesson 14, so keep the passages clearly separated from your system prompt and never let them trigger tools on their own.

Choosing a vector store

For a few thousand chunks, a Python list and a loop, as in the example, is fast enough and has nothing to install. Save the vectors to a file so you do not pay to embed the same notes on every start.

When the collection grows or you need filters and persistence, move to a real store. The choice depends on size, hosting, metadata filtering, hybrid search support and how backups work, not on which name is most popular.

• Libraries in your process: FAISS or plain NumPy arrays for fast similarity search without a server.
• Embedded databases: Chroma, LanceDB or SQLite with a vector extension, stored as local files.
• Your existing database: PostgreSQL with the pgvector extension keeps vectors next to normal tables.
• Search engines: Elasticsearch and OpenSearch combine keyword and vector search.
• Dedicated servers or managed services: Qdrant, Weaviate, Milvus or Pinecone for large collections.