The problem: the assistant answers from memory

A company builds an assistant for its internal policies. Asked about the leave policy, it answers fluently and incorrectly, because the model never saw the current document and fills the gap with something plausible. Fine-tuning does not fit: the documents change weekly, access rules differ per employee, and you need to show where an answer came from.

Retrieval-augmented generation (RAG) solves this by splitting the job. Your system finds relevant text from your own data, puts it into the prompt, and asks the model to answer using only that text. The model supplies language skills; your data supplies the facts.

The pipeline

 ingest (offline)                     query (online)
 ----------------                     --------------
 documents                            user question
    |                                      |
 parse + clean                       authorise + rewrite
    |                                      |
 chunk  -> metadata              retrieve top candidates
    |                              (keyword + vector)
 embed  -> index  <------------------------+
                                           |
                                   rerank / filter
                                           |
                                  build prompt + sources
                                           |
                                  LLM -> answer + citations
                                           |
                                    validate + log

Treat the ingest side as a data pipeline and the query side as a service with latency and failure modes. Calling the model safely is covered in calling an LLM from a backend service.

Chunking

Models and indexes work on passages, not whole files. Chunk boundaries decide what can be found.

  • Split by structure first (headings, sections, list items), then by size. A chunk should make sense alone.
  • Keep some overlap between neighbouring chunks if sentences depend on context.
  • Store metadata with each chunk: document id, title, section path, version, language, access group, last updated.
  • Too small and the meaning is lost; too large and the relevant sentence is diluted and costs more input. There is no universal size; tune it against your evaluation set.

Embeddings and retrieval

An embedding model maps text to a vector so that similar meaning gives nearby vectors. At query time you embed the question and search for the nearest chunk vectors. Several storage options exist: a dedicated vector database, or a vector extension in a database you already run, such as PostgreSQL. A familiar database often wins for moderate scale because it combines filters, transactions and operations you already know.

SELECT id, doc_id, section, contentFROM chunkWHERE access_group = ANY (:groups)   -- authorisation first  AND language = :langORDER BY embedding <=> :query_vector  -- distance operatorLIMIT 8;

Operator names depend on the extension; the point is that the filter is part of the query. Never retrieve first and filter later, and never put restricted text in a prompt for a user who may not read it.

Pure vector search misses exact identifiers (error codes, product names, article numbers). Hybrid retrieval, which combines keyword (full-text) and vector results and then merges or reranks them, is often more robust.

Grounding and citations

Give the model numbered sources and ask it to answer only from them, to cite source numbers, and to say when the sources do not contain the answer. Then verify in code: citations must refer to sources you actually supplied, and an answer without citations may be shown as "not found" instead. A response format with fields such as answer, citations[] and confidence can be validated like any other structured output.

Citations let users check claims and give you a debugging trail. They do not prove correctness, since a model can cite a passage that does not support its sentence, so sample and review them.

Freshness and lifecycle

Documents change, and stale answers are worse than none.

  • Store a content hash per document; re-ingest only changed ones.
  • Handle deletions: remove chunks and their vectors, and invalidate caches (see caching strategies).
  • Version the embedding model. Vectors from different models are not comparable, so a model change means re-embedding everything, ideally into a new index and then switching.
  • Show the document date in answers when it matters.

Evaluation: measure retrieval and answers separately

Many "bad answers" are retrieval failures. Build a small, versioned set of real questions with the documents that should answer them.

  1. Retrieval: for each question, is a correct chunk in the top results? Track simple hit rates over time.
  2. Answer: is the answer supported by the supplied sources, does it cite correctly, and does it refuse when nothing relevant exists?
  3. Regression: rerun on every change to chunking, embedding model, prompt or ranking.

Add questions that have no answer in the data; a system that always answers is dangerous.

When keyword search is enough

If users search for known names, codes and phrases, and documents are well titled, a full-text index with good ranking may satisfy the need at lower cost and complexity. Indexing fundamentals are in SQL indexes explained. Move to embeddings when people phrase questions differently from the documents, across synonyms or languages. A reasonable path is to start with keyword retrieval plus an evaluation set, then add vectors and compare results rather than assuming improvement.

Security

Retrieved documents are untrusted input: a stored page can contain instructions aimed at the model. Apply the same containment as in the LLM integration article, enforce document-level permissions at retrieval time (see API security and the OWASP list), and do not give the assistant tools beyond what the feature needs.

Processing the question itself can also help. A short question that depends on the conversation history ("and the second one?") must become a standalone query, otherwise retrieval does not know what to look for. Keep this as a separate step, log its output and include it in the evaluation set. Then, when an answer is bad, it is easier to tell whether the cause is the query rewrite, the search or the generation.

Common mistakes

  • Chunking by fixed characters and cutting tables and sentences in half.
  • Skipping metadata, then being unable to filter, cite or delete.
  • Ignoring access control until after retrieval.
  • Never re-ingesting, so answers drift from reality.
  • Judging quality by a few impressive demo questions.
  • Stuffing too many chunks into the prompt; irrelevant text hurts answers and cost.

Trade-offs

More context improves recall but increases cost, latency and distraction. Reranking improves precision but adds a step. A separate vector database scales independently but adds operations and synchronisation work. Aggressive caching speeds responses but can serve stale or unauthorised content if keys ignore the user's permissions.

Checklist

  • Chunks follow structure and carry metadata.
  • Authorisation filter is part of the retrieval query.
  • Hybrid or keyword baseline measured first.
  • Answers cite supplied sources and can say "not found".
  • Re-ingestion, deletion and embedding versioning are handled.
  • An evaluation set covers retrieval and answers, including unanswerable questions.

Where to go next