How to build a RAG application with citations (step by step)

What RAG is, and why citations matter

Retrieval-augmented generation (RAG) means: before the model answers, you retrieve the most relevant passages from your own documents and give them to the model as context. The model then answers from those passages instead of from memory.

Citations turn a RAG answer from "trust me" into "here is where it says so". For internal policy assistants, support bots and anything regulated, an answer without a source is not usable. Citations also make your system debuggable: when an answer is wrong, you can see whether retrieval found the wrong passage or the model misread the right one.

The pipeline at a glance

1. Load documents (PDF, HTML, Markdown).
2. Split them into chunks of a few hundred words, keeping the source and page/section with each chunk.
3. Embed each chunk into a vector and store it in a vector database.
4. Retrieve the top few chunks for a question.
5. Generate an answer that cites the chunks it used.
6. Evaluate with a set of known questions (covered in a separate guide).

Step 1–2: chunk with metadata

Chunking is where most RAG quality is won or lost. Keep chunks focused (roughly 300–800 words), split on headings or paragraphs rather than fixed character counts where you can, and always store where the chunk came from.

def chunk_document(text: str, source: str, max_words: int = 400):
    chunks, current = [], []
    for para in text.split("\n\n"):
        if sum(len(p.split()) for p in current) + len(para.split()) > max_words and current:
            chunks.append({"text": "\n\n".join(current), "source": source})
            current = []
        current.append(para)
    if current:
        chunks.append({"text": "\n\n".join(current), "source": source})
    return chunks

Each chunk keeps its source (file name, URL or page number). That is what you will cite later.

Step 3–4: embed, store, retrieve

Any embedding model and vector store will do for a first version: a local option such as FAISS or Chroma, or Postgres with the pgvector extension if you already run Postgres. The pattern is the same:

# index once
for c in chunks:
    c["vector"] = embed(c["text"])
store.add(chunks)

# at question time
hits = store.search(embed(question), top_k=5)

Two practical tips:

• Retrieve a few more chunks than you show. Retrieve 8–10, then keep the best 4–5 (optionally with a re-ranking step).
• Set a minimum similarity. If nothing relevant is found, the right answer is "I don't know", not a guess.

Step 5: ask for an answer with citations

Number the retrieved chunks and ask the model to cite the numbers it used. Ask for structured output so your code can check the citations.

context = "\n\n".join(f"[{i+1}] ({h['source']})\n{h['text']}" for i, h in enumerate(hits))

system = (
    "Answer only from the numbered context. After each claim, cite the source number like [2]. "
    "If the context does not contain the answer, say you don't know."
)
answer = llm(system=system, user=f"Context:\n{context}\n\nQuestion: {question}",
             response_format={"answer": "string", "citations": "list[int]"})

Then validate in code: every cited number must exist in the retrieved set, and an answer with no citations should be treated as "not found". Show the user the answer plus the source titles and links for the cited numbers.

Common failure modes (and fixes)

Symptom	Likely cause	Fix
Right document, wrong answer	Chunk too large; key sentence buried	Smaller, heading-aware chunks
Answer cites a source that doesn't say it	Model over-generalising	Stricter instruction, lower temperature, validate citations
"I don't know" for questions that are covered	Query and document use different words	Hybrid search (keyword + vector) or query rewriting
Confident answer to an off-topic question	No similarity threshold	Refuse below a minimum score
Old policy quoted	Stale index	Re-index on change; store and show the document date

From RAG to an agentic workflow

Once the assistant answers reliably with citations, the natural next step is to let it act: search more than one source, call an API (for example, check a leave balance), and ask a human to approve before submitting a request. That is an agentic workflow built on top of your RAG core, and it is only as good as the retrieval underneath it.

Checklist before you ship

• Every chunk stores its source
• Similarity threshold → "I don't know"
• Answers cite numbered sources, validated in code
• Source titles/links shown to the user
• A test set of 30+ real questions with expected answers
• Re-indexing when documents change