Giving AI your own knowledge (RAG)

The single biggest complaint about AI is "it makes things up." The solution is grounding: give the model the exact material to answer from, and force it to cite. That technique is called RAG — retrieval-augmented generation.

Why RAG exists

LLMs have two knowledge gaps:

• Training cutoff. They don't know what happened after their training date.
• Your data. They don't know your past proposals, brand guidelines, client history, internal wiki.

You could try to solve both by cramming everything into the prompt every time. That works for small collections (a single PDF), but breaks for 500 past RFPs — you'd blow the context window and pay a fortune per query.

RAG is the compromise: store your material as searchable chunks, retrieve the 5–10 most relevant chunks for each question, put only those in the prompt.

The RAG loop

Embeddings, briefly

An embedding is a list of ~1500 numbers that represents the meaning of a text chunk. Two chunks about the same topic have similar numbers, even if they use different words. That's how "How did we price the last kitchen renovation?" finds a chunk titled "Estimate #2041 — kitchen remodel, 240 sqft."

You get embeddings from an API — OpenAI's text-embedding-3-small is cheap and excellent. Cloudflare Workers AI has @cf/baai/bge-large-en-v1.5 as a free-tier equivalent. Store the numbers in a vector database.

Where to store vectors

A minimal RAG pipeline (Cloudflare Vectorize)

TypeScript · Cloudflare Worker// wrangler.toml binds VECTOR_INDEX (Vectorize) and AI (Workers AI)

export default {
  async fetch(req, env) {
    const { question } = await req.json();

    // 1. Embed the question
    const emb = await env.AI.run("@cf/baai/bge-large-en-v1.5", {
      text: [question]
    });

    // 2. Search the top 5 matching chunks
    const matches = await env.VECTOR_INDEX.query(emb.data[0], {
      topK: 5,
      returnMetadata: true
    });

    // 3. Build a context block with citations
    const context = matches.matches.map((m, i) =>
      `[${i+1}] (${m.metadata.source})\n${m.metadata.text}`
    ).join("\n\n");

    // 4. Ask the LLM using ONLY that context
    const answer = await env.AI.run("@cf/meta/llama-3.3-70b-instruct", {
      messages: [
        {
          role: "system",
          content: "Answer using ONLY the numbered sources below. Cite them like [1], [2]. If the answer isn't in the sources, say you don't know."
        },
        {
          role: "user",
          content: `Sources:\n${context}\n\nQuestion: ${question}`
        }
      ]
    });

    return Response.json({ answer: answer.response, sources: matches.matches });
  }
};

Chunking, done right

Chunking is where beginners lose. Rules that work:

• Chunk size: 400–800 tokens. Too small = fragments; too large = irrelevant material bleeds in.
• Overlap: 50–100 tokens between chunks. Prevents ideas being split at the seam.
• Split on structure, not length. Prefer breaking on headings, paragraphs, or sections rather than word count.
• Attach metadata. Every chunk should carry: source filename, page, section, project, date. You'll thank yourself at query time.

RAG vs. fine-tuning

People often conflate these. They solve different problems.

You almost always want RAG first. Fine-tune only when the model gets facts right but consistently misses a house style you can't fix with a prompt.