The problem: a brilliant assistant that doesn't know your business
A foundation model has read an enormous slice of the public internet, but it has never seen your price list, your service contracts, your installation manuals, or last week's policy update. Ask it a question about your company and it has two choices: admit it doesn't know, or — far more often — produce a confident, fluent, plausible-sounding answer that is simply made up. This is called a hallucination, and it's the single biggest reason a raw chatbot is unsafe to put in front of customers.
The fix is not to make the model "smarter." It's to stop asking the model to remember your facts at all. Instead, we look your facts up at the moment of the question and hand them to the model to read. That technique is Retrieval-Augmented Generation, or RAG.
The core idea of RAG: don't ask the model what it remembers — give it the right documents and ask it to answer only from those. The model becomes a careful reader, not an oracle.
Retrieve, then generate
RAG has two steps, and the order matters.
- Retrieve. When a question comes in, the system searches your knowledge base and pulls back the handful of passages most likely to contain the answer. Your documents aren't stored as whole files for this — they're sliced into small, self-contained pieces called chunks (a paragraph or two each), so the system can return just the relevant part instead of an entire 80-page handbook.
- Generate. Those chunks are placed in front of the model along with the user's question and a simple instruction: answer using only the text provided, and if it isn't here, say so. The model then writes a natural-language answer grounded in real, retrieved material.
A concrete example. A customer asks a heating installer's assistant, "Do you service Viessmann boilers in the Stuttgart area?" The system retrieves two chunks — the company's service-area document and its supported-brands list — and the model replies: "Yes, we service Viessmann units across the Stuttgart region; here's how to book." The fact came from your document, not the model's imagination.
How retrieval actually finds the right chunk
Old-fashioned search matches keywords. If your document says "annual maintenance" and the customer types "yearly service," a keyword search misses it. RAG usually relies on something better.
Vector (semantic) search
Every chunk is converted into a list of numbers — a vector — that captures its meaning, not its exact words. The question gets turned into a vector too, and the system returns the chunks whose meaning sits closest to the question's. "Yearly service" and "annual maintenance" land near each other because they mean the same thing. This is semantic search: matching on meaning rather than spelling. (Those meaning-vectors, called embeddings, are introduced in the foundation-models lesson.)
Hybrid search
Pure meaning-matching can stumble on exact identifiers — a part number like "WB2C-19" carries little "meaning" to match on. So most serious systems run hybrid search: classic keyword matching and semantic matching together, then blend the results, giving you the precision of exact terms and the flexibility of meaning. You won't tune this yourself, but knowing the term explains why a good grounded assistant rarely misses the obvious.
Why citations are the whole point
Because a RAG answer is built from specific retrieved chunks, the system knows exactly which sources it used. That means it can show them. A grounded answer can carry a citation — "according to your 2026 service policy, section 4" — that a human can click and verify.
This is not decoration. Citations turn an answer from "trust me" into "check me." A staff member can confirm each claim against its source in seconds; a customer sees the answer anchored to a real document, not a guess. It also flips the safety model: instead of hoping the model didn't lie, you can prove what it drew from. That makes an answer defensible when accuracy matters — and we build on this idea of human verification in the responsible-AI lesson.
RAG vs fine-tuning: keeping facts current
There's another way to teach a model your information: fine-tuning, where you continue training the model on your data so the facts get baked into its internal weights. For factual, fast-changing knowledge, this is usually the wrong tool.
- Fine-tuning bakes facts in. Change a price, and the model still confidently quotes the old one until you retrain it — an expensive, slow, and skill-heavy process. The knowledge goes stale the moment your business does.
- RAG keeps facts separate. Knowledge lives in documents the model reads at question time. Update a document — fix a price, add a new policy, retire a product — and the very next answer reflects the change. No retraining, no GPUs, no waiting.
Rule of thumb: fine-tune to change a model's style or skill; use RAG to change what it knows. For SME knowledge that changes weekly, RAG wins because editing a document is something your team can already do.
This separation has a quieter benefit too: you can read your knowledge base, correct it, and audit it. You can't read a fine-tuned model's weights.
How grounding maps onto a governed assistant
This is where RAG stops being a trick and becomes a governance model. On the Wegenty platform, one local model serves many knowledge bases, and grounding draws a clean line between what's private and what's public.
- The internal knowledge base is the superset. Everything your organisation knows — including sensitive, draft, or internal-only material — lives here. Your staff's assistant is grounded in all of it.
- The public agent is a governed projection of it. The customer-facing assistant is grounded only in the subset of chunks you've explicitly approved for the outside world. Knowledge flows from internal to public by deliberate promotion — never the reverse. Nothing reaches a customer that someone didn't approve.
When the public agent can't find an approved chunk that answers a question, the right behaviour is not to improvise — it's to say so and hand off to a human. That human's answer can then be captured back into the knowledge base through a correction loop, so the next person who asks gets a grounded, cited answer automatically. The assistant gets better the way a well-trained employee does: by writing down what it learns.
You can watch this play out — grounded answers, real citations, graceful hand-offs — in the live demo, where the agent answers on a real corpus without anything leaving the building.