What Is Retrieval-Augmented Generation (RAG)? A Leader's Guide

Retrieval-augmented generation (RAG) is a way of building an AI assistant so that it looks up relevant passages in your own documents before it writes an answer. Instead of relying only on what a large language model memorized during training, the system retrieves current, approved content (such as your HR handbook or IT knowledge base) and asks the model to answer from it. For business leaders, RAG is the most common pattern behind "ask our policies" chatbots, and its quality depends far more on your content and access controls than on the model itself.

This guide explains how RAG works step by step, how it compares to fine-tuning and prompt-only approaches, where it fits in HR and IT, and what risks and governance steps you should plan for before you approve a rollout.

Key takeaways

• RAG pairs a language model with a search step over your own content, so answers can reflect current policies instead of stale training data.
• The pipeline has five parts: indexing, embeddings, vector search, retrieval and generation. Weakness in any one of them shows up as a bad answer.
• RAG reduces but does not remove hallucinations. The model can still misread or ignore the passages it was given.
• The biggest enterprise risks are access control, data leakage and prompt injection hidden inside documents.
• Treat a RAG assistant as a governed system: named content owners, permission-aware retrieval, test sets and ongoing monitoring.

What is retrieval-augmented generation in plain terms?

RAG is an "open book exam" for AI: the model gets to read the relevant pages before answering, rather than answering from memory. The term comes from a 2020 research paper that combined a language model with a searchable index of documents.

In that original RAG paper by Lewis and colleagues, the authors describe two kinds of memory. Parametric memory is the knowledge stored inside the model's weights during training. Non-parametric memory is an external store the model can look things up in, which in their case was a dense vector index of Wikipedia. The researchers reported that the combined approach produced answers that were more specific and more factual than a model working from its own weights alone.

For a business, the external store is usually your own material: policy documents, benefits guides, SOPs, IT runbooks and FAQs. A large language model (LLM) is the text-generating engine, and RAG is the wiring that feeds it the right excerpts at the right moment.

How does RAG work, step by step?

RAG works in two phases: you prepare your content once (and keep it fresh), then each user question triggers a search followed by an answer. Here is the full sequence.

1. Indexing. Your documents are collected, cleaned and split into small passages called chunks (often a few paragraphs each). Each chunk keeps metadata such as source, owner, effective date and who is allowed to see it.
2. Embeddings. Each chunk is converted into an embedding, a long list of numbers that represents its meaning. Passages about "parental leave" and "time off after a birth" end up with similar numbers even though the words differ.
3. Vector search. The embeddings are stored in a vector database, a store built to find the closest matches by meaning. When an employee asks a question, the question is also turned into an embedding and compared against the stored chunks.
4. Retrieval. The system picks the top few matching chunks, ideally filtered by the user's permissions, location and role, and sometimes re-ranked for relevance.
5. Generation. The retrieved passages are placed into the prompt alongside instructions such as "answer only from these sources and cite them." The LLM writes the answer, and a good system shows the source links so the user can check.

Notice that the model never "learns" your handbook in this process. The handbook stays in your index, which is why updating a policy can be as simple as replacing a document and re-indexing it.

How does RAG compare with fine-tuning and prompt-only approaches?

RAG is usually the best fit when answers must reflect specific, changing documents; fine-tuning is better for teaching a model a style or a narrow task; prompt-only works for general help with no company knowledge. Many mature programs combine them.

Fine-tuning means further training a model on your examples so its weights change. Prompt-only means using an off-the-shelf model with carefully written instructions and no retrieval step.

Factor	Prompt-only	RAG	Fine-tuning
Where knowledge lives	Model training data plus the prompt	Your document index, looked up per question	Model weights, changed by extra training
Keeping content current	Manual, limited by prompt size	Update and re-index documents	Retrain when facts change
Citing sources	Not reliable	Built in: can show the passages used	Hard: knowledge is blended into weights
Access control	Not applicable	Possible per user at retrieval time	Very hard: anything trained in can surface for anyone
Best for	Drafting, summarizing, general questions	Policy, benefits and IT answers grounded in documents	Tone, format, classification or narrow specialized tasks
Main risk	Confident answers with no company grounding	Bad or outdated content, leaky permissions, injected instructions	Sensitive data baked into the model; costly to correct

The access control row matters most for people leaders. If you fine-tune a model on salary bands or investigation notes, you cannot easily stop that knowledge from appearing for the wrong person. With RAG, you can filter what gets retrieved for each user before the model ever sees it.

Where does RAG help HR and IT teams?

RAG is strongest wherever employees ask repeat questions that already have a written answer somewhere. The value comes from finding and summarizing that answer quickly, with a link back to the source.

• HR policy assistants. Questions about leave, holidays, expense rules or dress codes. Our guide on whether AI HR assistants can answer routine HR policy questions covers which questions are safe to automate and which need a person.
• Benefits and open enrollment support. Plan summaries change every year, so a retrieval approach that points to the current plan documents beats a model trained on last year's.
• IT help desks. Password resets, VPN setup, device policies and software request steps drawn from runbooks and knowledge articles.
• Manager guidance. Step-by-step processes for onboarding, performance cycles or approvals, scoped to the manager's region and business unit.
• Internal search for frontline teams. SOPs and safety procedures surfaced on a phone in plain language.

Use cases that involve decisions about individual people (pay, discipline, hiring) need more than retrieval. They need documented human review, which is a governance question as much as a technical one.

What are the main risks of RAG?

The main risks are wrong answers that sound right, answers that reveal information a user should not see, and documents that carry hidden instructions to the model. Each has a known set of controls.

Hallucinations still happen

NIST calls this risk "confabulation" and its Generative AI Profile (NIST AI 600-1) lists it among twelve risk categories, describing it as confidently stated but erroneous or false content. RAG lowers the odds by giving the model real sources, but the model can still blend two policies, miss an exception, or answer when nothing relevant was retrieved. Outdated or conflicting documents in the index make this worse. See our piece on AI hallucinations in HRIS platforms for how these errors show up in practice.

Access control and data leakage

If every document goes into one shared index with no permission filter, the assistant can quote a confidential document to anyone who asks the right question. The OWASP entry on vector and embedding weaknesses warns about unauthorized access, cross-context leakage between groups sharing a vector database, and poisoned knowledge bases. It recommends permission-aware vector stores, partitioned datasets and logging of retrieval activity. Employee data also carries privacy obligations, covered in our guide to generative AI and employee data privacy under GDPR and CCPA.

Prompt injection

Prompt injection is text crafted to override the system's instructions. The OWASP Top 10 for LLM Applications entry on prompt injection describes the indirect form, where instructions are hidden inside content the model reads, and gives a RAG example: an attacker alters a document in the repository so its hidden text skews answers. OWASP notes that RAG does not fully mitigate this risk and recommends least privilege, labeling untrusted content, output checks, human approval for high-risk actions and regular adversarial testing.

Data supply chain integrity

Your index is only as trustworthy as what feeds it. CISA, the NSA and the FBI released joint best practices for securing AI data that stress data integrity across the whole AI lifecycle, from development through operation. For RAG, that means controlling who can add or edit source documents.

How should leaders govern and evaluate a RAG system?

Govern a RAG assistant like any other system of record: assign owners, control inputs, test before launch and monitor after. The NIST AI Risk Management Framework, a voluntary framework released in January 2023, organizes this work under four functions: Govern, Map, Measure and Manage.

In practice, that translates into a few concrete habits:

• Content ownership. Every source collection has a named owner who approves what goes in, sets an effective date and retires old versions.
• Permission-aware retrieval. Retrieval filters mirror your existing access rules (by role, location, employment type) rather than inventing new ones.
• A test set before launch. Write 100 or so real questions with known correct answers, including tricky ones (state-specific rules, exceptions, questions the bot should refuse). Score accuracy, citation quality and correct refusals.
• Ongoing monitoring. Review unanswered questions, thumbs-down feedback and a sample of conversations each month. Rerun the test set after every content or model change.
• Clear escalation. The assistant should hand off to a person for sensitive topics such as accommodations, complaints or anything involving a specific employee case.

If you are building the wider program, our walkthrough on how to set up an AI governance framework step by step shows where a RAG assistant fits. When you buy rather than build, use an AI vendor due diligence questionnaire to ask how retrieval permissions, logging and data retention actually work.

Common mistakes when rolling out RAG

Most failed RAG pilots fail on content and permissions, not on the model. Watch for these patterns:

• Indexing everything. Dumping a whole shared drive into the index pulls in drafts, duplicates and expired policies, which produces contradictory answers.
• Ignoring permissions until later. Retrofitting access control after launch is harder than designing it in, and a single leak can end the project.
• No citations. If users cannot see where an answer came from, they cannot catch errors, and trust erodes after the first mistake.
• Judging by demos. A polished demo on five easy questions says little. A written test set with edge cases says much more.
• No owner after go-live. Policies change. Without someone responsible for re-indexing, the assistant quietly drifts out of date.
• Measuring only usage. Track answer accuracy, escalation rates and resolution, not just how many questions were asked. Our guide on measuring the ROI of an HR AI assistant with real metrics lists practical measures.

FAQ

Does RAG stop AI hallucinations?

No. RAG reduces them by giving the model relevant sources, but the model can still misread passages or answer when retrieval found nothing useful. Citations, refusal rules and regular testing are what keep errors visible and manageable.

Is RAG cheaper than fine-tuning?

It is usually simpler to maintain because updating knowledge means updating documents, not retraining a model. Total cost still depends on content cleanup, the vector database, model usage and the people needed to own and test the system.

Can a RAG assistant respect employee data permissions?

Yes, if it is designed to. Retrieval can filter chunks by the user's role, location and other attributes before anything reaches the model. If filtering happens only after the answer is written, or not at all, sensitive content can leak.

Do we need a vector database to use RAG?

Most RAG systems use one because it finds passages by meaning rather than exact keywords. Some combine vector search with traditional keyword search, which helps with policy codes, form numbers and names that meaning-based search can miss.

What to do next

Start small and measurable. Pick one well-documented domain, such as leave policies or IT access requests, name a content owner, clean up the source documents and write a test set before you choose a tool. Confirm how permissions will be enforced at retrieval time and how injected or poisoned content will be caught. If your assistant will later take actions (submitting requests, changing records), review our guidance on governing AI agents in HR with permissions, gates and audit logs, because acting is riskier than answering.

Sources

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv)
• NIST AI Risk Management Framework
• NIST AI 600-1: Generative Artificial Intelligence Profile
• OWASP Top 10 for LLM Applications: LLM01 Prompt Injection
• OWASP Top 10 for LLM Applications: LLM08 Vector and Embedding Weaknesses
• CISA: New Best Practices Guide for Securing AI Data Released

Last reviewed: November 2026.