How to Avoid Hallucinations in Enterprise RAG Systems

TL;DR: A naive RAG pipeline retrieves, generates, and hopes. A governed one answers only from evidence it can trace to a source. To reduce hallucinations in enterprise RAG, put eight controls between the question and the answer. Version the corpus, search by keyword and meaning together, resolve ambiguity first, refuse weak evidence, verify every citation in code, treat retrieved text as untrusted, fetch live data from the system of record, and score retrieval and faithfulness separately.

In 2024, a Stanford team ran expert-graded legal questions through commercial research tools built on retrieval-augmented generation (RAG). Lexis+ AI and Ask Practical Law AI hallucinated 17% of the time. Westlaw AI-Assisted Research hallucinated 33%. GPT-4 without retrieval hit 43% (Magesh et al., 2024).

These were not weekend prototypes. They sat on curated legal databases, built by companies whose business depends on getting citations right. Retrieval lowered the error rate and still left a third of one product's answers wrong. The vendors disputed how the study defined hallucination, and the tested versions date from mid-2024, so treat the exact figures as a snapshot. The direction of the result is the part that holds up.

For an enterprise team, that result describes the daily problem. The right document is in the index, and the answer still contradicts it.

What counts as a RAG hallucination? An answer containing a claim the retrieved sources do not support, whether or not the claim happens to be true. The working test is faithfulness. Every claim should trace to a passage the system actually retrieved.

The fix lives in the pipeline around the model. Pulling the current version out of three conflicting policies, filtering by the asker's country, and refusing when nothing qualifies are engineering problems. A bigger model does not solve any of them.

Why hallucinations in enterprise RAG systems happen

Calling every wrong answer "the model hallucinated" hides where the failure sits. The same wrong sentence can come from three different places, and each one needs a different gate.

Take an HR assistant answering leave questions.

• A retrieval error is a query about clause 14.2 returning paragraphs from clause 4.1, because the embedding blurred the clause number into noise.

• A context error is a 2021 policy and its 2025 replacement both reaching the prompt, and the model quoting the old one.

• A generation error is a passage that says "30 days from delivery" and an answer that says "within 30 days of purchase."

A naive pipeline is an open loop. It embeds the question, grabs the top k chunks, pastes them into a prompt, and returns whatever the model writes. Nothing checks whether the right passage arrived, whether the answer stayed inside it, or whether the system should have answered at all.

A governed pipeline closes the loop with feedback at every stage. The same open-loop versus closed-loop distinction runs through agentic AI architecture and design patterns.

The rest of this guide builds that loop one control at a time, starting where most failures begin.

Clean and version the corpus before you tune anything

Most enterprise hallucinations start before the model is involved. A company has three versions of its travel policy, a 2021 PDF still in the index, and a wiki page that contradicts the HR handbook. Retrieval returns all of them, and the model picks one.

That collision has a name, context clash, and it is one of four ways a context window goes bad. The PromptsBuddy podcast episode on context engineering walks through all four.

Check before indexing:

• Owner, effective date, version, and status (current, superseded, draft) on every document

• Near-duplicate files removed, with one canonical copy kept

• Document ID stored on every chunk, so one delete removes every trace

• Access groups stored on every chunk as metadata

• A short document-level header prepended to each chunk before embedding

Check at query time:

• Status filter set to current by default

• User permissions applied as a filter before the prompt is built, never after the answer is written

• Region, role, and product version passed as filters

Three of these deserve a closer look.

Deletions are the quiet failure. A retired policy that survives in the vector index keeps getting cited with full confidence. Wire deletes from the source system to the index through the document ID.

Permission changes deserve the same urgency as deletions. When someone loses access to a folder, their next query should stop returning its chunks. Syncing group membership nightly leaves a window in which the assistant quotes documents the user may no longer read.

Chunk headers are the cheap win. Anthropic reported that combining contextual embeddings with contextual BM25 cut top-20 retrieval failures by 49%, from 5.7% to 2.9%. Adding a reranker took the cut to 67%, down to 1.9% (Anthropic, 2024). The context text is usually 50 to 100 tokens, and the mechanics of contextual chunk headers include a worked policy example.

Cleaning the corpus is unglamorous and it is the cheapest fix on this list. Skip it, and every later control has to work against a corpus that disagrees with itself.

Search by keyword and meaning together

Enterprise questions mix identifiers with intent. "Does clause 14.2 cover SKU 88-1204 shipments to Germany?" Dense embeddings often handle a rare string like 88-1204 poorly. Keyword search (BM25) matches the string exactly but misses a paraphrase it has never seen. Run both.

Fuse the lists, then rerank

Reciprocal rank fusion merges ranks instead of scores, so BM25 and cosine similarity never need calibrating against each other. Its constant of 60 comes from Cormack, Clarke, and Buettcher's 2009 study.

A cross-encoder reranker then reads the query and each passage together, which catches relevance the first stage misses. A common setup pulls about 150 candidates, fuses the lists, and reranks down to a short list that fits the prompt. The full funnel is laid out in this guide to hybrid retrieval with reciprocal rank fusion and a reranker.

Chunk by structure

Split on headings, clauses, and table boundaries. Keep a table with its header row, because a row of numbers without its column names invites a guess. Test how your PDF parser handles tables, since garbled extraction quietly produces wrong figures.

Place evidence deliberately

Liu et al. found in Lost in the Middle that performance is often highest when the relevant passage sits at the start or end of the context and degrades significantly when it sits in the middle, even for models built for long contexts. Put the strongest passage closest to the question and send fewer passages. A short, well-ranked list usually beats a long, mixed one, because every extra distractor is one more thing for the model to blend into the answer.

Retrieve for recall and rerank for precision. The prompt only has room for precision.

Resolve ambiguity before you retrieve

Many wrong policy answers are correct answers to a question nobody meant. "What is the notice period?" depends on country, contract type, and start date. A bot that guesses will sound sure and be wrong for some of the staff.

Three moves handle it.

First, pass known context as filters, not as prose. Employee region, role, plan tier, and product version come from the identity system or CRM and narrow the search before it starts.

Second, rewrite follow-ups into standalone queries. "What about contractors?" retrieves nothing useful on its own. Use the conversation so far to turn it into "What is the notice period for contractors in Germany?" before it reaches the index, and log both versions so a reviewer can see what the retriever actually searched.

Third, when a missing detail would change the answer, ask. One question. "Which country is your contract under?" beats a four-paragraph answer covering every country, because the long answer lets the reader land on the wrong paragraph.

The pattern is borrowed from a prompt in the PromptsBuddy library, make it ask before it answers. It interviews the user one or two questions at a time and labels any assumption it still has to make. Wire the same logic into the retrieval layer so the clarifying question fires before a single passage is fetched.

Place a gate between the evidence and the model

Abstention is the cheapest hallucination control and the most resisted, because demos look better when the bot always answers. In production, a refused question costs a support ticket. A confident wrong answer can cost a refund, a complaint, or a compliance finding.

Build the gate in two layers.

Layer one lives in code

If the top reranker score falls below a threshold, or no passage clears the metadata filters, skip the model call entirely. Return a handoff that names the closest documents and routes to a person. Set the threshold from your golden set (covered below), not by feel. Reranker scores are not comparable across models, so recalibrate whenever the reranker changes.

Layer two lives in the prompt

Short, testable rules beat long, vague ones.

text

Answer only from the numbered sources provided.
If the sources do not contain the answer, say what is missing and stop.
Quote the exact wording for numbers, dates, amounts, and conditions.
Mark anything you infer as [inferred].
Never use outside knowledge for company policy.

The [inferred] label matters most. It lets a reviewer see where the model stepped past the sources, and it gives the verifier a marker to hold to a stricter standard. Prompt rules still fail sometimes, which is why they sit second to the code gate.

Structured data follows the same logic. For text-to-SQL, the SQL-from-a-vague-question prompt says to state exactly what is missing when the schema cannot answer, and to write the closest honest query instead of inventing columns.

Treat answer length as a risk surface

In the Stanford data, the tool with the longest answers hallucinated most. Westlaw averaged about 350 words against 219 for Lexis+ AI, and the authors noted that longer answers carry more falsifiable claims. For high-stakes domains, cap the length and prefer a direct answer plus the quoted source over an essay.

Track the refusal rate. If it climbs past what the business tolerates, the fix is better retrieval, not looser rules.

Give the gate a second try

A refusal does not have to end the request. Corrective RAG (Yan et al., 2024) adds a lightweight retrieval evaluator that scores the retrieved documents and triggers a different action when confidence is low, such as widening the search. In an enterprise setting, the fallback should be another internal source or a human, not the open web.

Agentic RAG goes further and lets the model decide to re-query on its own. Each extra loop is another step that can fail, so cap the retries and score the loop with the compound-error arithmetic shown later in this guide.

A refusal is a feature. A confident wrong answer is an incident.

Verify every citation in code

A citation is a claim about the evidence, so it needs checking like one. In Moffatt v. Air Canada (2024 BCCRT 149), the airline's chatbot reportedly told a customer he could claim a bereavement discount after travel, while the policy page it linked to said otherwise. The tribunal ordered the airline to pay C$812.02 and rejected the argument that the chatbot was a separate legal entity.

Nobody outside the company knows whether retrieval sat behind that bot, but the lesson transfers. A link beside an answer proves nothing about whether the answer matches it.

Five checks turn citations into a control.

1. Number every passage. Give each retrieved chunk a stable ID for the request and require an ID on every claim, at sentence level.

2. Check the IDs exist. In code, confirm each cited ID is in this request's retrieved set. This catches invented citations instantly.

3. Check the passage supports the claim. Run an entailment model or a second LLM judge with a narrow yes-or-no rubric. Drop or flag sentences that fail.

4. Exact-match the risky tokens. Numbers, dates, currency amounts, and negations such as "not refundable" must appear verbatim in the cited passage.

5. Show the span. Display the quoted passage beside the answer so a reviewer can confirm it in seconds.

A production answer contract

A structured answer turns the model's draft into something a checker can test. Here is a parental leave answer, with invented policy text.

json

{
  "abstained": false,
  "claims": [
    {
      "text": "Employees in Germany receive 14 weeks of paid parental leave.",
      "source_id": "S1",
      "quote": "14 weeks of paid parental leave",
      "checks": {
        "source_in_retrieved_set": true,
        "numbers_match_verbatim": true,
        "supported_by_passage": true
      }
    }
  ]
}

The model drafts the claim and picks the source. The checks decide whether it ships. A sentence with any false value is dropped or rewritten before the user sees it.

For the output shape, the summarise-a-document prompt is a useful template. It demands the quoted clause behind every obligation and a separate list of what the document leaves out. That second list matters in enterprise work, where the risk often lives in absence. The missing liability cap. The undefined response time.

Budget the latency and the cost

Every control above adds work, and teams that ignore the budget end up switching the controls off. The reranker scores each candidate against the query, so its cost grows with the candidate count. The judge in check 3 adds a model call per answer.

Four habits keep the bill predictable.

• Run the cheap checks (1, 2, and 4) on every answer, since they are plain code.

• Run the judge synchronously for high-stakes flows and asynchronously for the rest, flagging failures for review.

• Cap the candidate list before reranking and the retry count in the gate.

• Measure p95 latency and cost per answered question on your own stack, because published figures shift with model and hardware.

The model drafts the claim. The checker decides whether it ships.

Treat retrieved text as untrusted input

A RAG system pastes documents into the prompt, and the model cannot reliably tell a document from an instruction. Greshake et al. showed that LLM-integrated applications "blur the line between data and instructions," and that an attacker can plant instructions in content the application later retrieves (Greshake et al., 2023). OWASP lists prompt injection as the first risk in its Top 10 for LLM applications.

In an enterprise index, the poisoned page does not need to come from outside. A shared wiki, a support ticket, or an uploaded PDF is enough. The failure looks like a hallucination, because the answer ignores the real policy and follows text nobody vetted.

Four controls contain it.

• Label the boundary. Wrap retrieved passages in clearly marked source blocks and tell the model that their contents are data, never instructions.
  

• Limit who can write to the index. Treat ingestion like a deployment path, with owners and review for sources that anyone can edit.
  

• Give the model the least power. An assistant that can only answer cannot be talked into sending an email or issuing a refund.
  

• Verify the output anyway. The citation checks catch an answer that drifts to claims no source supports, whatever caused the drift.

No prompt wording guarantees the model will ignore an injected instruction. The strongest mitigation is architectural. Keep irreversible actions behind checks the model cannot override, which is the subject of the next section.

Fetch live data and gate actions

Some facts change faster than any index refreshes. Order status, account balances, stock levels, ticket state. An indexed copy of these guarantees a stale answer eventually, so fetch them from the system of record at answer time. Connecting the agent through an MCP server does that without pasting data into the prompt, and the tools directory on PromptsBuddy marks which of the tools its prompts rely on ship an MCP server.

When RAG feeds an agent that acts, errors compound. Per-step accuracy sounds excellent until you multiply it across a task.

The arithmetic is worked through in the math of compound error. Three guardrails contain it.

Check before the action runs:

• Schema validation on every tool call

• Permissions and business rules

• Risk tier, with irreversible actions routed to a person

Check after the action runs:

• Read the state back from the system of record

• Compare it with the intended effect

• Log the result with the trace

A refund that returns success while the ledger is unchanged is a hallucination too, just one with side effects.

See the controls work together

The controls compose inside a single request. Here is one HR question traced end to end. The policy details and scores are invented for illustration.

An employee in Germany asks how much parental leave they get.

Each step narrowed what the system was allowed to do next. The filters removed the superseded v11 policy before search began. The reranker put the v12 passage first. The gate compared its score against a threshold set from a golden set. The verifier checked that "14 weeks" appeared word for word in the cited passage.

Now a second question. The same employee asks about their equity refresh. No passage clears the gate, so the model is never called. The user gets the three closest documents and a route to the people team.

Count the decisions the model never got to make. Which policy version applied. Whether any evidence existed. Whether a number was safe to print. The model wrote the sentence, and the pipeline made every call that could go wrong.

Score retrieval and faithfulness separately

Teams that grade only "did the answer look right" cannot tell which stage failed. Split the scorecard. The RAGAS paper proposes reference-free metrics for retrieval quality, faithfulness, and generation quality, and its open-source library ships context precision, context recall, and faithfulness.

Why separate scores pay off

End-to-end accuracy is roughly the product of the stages. The table below uses invented rates and assumes the stages fail independently, which real systems only approximate.

In this example, three points of generation work buy less than five points of retrieval work. A team with one blended score would never find that out and would keep tuning the prompt. A team with two scores sees the weaker stage and fixes it first.

Build the golden set from real traffic

Pull 100 to 200 questions from support or search logs. Then add three kinds on purpose. Questions whose answer is not in the corpus. Ambiguous questions. And questions where an old document version holds a wrong answer.

Rerun the set whenever the embedding model, chunking rules, reranker, prompt, or LLM version changes. Compare every metric, not only the headline score, so a gain in one stage cannot hide a loss in another.

Trace every answer in production

Capture the query, the rewritten query, the filters, the retrieved IDs and scores, the prompt version, the model version, the citations, and the verification verdicts. Review a weekly sample by hand and add a "report a wrong answer" control to the interface. Every report becomes a new row in the golden set.

A single impressive demo run is weak evidence. A long streak of passes across the golden set is strong evidence.

Roll it out in four weeks

The order matters because week one produces the numbers that tell you whether weeks two and three worked.

Run this checklist before launch

An assistant that employees or customers will rely on should not go live until all six of these hold.

1. Version the corpus. Owners, dates, statuses, and deletions that propagate. Filter on current by default.

2. Search two ways and rerank. Keyword and vector, fused, then reranked, with the strongest passage placed nearest the question.

3. Refuse weak evidence in code. A score threshold and a handoff, not a hopeful prompt.

4. Verify every citation and every number. Existence, support, and exact-match, checked before the user sees the answer.

5. Treat retrieved text as untrusted. Passages are data, ingestion has owners, and the assistant holds no power it does not need.

6. Score the stages apart. Retrieval and faithfulness separately, on a golden set that includes unanswerable questions.

Models will keep getting better at reading. None of that repairs a corpus that disagrees with itself, a retriever that misses the clause, or a pipeline with no way to say "I don't know."

The quickest audit takes an evening. Pick the ten questions your business can least afford to get wrong, run them through the system, and read the retrieved passages before you read the answers. Then name each failure from the trace.

A team that cannot tell these apart will spend months tuning the wrong one.

A bigger model will not make a RAG system reliable. Reliability comes from the checks wrapped around it.

Conclusion

Hallucinations in enterprise RAG come from the pipeline around the model, so that is where the fixes belong. Start with the corpus and the abstention gate, the two controls that are cheapest to build. Add hybrid retrieval with a reranker and code-level citation checks next. Then wrap retrieved text in injection defenses before the assistant gets any power to act, and hold every later change to a golden set that scores retrieval and faithfulness apart.

No pipeline can promise zero hallucinations, and Stanford's 17% to 33% range for commercial legal RAG tools shows how far retrieval alone falls short. The realistic target is a bounded error rate, where the remaining errors are caught, flagged, or refused before a user acts on them. A team that can show traced answers and a passing golden set has a system worth putting in front of customers.