LLM hallucination detection is a review process, not a single prompt or tool. This checklist shows how to separate generated claims from evidence, classify risk, verify citations, and decide when human approval is required before an AI-generated answer is used.
Overview
An LLM hallucination is an answer that presents unsupported, inaccurate, or invented information as if it were reliable. The problem can appear as a fabricated citation, an incorrect summary of a source, a made-up technical detail, or a confident answer to a question the system cannot support.
Detecting hallucinations starts with a change in mindset: do not review an AI answer as one block of prose. Break it into individual claims and compare each important claim with an appropriate source. This approach works for chat assistants, retrieval-augmented generation (RAG) applications, internal automation, support workflows, and editorial review.
A useful verification workflow has five stages:
- Extract claims: identify statements that could be true or false.
- Classify risk: determine which claims could cause harm if wrong.
- Locate evidence: find the source, record, calculation, or user input that supports the claim.
- Check the match: confirm that the evidence actually supports the wording and scope of the answer.
- Record a decision: approve, revise, remove, or escalate the claim.
This is more dependable than asking an LLM to judge its own answer. A second model can help identify questions for review, but model agreement is not proof of factual accuracy. For application design context, see the prompt evaluation framework for LLM applications.
Checklist by scenario
1. General question answering
For a low-stakes informational answer, begin by marking every factual statement, number, date, named entity, quotation, and causal explanation. Then ask:
- Is the claim specific enough to verify?
- Does it depend on information that may have changed?
- Does the answer distinguish known facts from assumptions or examples?
- Are limitations stated where the available context is incomplete?
Check at least the claims that directly answer the user’s question. If the answer contains several minor details, it is often safer to remove unsupported details than to preserve them for completeness.
2. RAG and knowledge-base answers
Retrieved passages reduce the chance of unsupported output, but retrieval alone does not guarantee a grounded answer. Review the complete chain:
- Was the relevant document retrieved?
- Is the document current for the use case?
- Does the cited passage contain the information being claimed?
- Has the model added a conclusion that the passage does not support?
- Does the answer preserve qualifiers such as “may,” “only,” “except,” or “as of”?
Use a claim-to-source record rather than a general citation list. One source may support only one part of a paragraph. A practical record can use these fields:
- Claim: the exact sentence or proposition produced by the model.
- Evidence: the source identifier and relevant passage.
- Relationship: supported, partially supported, contradicted, or not found.
- Action: keep, qualify, correct, remove, or send for review.
For implementation details, pair this process with a RAG tutorial for beginners and test retrieval and generation separately.
3. Technical documentation and code
Technical answers need both factual and executable checks. Verify version numbers, function names, configuration keys, syntax, dependencies, permissions, and security implications. A code sample that looks plausible may still fail because it uses an unavailable library, an incorrect parameter, or an outdated interface.
Run examples in a controlled environment where possible. Compare the output with official project documentation or the application’s own test suite. Treat claims about security, data handling, authentication, and destructive operations as high risk even when the code is short.
4. Summaries and transformations
Summarization can hallucinate through omission as well as invention. Compare the output with the source for:
- names, dates, figures, and quotations;
- negation and uncertainty;
- the difference between a proposal, decision, and completed action;
- attribution of statements to the correct person or organization;
- material caveats that change the meaning.
For high-impact summaries, retain the source text and require a reviewer to confirm that the summary’s main conclusions are supported. A text summarizer online can be useful for drafting, but it should not replace source comparison.
5. High-consequence or externally published answers
Increase the review standard when an answer could affect health, safety, finances, legal decisions, access to services, employment, customer trust, or public communication. Require authoritative evidence appropriate to the subject, a named reviewer or approval role, and a clear escalation path for uncertainty.
In these workflows, “the model sounded confident” is not an acceptance criterion. If a claim cannot be verified, label the uncertainty or do not publish it.
What to double-check
Use the following risk classification to decide how much verification each claim needs:
| Risk level | Typical claims | Minimum review |
|---|---|---|
| Low | General explanations, brainstorming, clearly labeled examples | Check for obvious invention and misleading certainty. |
| Moderate | Product behavior, technical instructions, process summaries | Compare with a reliable source or run the procedure. |
| High | Safety, financial, legal, medical, security, or access decisions | Use authoritative evidence and human approval before action. |
Regardless of risk, check these recurring failure points:
- Unsupported specificity: precise figures, dates, names, and quotations deserve direct evidence.
- Source mismatch: a citation may be relevant to the topic but fail to support the exact claim.
- Scope drift: evidence about one product version, location, population, or time period may not support a broader statement.
- Conflicting evidence: the answer should acknowledge meaningful disagreement instead of silently selecting one position.
- Unclear provenance: separate user-provided facts, retrieved passages, model-generated reasoning, and reviewer judgments.
- Prompt injection: retrieved documents or user content may contain instructions that attempt to change the review process. See the prompt injection prevention checklist.
A useful output format makes verification visible. For example:
Claim: [one factual proposition]
Evidence: [source ID, quotation or data point]
Status: supported | partial | contradicted | not found
Reviewer note: [what changed or remains uncertain]
Decision: approve | qualify | correct | remove | escalate
Common mistakes
- Using confidence as evidence: fluent wording and a decisive tone do not establish truth.
- Checking only citations: a citation can be real while the answer misrepresents it.
- Asking the same model whether it hallucinated: self-review may identify useful concerns, but it is not independent verification.
- Reviewing only the final paragraph: unsupported assumptions often enter during retrieval, transformation, or intermediate steps.
- Accepting partial support: a source that supports half of a compound sentence does not support the whole sentence.
- Overloading the prompt with vague instructions: “be accurate” is less useful than requiring claim-level citations, uncertainty labels, and a refusal to guess.
- Skipping regression tests: a prompt, model, retrieval index, parser, or source update can change factuality even when the user-facing feature appears unchanged.
Design the application so that uncertain output is easy to inspect. Structured claims, source identifiers, audit logs, and explicit review states are generally more useful than relying on prose warnings alone. For prompt-level controls, consult the system prompt design guide.
When to revisit
Run this checklist again whenever the underlying inputs or workflow change. That includes a model replacement, prompt revision, retrieval-index rebuild, knowledge-base update, parser change, new tool integration, or change in the audience or stakes of the answer.
It is also worth revisiting before seasonal planning cycles, major product releases, policy or documentation refreshes, and any incident involving an incorrect answer. Keep a small evaluation set of representative questions and known evidence. Re-run it after changes and compare:
- unsupported-claim rate;
- citation support and citation completeness;
- correct handling of uncertainty and conflicting sources;
- refusal or escalation behavior for high-risk requests;
- review time and the number of claims requiring manual correction.
End each review with a concrete action list: update the prompt, improve retrieval, revise the source material, add a test case, change the approval threshold, or remove an unsafe automation step. A repeatable process is more valuable than a one-time accuracy check. For a broader testing workflow, see the LLM evaluation and prompt testing guide.
Quick final check: before acting on an AI-generated answer, identify its important claims, locate evidence for each, test whether the evidence matches the wording, classify the risk, and obtain human approval where the consequences justify it.