Fact-Checking AI Hallucinations

Version 2.0 Standard: Premium

AI does not know things. It predicts text. When it predicts confidently, the output looks like a fact. When it predicts a plausible but false fact, the output is a hallucination — and it looks exactly as confident as a true statement. You cannot tell the difference by reading the AI's output. The only defense is a systematic verification process applied to every claim, statistic, and name in every AI draft.

---

Part 1 — What Hallucinations Look Like

The Hallucination Spectrum

Types of Hallucination

Type	Example	Danger Level
Fabricated statistics	"According to a 2024 HubSpot study, 73% of marketers..." (study doesn't exist)	🔴 High — readers may cite your fake stat
Invented citations	"As noted in Smith et al. (2023)..." (paper doesn't exist)	🔴 High — destroys credibility if checked
Outdated information	"WordPress powers 43% of the web" (it was true in 2023, not 2025)	🟡 Medium — technically wrong, but close
Confident wrong claims	"Google's PageRank algorithm was deprecated in 2018" (incorrect)	🔴 High — presented with authority
Plausible nonsense	"The Flesch-Kincaid readability score measures semantic density" (it doesn't)	🟡 Medium — sounds right, is wrong
Fictional products/features	"Mailchimp's AI Subject Line Generator" (doesn't exist as described)	🟡 Medium — readers discover the lie when they try

Why AI Hallucinates

AI hallucinations happen because:

1. Training data noise: The model learned from web content — which includes errors, satire, and outdated information
2. Probability optimization: AI generates the most probable next word, not the most accurate one
3. No knowledge boundary: AI cannot say "I don't know." It always generates something
4. Date blindness: AI has a training cutoff and cannot access real-time information
5. Source conflation: AI merges facts from different contexts into a single, incorrect statement

The Confidence Trap

AI presents hallucinations with the same confidence as verified facts. There is no linguistic signal that a statement is fabricated. You simply cannot distinguish truth from hallucination by reading the AI output. Verification must be external.

---

Part 2 — The Verification Workflow

The 3-Pass Verification Process

flowchart TD
    A[AI Draft Complete] --> B[Pass 1: Flag\nEvery Claim]
    B --> C[Pass 2: Verify\nFlagged Claims]
    C --> D[Pass 3: Source\nAll Statistics]
    D --> E{All Claims\nVerified?}
    E -- Yes --> F[Draft Approved\nfor Editing]
    E -- No --> G[Remove or Replace\nUnverifiable Claims]
    G --> F

    style F fill:#217346,color:#fff
    style G fill:#F4A261,color:#000

Pass 1: Flag

Read the draft and highlight every:

• Statistic — any percentage, number, or quantified claim
• Named source — any study, report, person, or organization cited
• Factual claim — any assertion about how something works
• Product feature — any specific capability attributed to a tool
• Date-sensitive fact — any claim that could have changed since the AI's training data

Rule: If it sounds like a fact, flag it. Err on the side of over-flagging.

Pass 2: Verify

For each flagged item, verify using one of these methods:

Method	When to Use	How
Primary source	Statistics and citations	Find the original report/study. Check the exact number
Official documentation	Product features and capabilities	Visit the product's website or docs
Google search	General facts	Search the claim in quotes. If 0 results, it's likely fabricated
Recency check	Dated claims	Verify the claim is still accurate as of today
Cross-reference	Definitions and processes	Check 2–3 authoritative sources

Pass 3: Source

Every statistic that passes verification must be attributed in the text:

• ❌ "73% of marketers use content marketing" — no source
• ✅ "73% of marketers use content marketing (Content Marketing Institute, 2024)" — sourced

If a stat is real but you can't find the primary source, remove it. An unsourced stat is as bad as a fake one from the reader's perspective.

---

Part 3 — High-Risk Categories

Some content areas are more prone to hallucination than others. Apply extra scrutiny here.

Category	Risk Level	Common Hallucinations	Verification Priority
Statistics and data	🔴 Very High	Fabricated percentages, wrong study years	Always verify against primary source
Legal and compliance	🔴 Very High	Wrong regulations, outdated laws	Always consult official sources
Medical/health claims	🔴 Very High	Incorrect dosages, unsupported health claims	NEVER publish without expert review
Product comparisons	🟡 High	Wrong pricing, features that don't exist	Check each product's official page
Historical facts	🟡 Medium	Conflated timelines, wrong dates	Cross-reference with 2+ sources
Technical processes	🟡 Medium	Plausible but incorrect instructions	Test the process yourself if possible

---

Part 4 — Bad vs. Good Examples

❌ Unverified AI Claims

"According to a 2024 study by McKinsey, 87% of B2B buyers prefer to complete their purchase journey digitally without interacting with a sales representative. This represents a 23% increase from 2022. The shift has led companies like Salesforce and HubSpot to invest heavily in self-serve purchasing platforms."

(Why it fails: This stat sounds plausible — McKinsey does publish B2B research. But the specific "87%" and "23% increase" need verification. If you search for this exact study, you may find similar numbers but not this exact claim. The 87% may be a hallucination based on a real 2022 finding of 70%.)

✅ Verified and Sourced

"McKinsey's 2022 B2B Pulse survey found that 70% of B2B decision-makers are willing to make fully self-serve purchases up to $50,000 — and 27% are comfortable self-serving purchases above $500,000 (McKinsey, 'The B2B Digital Inflection Point,' February 2022).

Note: I could not verify more recent data from McKinsey on this specific metric. The 2022 figure is the most recent primary-source number available as of this writing."

(Why it wins: Exact study named. Exact year cited. Exact data point quoted. And critically — the writer acknowledges the data's age instead of inventing a 2024 update.)

---

Part 5 — AI Collaboration Guidelines

The "Hallucination Detector" Prompt

Role: Research fact-checker Task: Review this draft and flag every statement that could be a hallucination. For each flagged item:

1. Quote the exact claim
2. Rate the hallucination risk: Low / Medium / High
3. Suggest how to verify it (specific source to check)
4. If the claim is likely fabricated, suggest a real alternative or recommend deletion Rules: Flag ALL statistics, ALL named studies, ALL product features, and ALL date-specific claims. Input: [Paste Draft]

AI checking AI is not reliable.

Using one AI model to verify another AI model's facts does NOT work. Both models may hallucinate the same plausible-sounding information. The verification step MUST use primary human sources — real websites, real reports, real documentation.

---

Part 6 — Output Checklist

Before moving to the next lesson, confirm every item below.

• Hallucination types: You can identify 6 types of AI hallucination.
• 3-pass process: You flag, verify, and source every claim in every AI draft.
• Primary sources: Every statistic links to or names its primary source.
• High-risk awareness: You apply extra scrutiny to statistics, legal claims, and product features.
• Removal discipline: If you can't verify a claim, you delete it — no exceptions.
• No AI-on-AI verification: You never use AI to fact-check AI.

---

Internal use only. Do not distribute externally. For questions or suggested updates, raise with the content lead.