Last week I asked ChatGPT to summarize a 40-page vendor contract. It told me the termination clause required "90 days written notice." The actual number, sitting right there in section 14.2, was 60. Not a huge gap — but if I'd forwarded that summary to a client without checking, it would've been a real problem.
That's the thing about AI "hallucinations" (the term everyone uses, though "confident guessing" is closer to what's happening): the model isn't lying, and it isn't randomly broken. It's doing exactly what it's built to do — predict the next most plausible word — and most of the time that produces something true. Sometimes it produces something plausible-sounding instead. The output looks identical either way, which is what makes it dangerous.
It sounds exactly the same whether it's right or confidently guessing. There's no nervous tone, no hedge — which is why the fix has to happen on your end, not by listening for uncertainty that isn't there.
Three tests I actually ran
Not in the abstract — these are the situations where I personally caught it making things up, testing this for this site.
"Three studies on remote work productivity," with two real citations and one fabricated one — a plausible paper title, a real-sounding journal, wrong year.
Only two of the three sources existed. The fake one read exactly as convincingly as the real ones — nothing in the phrasing gave it away.
Asked who currently holds a specific minor political office, it gave a full, confident, named answer.
That person had left the role over a year earlier. The tool had no live web access, and nothing in the answer flagged that it might be outdated.
I also ran the contract test three separate times with the identical PDF: 90 days twice, 60 days once. The wrong answer wasn't random noise — 90 is simply a more common contract term, so the model leaned on that pattern instead of the actual text in front of it.
The pattern behind all three
Notice what these have in common: they're all places where the correct answer isn't the most statistically likely string of words. A contract's actual number is arbitrary — nothing about the word "termination" makes 60 more probable than 90 in general English. Real citations and current job-holders are the same kind of arbitrary, specific fact. General knowledge questions ("how does compound interest work") don't have this problem, because the correct explanation genuinely is the common pattern.
The more arbitrary and specific the fact, the more it needs checking separately — exact numbers, names, dates, quotes, and citations, specifically.
Stress-test a prompt before you rely on it
The free Prompt Checker flags places where a prompt is likely to invite a confident-but-unverified answer — no signup.
Try the Prompt Checker →What I do differently now
- For documents: I ask the model to quote the exact sentence it's basing an answer on. If it can't produce the quote, I go find the section myself.
- For citations or sources: I search for the source by name before using it anywhere. Takes about 20 seconds, has caught a fabricated one roughly 1 in 8 times for me.
- For anything time-sensitive: I check whether the tool has live web access at all. If it doesn't, I treat any current-events answer as a starting guess, not a fact.
- For high-stakes single numbers: I ask the same question twice, worded differently, in separate conversations. Two different answers means I've found a weak spot before anyone else does.
The short version
AI doesn't know when it's wrong. It sounds exactly the same whether it's right or making an educated guess. The fix isn't a rule of thumb about when to trust it — it's a two-second verification habit for the specific kinds of facts where being wrong actually costs you something.
Frequently asked questions
Not reliably from tone alone — confident wrong answers read identically to confident right ones. The reliable signal is the type of fact: arbitrary specifics like exact numbers, citations, and current job-holders carry more risk than general explanations.
Sometimes — asking it to quote the exact source line it's basing an answer on works better than asking "are you sure," since it forces a check against the actual text rather than a restated guess.
The rate has been dropping, and tools with live web search do noticeably better on current-events and citation questions. But the underlying cause — predicting plausible text rather than retrieving verified facts — hasn't gone away, so the verification habit still matters.
Search for the source by name before using it anywhere. It takes about 20 seconds and catches fabricated sources a meaningful fraction of the time.