What “making things up” means
People who use chatbots sometimes find that a reply contains a book that does not exist, a quotation nobody said, a legal case with a convincing name but no record, or a wrong date delivered without any hint of doubt. This is commonly called hallucination, though the word is a loose label rather than an exact scientific term. The tool is not lying in the human sense, because it has no intention to deceive. It is producing text that looks right and is not.
Understanding why helps you decide when to trust a reply and when to check it.
The root cause: plausible, not verified
A chat tool is built on a language model, which is trained to predict likely text, as described in the explanation of what a language model does. The aim during training is to produce wording that fits the pattern of the text before it. It is not to consult a verified record of facts.
Often plausible and true coincide, because the training material contained the right answer many times. But when the model has little or conflicting information, it still produces something, because it is designed to continue the text. A question about a little-known person is a good example. The model may generate a biography with a believable career, believable dates and a believable university, assembled from patterns of how biographies look.
Specific reasons it happens
Gaps in the knowledge. Topics that were rare in the training material, very recent events, and niche or local subjects are weak spots. The model has no “I have not seen this” signal that works reliably.
No built-in fact-checking. There is typically no step where the model looks up a claim and compares it with a trusted source. Some products add search or document tools, which can help, but those are extra layers.
Pressure to answer. Models are shaped by human feedback toward being helpful. An answer tends to be rated better than “I don’t know”, so some tools learn to give an answer even when unsure. Developers work to reduce this, with mixed success.
Patterns without meaning. A citation has a recognisable shape: author, year, journal, title. The model can generate something that fits the shape perfectly without any real paper behind it. The same goes for web addresses, product model numbers and statistics.
Errors in the source material. If the training data contains mistakes or myths repeated widely, the model may repeat them. See the article on training data.
Leading questions. If you ask “Why did scientists prove X?”, the model may accept the premise and explain a result that never happened. The way a question is worded can push it along.
Long or complex tasks. Errors can creep in during multi-step reasoning, arithmetic and long summaries, where one slip carries forward.
Randomness. Many tools introduce a controlled amount of randomness in word choice. Asking the same question twice can give different answers, which is a hint that at least one may be unreliable.
Why it is hard to notice
The writing is fluent and organised. Wrong answers often come in the same smooth, confident tone as right ones, so tone gives no clue. People also tend to trust text that is well written and detailed, even though detail is exactly what a model can invent. The risk is greatest when you ask about something you cannot easily judge yourself.
Where mistakes matter most
Be especially careful in these areas:
- Health. Symptoms, medicines, doses and interactions. A wrong answer can cause harm.
- Law and official processes. Rules differ by place and change over time.
- Money and tax. Figures and eligibility rules can be wrong or out of date.
- Safety. Instructions involving electricity, chemicals or machinery.
- Quotations, citations and statistics. These are common targets for fabrication.
- Information about real people. Invented claims about named individuals can be damaging and unfair.
In these areas use the tool, at most, to understand terms or prepare questions, then confirm with an official source or a qualified professional. The concerns are especially sharp for medical AI and for any use where diagnostic tools are involved, where errors have real consequences.
How to check an answer
- Verify specifics independently. Look up names, dates, numbers and quotations in a reliable source such as an official site, a reputable publication or the original document. Do not ask the same chatbot to confirm itself, since it can repeat the mistake.
- Check that sources exist. Search for the title and author of any cited work. If you cannot find it, treat it as invented. Open any links and confirm they say what the tool claims.
- Give it the material. Paste in the document and ask it to answer only from that text and to quote the relevant passage. You can then compare the quote to the original.
- Ask for uncertainty. A request such as “tell me which parts you are unsure about” can sometimes surface weak points, although the tool’s self-assessment is also imperfect.
- Look for corroboration. If two or three independent reliable sources agree, confidence rises. If you can only find the claim in the chatbot’s answer, be sceptical.
- Compare across attempts. Rephrase the question, or ask again, and see whether the key facts stay the same. Consistent does not prove correct, but inconsistent is a warning.
- Use tools with search or citations carefully. They can improve accuracy, but they can still misread or misquote a source, so open the source itself.
Clearer requests also reduce the problem. Specific context and a direct instruction to avoid guessing can help, as covered in the advice on AI prompts.
Is it getting better?
Developers report progress, and tools that combine models with search or retrieval from trusted documents often make fewer unsupported claims. But the underlying design means the risk cannot be assumed to be gone, and it is sensible to expect occasional errors from any current tool. Treat the output as a draft from a quick, well-read assistant who sometimes bluffs, and make the checking step part of your routine.