Why AI-written drafts contain mistakes

What the mistakes are

A hallucination is a response from an AI tool that contains false or misleading information presented as fact¹. The word is borrowed from psychology. It's a loose comparison, and some people dislike it. The plain meaning is simple: the text sounds sure of itself and is wrong.

Chatbots built on large language models, the kind of tool behind most AI writing assistants, may embed plausible-sounding random falsehoods in their text, such as fabricated citations¹. A fabricated citation is a reference to a report, law or article that doesn't exist, or that doesn't say what the draft claims.

For a small business, the practical point is narrow. A draft can have perfect grammar and a tidy layout. It can still name the wrong supplier, state a rule that doesn't exist or quote a figure nobody published.

Why they happen

There isn't one cause. Wikipedia's summary lists several.

• Guessing can be built in. The pre-training of generative pretrained transformers (GPT) involves predicting the next word, which incentivizes them to "give a guess" even when they lack information¹.
• The training material has gaps. Errors can come from incomplete, inaccurate or unrepresentative datasets¹. A dataset is the large collection of text a tool learns from.
• Friendliness has a cost. Training a chatbot in favour of a friendlier personality tends to reduce accuracy, because it's more inclined to agree with the user¹.

That last point matters for everyday prompts. If you ask "write a paragraph saying our product meets the standard", an agreeable tool may simply write it.

Why a smooth draft is harder to check

Fluent text doesn't look like it needs checking. Researchers who study people using automated aids describe automation bias. One form is when users follow an automated directive without taking into account other sources of information².

One explanation offered is the tendency of humans to view automated aids as having an analytical ability superior to their own². In writing terms, a polished sentence gets the benefit of the doubt it hasn't earned.

What stays the same

Some things don't change from draft to draft.

• The tool doesn't know your business unless you've told it, and it can't tell which details you left out.
• A longer, more detailed answer has more places to be wrong.
• Finding one mistake is a reason to look harder for others of the same kind.

Exceptions

Not every draft is risky. A rewritten sentence that only changes your own wording, or a list of headings for a document you already wrote, carries little room for invented facts. The risk rises when the draft adds something new: a number, a name, a date, a rule, a quote or a source. The next pages cover what to check and a routine to follow before you send.