.png)
Why AI Makes Things Up (and How We Engineer Around It)
A client asked us recently why their assistant had invented a paragraph of a contract, complete with a clause number that never existed. The honest answer surprised them: it was doing exactly what it was built to do.
A language model has one job. Given everything so far, produce the most plausible next piece of text. Not the true one. The plausible one.
Most of the time, plausible and true are the same thing, because the model has read more than any of us and truth is usually the most likely continuation. That overlap is why the technology feels magical.
But when the model does not know something, it does not stop. There is no gap where the knowledge should be. There is always a most plausible next word, and the machine produces it with the same fluent confidence either way.
This is why hallucination is a slightly misleading word. The model is not malfunctioning. It is interpolating, the way it always does. A clause number that looks exactly like real clause numbers is a triumph of pattern learning. It is just not your contract.
Two consequences follow, and they shape how we build.
First, confidence tells you nothing. The model sounds equally sure when it is right and when it is inventing. Any workflow that relies on the AI seeming certain is built on sand.
Second, you cannot fix this with a better prompt. You can reduce it, and newer models are meaningfully better at admitting they do not know. But "please do not make things up" is a request, not a mechanism. Production systems need mechanisms.
Here is the stack we build instead. Four layers, from foundation to final check.
Ground it. The model answers from retrieved documents: your contracts, your manuals, your records, not from its training memory. If the answer is not in the documents, the correct output is "not found", and we test for exactly that case before go-live.
Constrain it. Where outputs feed a system, we do not accept free text. Structured fields, allowed values, schemas. A model that must fill in currency, amount and IBAN cannot wander into prose.
Verify it. Everything checkable gets checked by ordinary code. Does the invoice total match the line items? Does the cited clause exist in the source document? Deterministic checks are cheap, and they do not have good days and bad days.
Review it. Whatever passes the first three layers and leaves the company still crosses a human desk, until the error rate has earned autonomy. We wrote about that trust curve in the agents essay. It applies double here.
Notice what this stack does to the hallucination problem: it stops being a model property and becomes a system property. The model still interpolates. The system no longer lets an interpolation reach an invoice.
The results from our projects, stated honestly: grounded retrieval alone removes the large majority of fabrications. Constraints and verification catch most of the rest. The human layer exists for the remainder, which never quite reaches zero. The same is true of people.
Two failure modes still deserve respect. The model can misread a real document, which grounding does not prevent. And retrieval can fetch the wrong document, which produces a well-sourced wrong answer. Both are why verification checks against the system of record, not against the retrieved text.
If you take one thing from this: stop asking whether a model hallucinates. Ask what happens in your process when it does. A team with an answer to that question can put AI on serious work. A team without one is doing theatre, and the audience is your client.
Frequently asked questions
Why do AI language models hallucinate?
Because they are built to produce the most plausible continuation of a text, not to consult a store of facts. When the model lacks knowledge, there is no gap signal: a plausible answer is generated with the same fluency as a correct one. Hallucination is the normal mechanism operating outside its knowledge, not a malfunction.
Can AI hallucinations be completely eliminated?
No, and anyone promising zero should be asked how. They can be reduced sharply: grounding answers in retrieved documents removes most fabrications, structured outputs and deterministic checks catch most of the rest, and human review covers the remainder. The goal is a system where a hallucination cannot reach anything that matters unchecked.
What does grounding or RAG mean in practice?
The system first retrieves the relevant documents from your own sources, then instructs the model to answer only from them, with citations. Crucially, "not found in the documents" becomes a valid and tested answer. Done properly, the model shifts from remembering the world to reading your records.
How do you catch hallucinations in a production system?
Mostly with ordinary engineering: outputs constrained to structured fields, deterministic code that checks every verifiable claim against the system of record, and human approval on anything outbound until the measured error rate justifies autonomy. Confidence scores are not a control; checks are.
Are newer AI models better at not making things up?
Meaningfully better, yes: they decline to answer more often and fabricate less. But improvement is not a mechanism, and no model knows your contracts. The containment stack stays worthwhile regardless of which model is inside it, which is also what makes the model swappable.
