AI Hallucinates. We All Know That. It Could Be Our Fault.
The wrong answer is only the beginning of the story
Ask a general AI assistant for a product specification, and it may give you a confident number that turns out to belong to another model. Most people recognize that as a hallucination. They have seen the same thing happen with dates, quotations, and books that do not exist.
The chat window makes the mistake easy to spot. You asked a question, read the response, and found a false detail. Put AI inside a business application, and the same kind of detail can become a proposed product attribute, a line in a customer email, or a recommendation passed from one agent to another. It may arrive in a perfectly formatted field.
When we talk about hallucinations, we tend to focus on what the model did after it received the question. There is another question for anyone designing an AI solution: What was the system built to know, where was it allowed to get that information, and what could it do when the information was missing?
Errors will still happen, but those choices can reduce the opportunities for an unsupported statement to appear and limit how far it travels.
The metaphor only takes us so far
“Hallucination” is a useful warning. It describes the unsettling experience of reading something false that sounds entirely plausible. The word becomes less useful when it makes the mistake seem mysterious, as though the AI suddenly imagined a detail no one could have anticipated.
A language model has no human perception to mislead it. It generates an answer using the patterns it has learned and whatever context the application provides. We discussed that capability in our earlier Perspective on next-word prediction.
Why does the answer sometimes sound so sure when the information is missing? A 2025 paper on why language models hallucinate describes how models can produce plausible guesses when uncertain and how some evaluations reward guessing over acknowledging uncertainty. That tendency is worth accounting for when we plan what an application will ask the model to do.
A general model's training is outside the scope of an individual application. The application does determine how much the model is asked to guess. Planning begins there.
Decide what the AI is actually responsible for
Imagine a company wants an AI assistant to help prepare product comparisons. “Compare these products” sounds like one task, but it contains several different kinds of work. Someone has to retrieve the correct product records. Someone has to determine which attributes are comparable. The final explanation needs to be understandable to a customer.
At Impartium, we begin by separating the parts that require judgment from the parts the company already knows. The application can retrieve established facts from the systems responsible for them. The model can explain how two products differ using those facts, without being invited to improvise a specification from similar items.
Then we can give the AI a narrower, useful job: explain meaningful differences using the retrieved information, perhaps adapting the language to the customer's question. It still has room to interpret and write. The boundaries make clear which details it must carry through accurately.
This is a business decision before it is a prompt-writing exercise. A team has to decide which information is approved, which comparisons require interpretation, and whether a missing specification should prevent a particular claim. The model cannot invent that policy for the company.
Start with the source that owns the fact
For a known product attribute, the first step is to look for an approved value in the system that manages it. If the product information system holds current dimensions and finishes by SKU, the application should retrieve those values directly. If inventory lives in the ERP, it should ask the ERP. There is little reason to have a model search brochures for a fact the business already maintains as structured data.
Documents still have an important role. An installation guide may contain instructions that have no equivalent database field. A new supplier product may arrive with a specification sheet before its attributes have been entered into the product system. For those questions, the agent needs the appropriate current document and enough context to tell whether a statement applies to the requested product or to a broader collection.
That order matters. An old brochure or a draft sales presentation should not quietly override an approved structured value. Where the structured source has no answer, the application can use an approved document if it supports the detail. If neither provides it, the gap should remain visible. A related paragraph or a plausible value from a nearby product does not fill it.
The agent also needs to carry that distinction into its output: which value came from an approved record, which statement came from a document, and which part was an interpretation. A source link helps when someone can open it and find support for the specific statement.
Simply telling a model, “Do not hallucinate,” does not work. It may help a little, but the instruction assumes the model can recognize a false detail while generating it. Often it cannot. It can produce that detail with the same fluency as a correct one, and the mistake may become clear only when the output is checked against a trusted source. Giving the agent clear direction, appropriate sources, and a way to leave an answer unresolved makes unsupported claims less likely and easier to catch. That works a lot better than telling it, “Don’t do the thing you don’t realize you’re doing.”
If you are planning an AI assistant or agent workflow, we can help define what it should produce, which sources it can use, and where it must stop or ask.
Plan a more reliable AI workflow →
Design the handoffs between agents
Adding agents can divide a complicated assignment into manageable parts. It can also give one unsupported detail several chances to become established.
Suppose one agent assembles product facts and a second writes a comparison. If the first passes along “Available finish: brushed nickel” without indicating that it was inferred from another model, the writer has little reason to question it. A third agent checking that the comparison is complete may approve the result because every required field is filled. Three agents have now handled the statement; none has verified it against the right product record.
Each handoff needs a clear definition of what information travels with it. For a factual field, that may include the product identifier, the source used, and whether the value was retrieved, calculated, or proposed by the model. The receiving step can then apply an appropriate check. A required field can be validated against approved data; an interpretation can be marked for review. Asking another model whether the answer “looks right” does not give it an independent source of truth.
Sometimes a second agent is useful. Sometimes a lookup and a simple rule will do the job more reliably. The number of agents is a design choice, not a measure of how advanced the solution is.
When no one reads the output
Many AI applications never answer a customer or show an employee a draft. For a new item that has not yet been entered into the product system, an agent might read a supplier's product sheet, prepare its attributes, and send them to that system for distribution to an ecommerce site or dealer feed. The purpose of that automation may be to process hundreds of routine updates without someone approving each one.
Suppose the sheet lists a product's width and height but leaves out its depth. If the agent fills that field with a plausible measurement from a nearby model, the receiving system may accept it as an ordinary number. No one has to see the false value for it to reach a product page.
Putting a person in every step would defeat some useful automations. The decision should depend on what the field means and what a mistake could affect. We might copy documented measurements directly, check values against the source, and leave unsupported fields empty. A questionable update can be held for review while routine, well-supported records continue. Even when individual records move without human approval, the business needs a way to examine results and catch patterns of error.
Make room for an honest answer
An AI assistant that is expected to answer every question will eventually face one its available information cannot settle. At that point, “I cannot confirm that for this model” may be the best answer it can give. It could ask for a model number or direct the question to someone who has the information.
We need to design that path before the system handles real work. Otherwise a blank field or an ambiguous document can become pressure to produce a polished guess. In a customer conversation, the assistant might give the verified part of an answer and explain what remains uncertain. In a background workflow, an empty field, a status for review, or a held record can express the same limit without putting a person in front of every result.
When an approved source clearly provides the value, the system should be able to use it. The important choice is what happens when that support is absent, especially if the next system will treat the output as a fact.
Test the claims people will rely on
Before launch, the proposed system should face examples drawn from the work it will actually receive. Include two products with similar names but different specifications. Try a discontinued item whose old brochure is still available, a product sheet with a missing measurement, and a question whose answer spans two documents.
The review should look beyond whether a response reads well or a record meets its required format. Did it match the right product? Did important values come from appropriate sources? Did it leave an unsupported detail out, or fill the gap with something plausible? If an agent passed information to another or wrote to a business system, can we see what it sent and where a mistake began?
Those examples tell us which design choice to improve. Perhaps the product match is weak. Perhaps the source collection includes outdated material. Perhaps the assistant needs a clearer boundary around what it can infer. Testing turns “the AI hallucinated” into a problem the team can address.
A better question for the project team
We will probably keep using the word “hallucination.” It gives people a quick way to report an answer that sounds credible and proves false. The word alone offers little guidance for building the next version.
When a company brings the question “How do we stop it hallucinating?” to Impartium, we start with the work the AI is supposed to do. Which facts already exist in company systems? Which sources are approved? Where must the model interpret information, and what should happen when the answer is unavailable? Those answers shape an agent that has less reason to guess and a better chance of showing its work.
If you have an AI use case in mind, we can help map the decisions, information, and checks it needs before an agent starts handling your business data.
Design the AI approach →
