--- title: "AI hallucinations: how to avoid them | JustAutomate" description: "Why a model invents facts, how to ground an answer in company data, how to design and test a refusal, and what to do after a wrong answer." url: "https://justautomate.ai/en/guides/ai-hallucinations-how-to-avoid/" lang: en --- AI agents and language models # AI hallucinations: how do you avoid them? ground the answer in a source, then test the refusal A hallucination is not a defect you remove with a patch. It is an operational risk: the model predicts the next piece of text instead of checking a fact, so it sounds exactly as confident when it knows and when it is guessing. What brings it down: answers built from your company data with the source named, a refusal you design on purpose, and a test you can run in a day. 8 minutes read - 5 to 10 secondshow long an AI agent takes to answer from your company data, with the source named so anyone can check it - 20 questionsthe control set before go-live: 10 that have an answer in the data and 10 that have none, to see whether the agent refuses - EUR 0.005 to 0.07the cost of one query to the model, paid to the provider directly; checking the answer with a second model doubles that line - 30 and 90 dayswhen we re-run the same question set, because a change of model or of source changes how the agent behaves Short answer ## How do you avoid AI hallucinations in a company? You cannot get them to zero, you can manage them. The model works out which piece of text is most likely after what it has already written; there is no fact database inside it and no signal that says „I do not know this”. Treat it like any other error risk in a process: you limit it technically, and you write into the procedure who checks the result. Countermeasure one: **the agent answers from your company data and shows where the answer came from.** Instead of asking the model for general knowledge, you pull the CRM record, the invoice or the fragment of the document, and only then does the sentence get written. An answer with no source counts as unverified. Countermeasure two: the „I do not know” behaviour is set on purpose and tested. Before go-live you ask the agent questions the data cannot answer, and you count the refusals. Then you split the use cases: internal lookup runs unsupervised, an offer to a customer has a person in front of it. ## Why does a language model invent facts? The model predicts text. A true sentence and an invented one come out of the same mechanism, so both read just as smoothly and both get the same confident tone. There is no truth database inside, and no gauge of its own certainty. The narrower the domain and the thinner the training data, the more likely the model fills the gap with something plausible: a clause number, a product name, a figure that fits the sentence. How companies use these models day to day is covered in our guide to [LLMs in business](https://justautomate.ai/en/guides/llm-in-business/). - The model predicts text, it does not verify a fact: there is nothing to query and nothing to contradict - No confidence signal: the refusal has to be imposed by instruction and confirmed by a test - Gaps in the training data get filled with content that is plausible, not content that is true - More randomness in word choice (the temperature parameter) means more invention, so we set it close to zero **Note.** A hallucination does not show up in an error log. It looks like a correct answer, so what catches it is the process and the source named next to the number, not monitoring. ## How does an answer from company data differ from an answer from the model? This split is what decides the risk. „How many invoices did we issue in August” has an answer in your system: the agent reads it and writes it down. „What is the limitation period on this claim” has no answer in your data, so the model reaches for general knowledge, and that is where the trouble starts. This is why an agent on company data is safer than a chat window in a browser: it answers from connected systems, it names the source, and it says plainly when the answer is not there. How that works in practice is on the [BOND business AI agent](https://justautomate.ai/en/services/bond-business-ai-agent/) page. | Type of question | Where the answer comes from | What brings the risk down | | ---------------------------------------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------- | | Numbers from a system: invoices, orders, CRM opportunities | a query to the system at the moment of asking | the record and the filter named, so anyone can re-run it | | Content of a document: contract, offer, procedure | the fragment of the file handed to the model with the question | a quote and the file name next to the answer | | Summarising or rewriting a text | material you handed the model yourself | comparison with the original, the lowest risk in this table | | General knowledge: law, tax, the market | what the model picked up in training | put the topic outside the agent's scope, or send it to a specialist | | A question about data nobody connected | no source at all | a refusal: I do not have this in the connected sources | Do this split before you build, on real questions from the team. Most of them usually have an answer in a system, and the risk sits in the last two rows. ## How do you ground an answer in a source? Grounding means this: the model answers from material delivered in the same request, not from memory. First the lookup of a record or a document fragment, then an answer built only from that, then the source named at the end. That last part matters more than the retrieval technique itself. An answer carrying the system name, the document number and the time it was read takes fifteen seconds to verify. Without it, verification starts from scratch, so nobody does it. - Lookup before answer: the agent queries the system first, then writes the sentence - A narrowing instruction: answer only from the material below, do not top it up from general knowledge - Every number labelled: system, table, document number, time of reading - A quote instead of a paraphrase wherever the exact wording carries weight - Freshness: the agent states when it read the data, because yesterday's stock level is also a false answer **Note.** A knowledge base built from outdated files produces answers that are correctly grounded and untrue at the same time. Before you connect a drive, settle which version of a procedure is in force and who keeps it current. ## When should an AI agent say it does not know? Whenever the connected sources hold no answer. It sounds obvious, and it is the setting people skip most often: the model would rather answer something than admit a gap. A refusal is behaviour you design and confirm with a test, not a side effect of a good prompt. The refusal has to be useful, not a wall. A good one reads like this: I do not have this in the connected sources, margins sit with finance, I can show you revenue for the same period. - System instruction: no data in the source means a refusal, not an attempt to assemble an answer - The refusal names what is missing and points to whoever holds that data - Refusals go into a log: that is a ready-made list of missing integrations for next quarter - A question outside the asker's permissions also ends in a refusal, not in an answer from somebody else's data **Note.** An over-tight refusal has its own cost: an agent that turns people away on every second question stops being used within two weeks. You set the level during the pilot. ## Where can the automation be trusted, and where do you need a person? The line is drawn by the cost of a wrong answer, not by the difficulty of the task. A question where a mistake costs a minute of confusion can run unsupervised. A sentence that goes out to a customer or up to the board has a person approving it. Part of the line comes from regulation. The AI Act, regulation 2024/1689, requires in article 50 that a person knows they are dealing with an AI system, and the areas listed in annex III, recruitment among them, count as high-risk systems. The other half of that decision is access scope and a log of calls: without them you cannot establish where the agent got the data. We cover that in the guide to [AI security in business](https://justautomate.ai/en/guides/ai-security-in-business/). | Use case | Who approves the answer | Why it works that way | | -------------------------------------------------- | --------------------------------------------------- | ------------------------------------------------------------- | | Internal question about data in a system | nobody, the named source is enough | the error shows immediately and costs a minute | | Draft of an offer or an email to a customer | the salesperson, always before sending | price and scope leave the company | | A figure in a board report | the person who owns the report | a number with nothing behind it goes straight into a decision | | Legal, tax or HR matter | a specialist; the agent only retrieves the material | the cost of a mistake is out of proportion | | An operation on data: invoice, shipment, CRM entry | a person confirms the action | a wrong answer turns into a wrong event | The same split shows where the model is not needed at all: some questions about the state of a system are answered by a plain report, with no risk of invention. ## How do you check whether your own AI agent hallucinates? With a test, not an impression. The question set is written by the team that will use the agent, not by the people who built it. Half the questions have an answer in the data, half have none at all, and that second half is the real test. 1. ### A set of twenty questions Write down 20 real questions from the team: 10 with an answer in the connected data and 10 without. Into the second ten put questions about data you did not connect and about periods before go-live. 2. ### Reference answers For the first ten, write the correct answer from the system together with the place it is read from. Without that you cannot settle whether the agent got it wrong or the reviewer did. 3. ### Run it and score it Ask all 20 questions and score the answers in three categories: matches the source, refusal, invented. An invented answer in the second ten is a critical failure, because that is where the agent was supposed to refuse. 4. ### Repeat at 30 and 90 days You run the same set after a month and after a quarter, adding the questions that caused trouble. A change of model, of instruction or of source is its own reason to repeat it. ## What do you do after a wrong answer? First establish whether it was a hallucination or the data. An agent that correctly read an outdated spreadsheet invented nothing: the problem sits in the source. That distinction decides whether you fix the instruction or tidy the files. Then write the case down before somebody retells it in the corridor. One wrong answer with no explanation destroys trust faster than a hundred correct ones build it. - Preserve the trail: question, answer, named source, timestamp - Settle the cause: missing data, the wrong source, or an instruction that was too broad - Fix the narrowest layer: usually a filter or a document, rarely the model itself - Add that question to the control set so the error does not come back after the next change - Tell the team what happened and what was fixed; silence costs more than the error did **Note.** If the wrong answer reached a customer or entered a report, the technical fix is half the job. The other half is a correction sent to whoever received it, with the date and with what was wrong. Common mistakes ## Five mistakes that leave the hallucination to surface in production - A pilot run only on questions that have an answer The agent looks flawless, because nobody checked how it behaves with no data behind the question. The first invented number lands on a real decision. - Answers with no source named Verifying one figure takes half an hour, so nobody does it. The error gets caught by whoever knows that figure by heart, usually in the board meeting. - The instruction „do not make things up” instead of grounding in data The instruction lowers the frequency, it does not change the mechanism: the model still predicts text. With no data pulled from a system, the risk stays where it was. - A knowledge base built from outdated files The answers are correctly grounded and untrue at once. The hardest case to catch, because the source is named and looks credible. - Human approval on everything, with no exceptions The control eats the gain from the automation, and after two weeks it turns into clicking „approve” without reading. Point the oversight where the error hurts. Checklist ## Checklist before an AI agent goes into production A no on any line does not stop the project, it goes on the list to close before go-live. - Team questions split into those with an answer in the data and those that need the model's general knowledge - The agent pulls data at the moment of the question instead of answering from the model's memory - Every answer names the system, document or filter, plus the time it was read - The refusal „I do not have this in the connected sources” is set and confirmed by a test - A set of 20 control questions, 10 of them with no answer in the data, with the result written down - A list of use cases where a person approves the answer - A procedure after an error: trail, cause, fix, message to the team - The repeat test at 30 and 90 days already in the calendar Sources ## Sources 1. [Regulation 2024/1689 (AI Act): article 50 on telling people they are dealing with an AI system, and annex III listing high-risk systems](https://eur-lex.europa.eu/eli/reg/2024/1689/oj) 2. [Regulation 2016/679 (GDPR): article 5(1)(d), the accuracy principle for personal data](https://eur-lex.europa.eu/eli/reg/2016/679/oj) 3. [Anthropic, maker of the Claude models: API documentation and usage terms](https://www.anthropic.com/) 4. [OpenAI: model documentation and call parameters, temperature among them](https://openai.com/) 5. [JustAutomate, BOND business AI agent: what an answer with a named source looks like](https://justautomate.ai/en/services/bond-business-ai-agent/) Workshop ## The line between an answer and a guess gets drawn before go-live On a free consultation of 30 minutes we take twenty real questions from your team and split them into the ones your data can answer and the ones where the agent has to refuse. If the data is not there, we say so on the call. The process workshop costs EUR 3,500 net per week and ends with a list of sources, a set of control questions and a split of what can run unsupervised. [Book a free consultation](https://justautomate.ai/en/contact/) ## Frequently asked questions Can AI hallucinations be eliminated completely? No. The model predicts text rather than verifying a fact, so the risk stays for good. You can bring it down to a level somebody accepts in writing: answers from company data, a named source, a designed refusal, and a person wherever the error hurts. What does grounding an answer in a source mean? It is a way of working where the agent first retrieves the right record or document fragment, and only then does the model build an answer out of it. It often goes by the name RAG. The second part is the one that counts: next to the answer stands the system, the document number and the time it was read. How do you make an AI agent say it does not know? With a system instruction that says it plainly: no data in the connected source means a refusal. Then you test it with ten questions that have no answer in the data. A good refusal names the missing source and points to whoever in the company holds that data. How do you check whether an AI agent hallucinates on your data? With a set of 20 real questions: 10 with an answer in the connected systems and 10 with none. For the first ten prepare reference answers. You score in three categories: matches the source, refusal, invented. You repeat the test at 30 and 90 days. Is it worth checking answers with a second model? It is worth it where a wrong answer is expensive, and it costs a second call, so EUR 0.005 to 0.07 per query paid to the model provider. The second model catches a contradiction against the source, but it gets things wrong too, so it does not replace a person. Which AI tasks need approval from a person? The ones where an error leaves the company or feeds a decision: an offer or an email to a customer, a figure in a board report, legal and HR matters, and any operation that changes data. An internal question about the state of a system can run unsupervised. What do you do when an AI agent gave a customer wrong information? Preserve the trail: question, answer, named source and time. Establish whether the model failed or the data was out of date, because each leads to a different fix. Send a correction to the recipient and add that question to the control set. ## Related - [BOND: business AI agent](https://justautomate.ai/en/services/bond-business-ai-agent/) - [AI security in business](https://justautomate.ai/en/guides/ai-security-in-business/) - [LLMs in business](https://justautomate.ai/en/guides/llm-in-business/) - [How to implement an AI agent](https://justautomate.ai/en/guides/how-to-implement-ai-agent/) - [How to build an AI knowledge base](https://justautomate.ai/en/guides/how-to-build-ai-knowledge-base/) - [Process workshop](https://justautomate.ai/en/services/process-workshop/) --- **JustAutomate** · contact: contact@justautomate.ai · form: https://justautomate.ai/en/contact/