Why I Do Not Trust a Confident Model: Who Checks the Numbers?
A fluent answer is not a correct answer. Why language models have no built-in sense of doubt, what calibration means for decisions about people, and the one rule I ask every client to adopt: no AI-generated figure enters a decision document without its source and its checker.
I see the same scene in almost every company I visit. Someone pastes a generated slide into a board deck. The slide has a market size, a default rate, a headcount forecast. The numbers look plausible, the formatting is perfect. And nobody at the table asks the only question that matters: who verified this? For me that question is the core of AI control. Not the model, not the prompt, but the moment a figure leaves the system and starts to shape a decision.
Fluency is not accuracy
A large language model is trained to produce the next token that fits best. Fits means statistically plausible given everything the model has seen, not true. That is why a hallucination does not look like a hallucination. It looks like the correct answer, with the same tone and the same confidence. We have spent a lifetime learning to read fluent, confident speech as competence. With a language model that signal is gone.
This is not a bug that the next version fixes. Newer models invent fewer facts in tests, but a lower rate is still a rate, and nobody in the boardroom knows which of the twenty figures on the slide is the wrong one. The better the models get at sounding right, the more dangerous the remaining errors become, because fewer people check.
There is no built-in “I am not sure”
People ask me why the model does not simply say when it does not know. The honest answer: it has no place where knowing lives. The phrase “I am not sure” is itself just text the model has learned to produce in certain contexts. If you want a usable uncertainty signal, you have to build it around the model. In our projects we work with four sources:
Self-consistency. Ask the same question several times with varied phrasing or sampling. If the figure changes between runs, you do not have a figure, you have a guess. We log the spread, not only the answer.
Retrieval support. Every factual claim has to point to a passage in the company’s own documents. No passage, no claim. The citation is checked mechanically: does the cited passage actually contain that number?
Agreement between models. Two models arriving at the same number independently are weak evidence. Two that disagree are strong evidence that a human needs to look.
Abstention. The system is allowed, and in defined cases obliged, to answer “not determinable from the available data”. A tool that always answers is regularly wrong.
None of these signals is perfect. Together they turn a bare sentence into something a reviewer can judge.
What calibration means for decisions about people
Calibration is a simple idea with hard consequences. If a system says it is 90 percent confident, it should be right about nine times out of ten. Measured over many answers, most systems are not, and they are most overconfident where the data is thin. For a loan decision, an insurance claim or an application screening it is a governance problem. A badly calibrated score means the human in the loop is looking at the wrong cases: the system flags fifty decisions as uncertain and waves through the five that were actually wrong.
This is why we measure calibration before a model may influence a decision about a person, and again in production, because confidence scores drift when the data drifts. The EU AI Act, in force since August 2024, phases in its obligations over the coming years, and for high-risk systems it demands exactly this kind of ongoing accuracy and robustness monitoring. Without it the human in the loop is a decoration.
Verification belongs before the number leaves the system
The usual pattern is: generate, paste, present, and if someone objects in the meeting, check afterwards. That is the wrong order. By the time a number has been on a slide in front of a board, it has done its work. People have anchored on it. In the governance layer we are building at SIFAMO, verification is a step in the pipeline, not a step in the calendar:
Every generated figure is logged with prompt, model version, retrieved sources and the uncertainty signals above.
A figure without a resolvable source is marked unverified and cannot be exported into a document template. Technically blocked, not just discouraged.
Figures above a defined materiality threshold, for example anything that goes into a regulatory report or a decision about a customer, are routed to a named checker. Name and timestamp go into the log.
If the self-consistency spread or the disagreement between models exceeds a threshold, the figure is held back and the owner of the use case is paged, not the person who typed the prompt.
This is the same logic as our rule for models, no registration, no deployment. A number that nobody can trace back has no business in a decision. Where the check happens decides whether it is AI governance or damage control.
The simple organisational rule for AI control
Every AI-generated figure carries its source and its checker, or it does not enter a decision document. That is the whole rule, and it survives every model change. It changes behaviour fast, because pasting without a source now produces a visible gap. And it makes the responsibility question answerable. When the supervisor asks, you do not say the AI said so. You say who checked it, against what, and when.
What this means for internal tooling
Three consequences for every internal tool we build or review, whether it drafts reports, answers questions over documents or summarises cases:
Generated content is labelled as generated. In the document, in the API response, in the database field. Not a footnote in the user guide but a flag on the data itself.
Every published number has a human owner. A person, not a team mailbox. If the owner leaves, the number is re-owned or withdrawn.
Label and owner travel with the data. When a figure is copied from the tool into a presentation, the metadata comes with it. If a tool cannot do that yet, it does not feed decision documents yet.
It is the same thinking as the license plate for models that I keep returning to: you should see at a glance where a thing came from and who answers for it.
Open questions
Oversight at scale. A named checker works for fifty figures a week. Who checks when a system produces fifty thousand, and how do you sample so that the dangerous ones are in the sample?
Robustness of the signals. Self-consistency and model agreement can be fooled by a model that is confidently and consistently wrong. How well do these proxies hold up under distribution shift?
Interpretability. Can a model’s internal representations yield a better uncertainty signal than its own words, and would an auditor accept such a signal as evidence?
Honesty. A model trained to be helpful tends to under-report doubt. How do you train and verify a model that prefers “I do not know” over a plausible number when the data is thin?
Frequently asked questions
What is a hallucination in a language model?
A fluent, confident statement that is not supported by the training data or the provided documents. It arises because the model optimises for plausible text, not for truth. It cannot be recognised from the wording alone, only through external checks such as retrieval support or repeated sampling.
What does calibration mean for an AI system?
Calibration describes whether the confidence a system reports matches how often it is actually right. A well calibrated system that says 80 percent is right about four times out of five.
Who should check AI-generated figures in a company?
A named person who owns the figure, not the person who typed the prompt. The check belongs in the pipeline, before export into a document, with source, model version and checker in the log. No source and no checker, no entry into a decision document.
Why I Do Not Trust a Confident Model: Who Checks the Numbers?
A fluent answer is not a correct answer. Why language models have no built-in sense of doubt, what calibration means for decisions about people, and the one rule I ask every client to adopt: no AI-generated figure enters a decision document without its source and its checker.
I see the same scene in almost every company I visit. Someone pastes a generated slide into a board deck. The slide has a market size, a default rate, a headcount forecast. The numbers look plausible, the formatting is perfect. And nobody at the table asks the only question that matters: who verified this? For me that question is the core of AI control. Not the model, not the prompt, but the moment a figure leaves the system and starts to shape a decision.
Fluency is not accuracy
A large language model is trained to produce the next token that fits best. Fits means statistically plausible given everything the model has seen, not true. That is why a hallucination does not look like a hallucination. It looks like the correct answer, with the same tone and the same confidence. We have spent a lifetime learning to read fluent, confident speech as competence. With a language model that signal is gone.
This is not a bug that the next version fixes. Newer models invent fewer facts in tests, but a lower rate is still a rate, and nobody in the boardroom knows which of the twenty figures on the slide is the wrong one. The better the models get at sounding right, the more dangerous the remaining errors become, because fewer people check.
There is no built-in “I am not sure”
People ask me why the model does not simply say when it does not know. The honest answer: it has no place where knowing lives. The phrase “I am not sure” is itself just text the model has learned to produce in certain contexts. If you want a usable uncertainty signal, you have to build it around the model. In our projects we work with four sources:
None of these signals is perfect. Together they turn a bare sentence into something a reviewer can judge.
What calibration means for decisions about people
Calibration is a simple idea with hard consequences. If a system says it is 90 percent confident, it should be right about nine times out of ten. Measured over many answers, most systems are not, and they are most overconfident where the data is thin. For a loan decision, an insurance claim or an application screening it is a governance problem. A badly calibrated score means the human in the loop is looking at the wrong cases: the system flags fifty decisions as uncertain and waves through the five that were actually wrong.
This is why we measure calibration before a model may influence a decision about a person, and again in production, because confidence scores drift when the data drifts. The EU AI Act, in force since August 2024, phases in its obligations over the coming years, and for high-risk systems it demands exactly this kind of ongoing accuracy and robustness monitoring. Without it the human in the loop is a decoration.
Verification belongs before the number leaves the system
The usual pattern is: generate, paste, present, and if someone objects in the meeting, check afterwards. That is the wrong order. By the time a number has been on a slide in front of a board, it has done its work. People have anchored on it. In the governance layer we are building at SIFAMO, verification is a step in the pipeline, not a step in the calendar:
This is the same logic as our rule for models, no registration, no deployment. A number that nobody can trace back has no business in a decision. Where the check happens decides whether it is AI governance or damage control.
The simple organisational rule for AI control
Every AI-generated figure carries its source and its checker, or it does not enter a decision document. That is the whole rule, and it survives every model change. It changes behaviour fast, because pasting without a source now produces a visible gap. And it makes the responsibility question answerable. When the supervisor asks, you do not say the AI said so. You say who checked it, against what, and when.
What this means for internal tooling
Three consequences for every internal tool we build or review, whether it drafts reports, answers questions over documents or summarises cases:
It is the same thinking as the license plate for models that I keep returning to: you should see at a glance where a thing came from and who answers for it.
Open questions
Frequently asked questions
What is a hallucination in a language model?
A fluent, confident statement that is not supported by the training data or the provided documents. It arises because the model optimises for plausible text, not for truth. It cannot be recognised from the wording alone, only through external checks such as retrieval support or repeated sampling.
What does calibration mean for an AI system?
Calibration describes whether the confidence a system reports matches how often it is actually right. A well calibrated system that says 80 percent is right about four times out of five.
Who should check AI-generated figures in a company?
A named person who owns the figure, not the person who typed the prompt. The check belongs in the pipeline, before export into a document, with source, model version and checker in the log. No source and no checker, no entry into a decision document.
Related: AI Governance · EU AI Act · AI Consultant
Recent Posts
Recent Comments