AI Control Means Three Things: Explainable, Verifiable, Correctable
Before an AI system touches a credit decision, a claim or a job application, I want three answers: why this output, is it right, and how do we take it back. What that looks like in an architecture, and where teams cut corners.
I say this deliberately simply. Every AI system that makes or prepares decisions about people in a regulated company needs three properties. It must be explainable, verifiable and correctable. If one is missing, nobody can take responsibility for the system, and it does not belong in production, whether at a bank, an insurer or a public authority. That, for me, is the whole meaning of AI control.
Today I want to show what the three words mean concretely: what gets stored, which interfaces exist, who owns them. And the shortcuts teams take, because that is exactly where the audit later fails.
Most teams pick one of the three. But an explanation you cannot verify is a story. Verification without correction is a report proving you knew and did nothing. Correction without explanation is trial and error on people’s lives. They only work together.
Explainable: someone can say why this output
Explainable does not mean the model is interpretable all the way down. It means a named person can answer, for one specific decision, which inputs went in, which version of which model produced the output, which rules ran on top, and whether a human looked at it.
In the architecture this is a decision record. Every output gets an ID. Behind it we store the pseudonymised input snapshot (the mapping lives elsewhere), model identifier and version from the register, the raw output, the post-processing rules that fired, and any human who touched it. The record is written before the output leaves the system, never reconstructed afterwards. The interface is a lookup by decision ID, readable by a case handler or auditor without a data scientist. The owner is the business unit that owns the decision, not the ML team that owns the model.
The typical shortcut: a team bolts on a feature-attribution method and calls the system explainable. At audit time the question is not which features mattered on average, but why this applicant was rejected on this day. If the model has been retrained twice since and nobody stored the version, the chart answers a question nobody asked. Second shortcut: asking the language model to explain itself. A generated rationale is another output of the same system with the same failure modes: a useful draft, not evidence.
Verifiable: someone can check the output against ground truth or policy
Verifiable means a second party can check whether the output was right, either against ground truth (did the loan default, was the claim fraudulent) or against policy (does the rejection respect the four-eyes rule and the fairness constraints). The check has to work on a sample and, if the supervisor insists, in full.
This needs three things. The decision record carries the policy version in force at that moment. A ground-truth feed links the outcome of the case back to the decision ID. And a sampling interface lets the second line of defence pull a random or stratified sample from a period, re-run the policy checks and compare against outcomes. The thresholds live here, per use case. In our deployments a shift in the approval rate for a protected group beyond an agreed band pages the model owner the same day and freezes new deployments. A rise in overturned decisions above its threshold switches the system from automatic to human-confirmed mode until someone has looked. Risk or compliance owns sampling and thresholds; the ML team can propose, not decide.
The shortcut: validation only at go-live. A pre-launch test set says nothing about the model six months later. The second: averages without drill-down to individual decisions. When the supervisor asks for fifty specific cases and you can only show a dashboard, you have monitoring, not verification.
Correctable: a wrong output can be caught, reverted and the system adjusted
Correctable means three capabilities at three time scales. In the moment, a human can override a single output, with a recorded reason. Within days, every affected decision can be identified and reopened. Within weeks, the system can be adjusted without rebuilding the pipeline or losing history.
The override is a first-class event, not a note in a ticket system. It writes to the same decision record and feeds back into verification, because the override rate is one of the strongest signals you have. For reopening, the decision ID is linked to the model version, so affected decisions can be listed, not estimated. For adjustment, rules and thresholds live outside the model as configuration under version control, so a changed threshold is a reviewed change with a date and an author, not a retraining. And a kill switch: setting the register entry to blocked stops new cases from reaching the system within minutes, with the manual fallback already defined. The business owns override and fallback, the platform team the kill switch, the model owner the adjustment.
The shortcut: “we will retrain if problems come up”. Retraining takes weeks, changes everything at once and makes it impossible to say what fixed the problem. The other: a rollback for the model but not the data. If downstream systems already processed the outputs, you reverted nothing.
The three properties across technology, law and society
I describe AI governance as three layers: technology, law, society. The three properties are the technical layer’s answer to the two above it.
Property
Technology: what is built
Law: what the duty is
Society: what the person gets
Explainable
Decision record with model version, inputs, rules, human touchpoints
Logging, documentation, transparency
A reason they can read and contest
Verifiable
Ground-truth feed, sampling interface, thresholds owned by the second line
Accuracy and bias monitoring, model validation, supervisory access
A neutral check that the reason was true
Correctable
Override event, reopen by version, configuration under version control, kill switch
Human oversight, intervention, shutdown
A remedy, not just an apology
The EU AI Act, in force since August 2024, phases in its obligations over the coming years, and for high-risk systems it asks for exactly this: logging, human oversight, accuracy monitoring and the ability to intervene. Banking supervision has asked models the same for years. For the person affected, only the third column matters: without a remedy, reason and check are decoration.
Open questions
Oversight at scale. At a hundred thousand decisions a day, sampling catches aggregate drift but rarely the one wrong decision that ruins one person. Which sample designs find rare individual harm, at what review cost?
Robustness of explanations. The decision record says what went in and came out, not whether a small input change would have flipped the result. How stable must an explanation be before a case handler may rely on it?
Faithful versus plausible. Language models produce a convincing reason for anything. How do we tell a faithful account from a plausible story, and should faithfulness be a deployment requirement?
Honesty under correction. If a system is tuned on override data, does it learn to avoid overrides rather than to be right? How would we detect that shift early?
Which of the three is weakest in your systems today? In my experience almost always correctability, because nobody gets credit for it until the day it is needed. If you can do one thing this quarter, build the decision record with the model version in it. Everything else attaches to it.
Frequently asked questions
Does explainable mean we can only use simple models?
No. Explainability here is a property of the system, not the model. A complex model with a complete decision record and a registered version beats a simple model whose outputs nobody stored.
Who should own the three properties?
Split it: the business unit for decision record, override and fallback; risk or compliance for sampling and thresholds; the platform team for register and kill switch; the model owner for adjustment. One team owning everything is itself a control failure.
Does the EU AI Act require all three?
For high-risk systems it requires logging, human oversight, accuracy and robustness monitoring and the ability to intervene: together, the three properties. Building them now is cheaper than retrofitting under supervisory pressure.
AI Control Means Three Things: Explainable, Verifiable, Correctable
Before an AI system touches a credit decision, a claim or a job application, I want three answers: why this output, is it right, and how do we take it back. What that looks like in an architecture, and where teams cut corners.
I say this deliberately simply. Every AI system that makes or prepares decisions about people in a regulated company needs three properties. It must be explainable, verifiable and correctable. If one is missing, nobody can take responsibility for the system, and it does not belong in production, whether at a bank, an insurer or a public authority. That, for me, is the whole meaning of AI control.
Today I want to show what the three words mean concretely: what gets stored, which interfaces exist, who owns them. And the shortcuts teams take, because that is exactly where the audit later fails.
Most teams pick one of the three. But an explanation you cannot verify is a story. Verification without correction is a report proving you knew and did nothing. Correction without explanation is trial and error on people’s lives. They only work together.
Explainable: someone can say why this output
Explainable does not mean the model is interpretable all the way down. It means a named person can answer, for one specific decision, which inputs went in, which version of which model produced the output, which rules ran on top, and whether a human looked at it.
In the architecture this is a decision record. Every output gets an ID. Behind it we store the pseudonymised input snapshot (the mapping lives elsewhere), model identifier and version from the register, the raw output, the post-processing rules that fired, and any human who touched it. The record is written before the output leaves the system, never reconstructed afterwards. The interface is a lookup by decision ID, readable by a case handler or auditor without a data scientist. The owner is the business unit that owns the decision, not the ML team that owns the model.
The typical shortcut: a team bolts on a feature-attribution method and calls the system explainable. At audit time the question is not which features mattered on average, but why this applicant was rejected on this day. If the model has been retrained twice since and nobody stored the version, the chart answers a question nobody asked. Second shortcut: asking the language model to explain itself. A generated rationale is another output of the same system with the same failure modes: a useful draft, not evidence.
Verifiable: someone can check the output against ground truth or policy
Verifiable means a second party can check whether the output was right, either against ground truth (did the loan default, was the claim fraudulent) or against policy (does the rejection respect the four-eyes rule and the fairness constraints). The check has to work on a sample and, if the supervisor insists, in full.
This needs three things. The decision record carries the policy version in force at that moment. A ground-truth feed links the outcome of the case back to the decision ID. And a sampling interface lets the second line of defence pull a random or stratified sample from a period, re-run the policy checks and compare against outcomes. The thresholds live here, per use case. In our deployments a shift in the approval rate for a protected group beyond an agreed band pages the model owner the same day and freezes new deployments. A rise in overturned decisions above its threshold switches the system from automatic to human-confirmed mode until someone has looked. Risk or compliance owns sampling and thresholds; the ML team can propose, not decide.
The shortcut: validation only at go-live. A pre-launch test set says nothing about the model six months later. The second: averages without drill-down to individual decisions. When the supervisor asks for fifty specific cases and you can only show a dashboard, you have monitoring, not verification.
Correctable: a wrong output can be caught, reverted and the system adjusted
Correctable means three capabilities at three time scales. In the moment, a human can override a single output, with a recorded reason. Within days, every affected decision can be identified and reopened. Within weeks, the system can be adjusted without rebuilding the pipeline or losing history.
The override is a first-class event, not a note in a ticket system. It writes to the same decision record and feeds back into verification, because the override rate is one of the strongest signals you have. For reopening, the decision ID is linked to the model version, so affected decisions can be listed, not estimated. For adjustment, rules and thresholds live outside the model as configuration under version control, so a changed threshold is a reviewed change with a date and an author, not a retraining. And a kill switch: setting the register entry to blocked stops new cases from reaching the system within minutes, with the manual fallback already defined. The business owns override and fallback, the platform team the kill switch, the model owner the adjustment.
The shortcut: “we will retrain if problems come up”. Retraining takes weeks, changes everything at once and makes it impossible to say what fixed the problem. The other: a rollback for the model but not the data. If downstream systems already processed the outputs, you reverted nothing.
The three properties across technology, law and society
I describe AI governance as three layers: technology, law, society. The three properties are the technical layer’s answer to the two above it.
The EU AI Act, in force since August 2024, phases in its obligations over the coming years, and for high-risk systems it asks for exactly this: logging, human oversight, accuracy monitoring and the ability to intervene. Banking supervision has asked models the same for years. For the person affected, only the third column matters: without a remedy, reason and check are decoration.
Open questions
Which of the three is weakest in your systems today? In my experience almost always correctability, because nobody gets credit for it until the day it is needed. If you can do one thing this quarter, build the decision record with the model version in it. Everything else attaches to it.
Frequently asked questions
Does explainable mean we can only use simple models?
No. Explainability here is a property of the system, not the model. A complex model with a complete decision record and a registered version beats a simple model whose outputs nobody stored.
Who should own the three properties?
Split it: the business unit for decision record, override and fallback; risk or compliance for sampling and thresholds; the platform team for register and kill switch; the model owner for adjustment. One team owning everything is itself a control failure.
Does the EU AI Act require all three?
For high-risk systems it requires logging, human oversight, accuracy and robustness monitoring and the ability to intervene: together, the three properties. Building them now is cheaper than retrofitting under supervisory pressure.
Related: AI Governance · EU AI Act · AI Consultant
Recent Posts
Recent Comments