Keeping AI Controllable in Production: The Governance Layer We Built for Banks and Insurers
Umair Zaffar November 18, 2026 0 Comments

Every production AI system in a bank or an insurer makes or prepares decisions about people. Most of them run without a register, without boundaries and without a log a supervisor could read. This is the architecture we have been building at SIFAMO since 2024 to change that: a governance layer that sits between the business and the models. Here is how it works, what it refuses by construction, what we got wrong on the way, and the questions we cannot answer yet.

I have described the idea in short videos and posts before: a registration office for AI, like the one every car has to pass before it gets a number plate. This post is the long version for engineers, risk teams and researchers. It describes the components, the data that flows between them and the rules that are enforced in code rather than in a policy document. It is written from practice in regulated industries, where BaFin, DORA, MaRisk and the EU AI Act set the frame, but nothing in it is specific to Germany.

The design principle: control before model

We never start a project with the model. We start with four questions: Who owns this system? What is it allowed to see and do? How do we know afterwards what it did? Who can stop it, and how fast? The model is chosen last, because models change every few months and the answers to those four questions should not.

The consequence is an architecture in which the business application never talks to a model directly. Every call goes through a gateway that knows the registry, enforces the scope, pseudonymises the data, writes the evidence and escalates to a human when the signals say so. If the gateway does not know you, you do not get a model. That single rule removes the most common failure mode we see in the field: the shadow deployment that nobody registered, nobody evaluated and nobody can switch off.

The architecture

BUSINESS SIDEGOVERNANCE LAYERMODELS AND AGENTSBusiness applicationsClaims, credit, HR, service deskCall the gateway, never a modelCarry a use-case ID per requestHuman oversightTiered review: full, sampled, noneEscalation queue with ownersDecisions feed back into policiesBusiness ownerSigns the registration recordHolds the kill switch per modelSets risk tier and re-evaluationRegistryOne record per model or agent:owner, purpose, risk class, scope,version, eval results, expiryNo registration, no routeUnique ID: the license platePolicy gatewaySingle entry point for all callsScope and data-zone checksPseudonymisation: IDs in, IDs outTool allow-lists and budgetsAbstain or escalate on low supportEvidence logAppend-only, per decision:input hash, data reference, modeland version, sources, policy version,output, signals, reviewer actionReconstructable months laterLanguage modelsHosted or on-premise, by versionReached only through the gatewayRetrieval (RAG)Access rights enforced at query timeProvenance stored with every chunkAgents and toolsLeast privilege, own credentialsSandbox by network policy, not promptrequest with use-case IDlookupwriterouted callescalationalerts, drift, out-of-scopeMonitoring reads the evidence log and the gateway; a model whose registration expires loses its route automatically.

Figure: the governance layer between business applications and models. Arrows show the request path, the registry lookup, the evidence write and the two escalation paths to humans.

1. Registry: the license plate

The registry holds one record per model or agent in production. It is the source of truth for the question “what is allowed to run here, for whom, and until when”. A record looks like this, simplified:

{
  "model_id": "DE-B-2026-0143",
  "owner": "Head of Claims, Business Unit Motor",
  "purpose": "Pre-classification of incoming claims (routing, not decision)",
  "risk_class": "limited (EU AI Act), tier 2 (internal)",
  "scope": {"data_zones": ["green", "gray-with-pseudonyms"], "actions": ["classify", "summarise"], "tools": []},
  "model": {"provider": "hosted LLM", "version": "2026-09-r2", "prompt_version": "claims-routing-v14"},
  "evaluation": {"set": "claims-eval-2026Q3 (412 cases)", "passed": "2026-10-02", "bias_check": "passed"},
  "oversight": {"review": "sampled 5 %, blind re-check 1 %", "escalation": "confidence below threshold or disagreement with rules engine"},
  "expiry": "2027-04-02",
  "kill_switch": "business owner, risk, IT operations"
}

Three fields do most of the work. Scope is what the gateway enforces per request: data zones, allowed actions, allowed tools. Evaluation is the proof that the model passed the pre-deployment gates on a test set the business owns, not the vendor. Expiry forces re-evaluation: a model whose registration lapses loses its route automatically, because models drift, data drifts and the people who signed the record move on.

2. Policy gateway: the only door

All model calls, retrieval queries and agent tool calls pass through the gateway. Per request it does five things, in order: it resolves the use-case ID to a registry record and rejects unknown ones; it checks the data zone of the payload against the scope; it pseudonymises personal data, so that IDs go to the model and names never do, and maps the IDs back on the way out; it applies budgets and allow-lists for agents (steps, tokens, time, tools); and it evaluates the output signals that decide between return, abstain and escalate. The gateway is boring software on purpose. Its value is that there is exactly one of it.

3. Evidence log: reconstructable decisions

The evidence log is append-only and is written by the gateway, not by the application. Per decision we store: a hash of the input and a reference to the pseudonymised payload, the model and prompt version, the retrieved sources with their document versions, the policy version in force, the output, the signals (support from retrieval, agreement with a rules engine where one exists, abstention), and the reviewer action if a human was involved. The test we apply is simple: can an auditor, nine months later, reconstruct why this claim was routed this way without asking the engineer who built it? If the answer is no, the log is incomplete.

4. Human oversight: tiered, not total

“A human checks every output” is the promise everyone makes and nobody keeps at volume. We tier it. High-impact decisions about people get full review before anything leaves the system. Medium-impact outputs get sampled review at a rate that is statistically meaningful for the volume, plus a blind re-check of a smaller sample to catch rubber-stamping. Low-impact outputs get monitoring only. Every reviewer action is logged and fed back: into the prompt version, the retrieval set or the policy. Oversight that does not change the system is theatre.

5. Monitoring and the kill switch

Monitoring reads the gateway and the evidence log and watches for three things: drift in the distribution of outputs, out-of-scope attempts (a request with data the model is not registered for, an agent trying a tool that is not on its list), and disagreement between the model and whatever independent signal exists. Alerts go to the business owner, not only to engineering. The kill switch is a registry flag that the gateway reads on every call, and the business owner can set it without a deployment. We learned the hard way that a kill switch only engineers can pull is not a control the business trusts.

What the layer refuses by construction

  • An unregistered model. There is no route for it. This is the single most effective control we have, because it turns governance from a review meeting into a routing table.
  • Raw personal data in a prompt. The pseudonymisation stage replaces identifiers with IDs before the model sees anything. Free text is scrubbed for names and addresses as well, because people write them into comment fields.
  • A tool call outside the allow-list. Agents get their own credentials with the least rights needed and a sandbox defined by network policy. A prompt that says “do not access production” is an instruction, not a boundary.
  • A high-impact decision without a reviewer. The gateway holds it in the escalation queue. The application receives “pending”, not an answer.
  • An answer without support. Where retrieval is involved, an output whose claims are not backed by a retrievable source is returned as unverified or not at all, depending on the tier.

Where this meets the regulation

I am an engineer, not a lawyer, so I keep this short. The EU AI Act asks high-risk systems for risk management, logging, human oversight and technical documentation. BaFin, MaRisk and DORA ask banks for traceable decisions, controlled outsourcing and operational resilience. The governance layer is our answer to all of them with one piece of infrastructure: the registry is the documentation, the evidence log is the logging, the tiers are the oversight, the gates are the risk management. Supervisors do not want a slide deck about responsible AI. They want to pull a decision and see its history. More on the legal side in my overview of the EU AI Act.

What we got wrong

  • We logged everything, then could not find anything. The first evidence log was a dump of requests and responses. Audits became archaeology. The structured record above is the result of that failure.
  • We trusted the model’s own confidence. A fluent answer is not a correct one, and a self-reported score is not calibrated for decisions about people. We moved to external signals: retrieval support, agreement with independent rules, consistency across samples, and abstention as a first-class outcome.
  • We put boundaries into prompts. They held until the first creative user. Boundaries now live in the gateway and the network, and we try to break them in staging before anyone else does.
  • Reviewers started waving things through. Sampled review at a steady rate produced automation bias within weeks. The blind re-check and the feedback loop fixed most of it, not more reviewers.
  • Nobody wanted to own the kill switch. Until an incident. Ownership is now part of the registration record and nothing goes live without a name in that field.

Open questions

This is the part I find most interesting, and the reason I read the research coming out of the safety teams of the large labs more closely than their product announcements.

  • Oversight at scale. Tiered review works at today’s volumes and today’s model capabilities. How do we keep humans meaningfully in the loop when models prepare more decisions than any sampling scheme can cover, and when the models are better at the task than the reviewer?
  • Control that holds against a capable adversary. Our boundaries are designed against mistakes and against users. The reports this year about agents leaving their test environments suggest the question “would this boundary hold against a model that is trying to get around it” is no longer academic. I wrote about those reports in AI agents are breaking out.
  • Explanations that are faithful. The EU AI Act wants decisions explained. A model can produce a plausible explanation that has nothing to do with how it arrived at its output. Interpretability research is the only path I see to explanations a supervisor should actually trust.
  • Honesty signals. Abstention and retrieval support are proxies. What we would want is a reliable signal from the model itself that it does not know, and a way to verify that signal is not gamed.

Frequently asked questions

Does the gateway add latency?

Yes, in the order of tens of milliseconds for registry lookup, pseudonymisation and logging. Nobody in a regulated process has ever rejected the layer for that reason. They reject it when it is not there and the audit comes.

Does this work with any model provider?

Yes. The layer is provider-agnostic by design; the registry record names the provider and version, the gateway speaks to whatever is behind it. Switching models is a registration change plus a run through the gates, not a rebuild.

Is this only for high-risk systems?

No. The tiers make it cheap for low-risk systems: registration, a light gate, monitoring only. The point is that every system is known. The expensive controls are reserved for decisions about people.

About the author

I am Umair Zaffar, founder and CEO of sifamo GmbH, an AI consultancy for regulated industries with offices in Berlin and Frankfurt. I come from enterprise architecture and have built large platforms for banks, insurers and the public sector before turning to AI in production. How the company came about is on my founder page; what we do for clients is under AI Governance. Comments, objections and better ideas are welcome on LinkedIn.

Related: AI Governance · EU AI Act · AI Consultant · AI Expert

Leave Comment