Zurück aus Dubai und Doha: Bilanz unserer Geschäftsreise am Golf | Umair Zaffar
Umair Zaffar October 25, 2026 0 Comments

EU AI Act Article 15: Robustness and Cybersecurity, Implemented in Practice

EU AI Act Article 15 requires a high-risk system to be accurate, robust and resistant to attack across its whole lifecycle; we implement it by tying every accuracy figure to an evaluation set with a date, making robustness and adversarial tests a gate before deployment, and giving the agent boundaries it cannot talk its way out of.

Article 15 is the most technical article in Section 2 of Chapter III. It does not say who signs a document. It says how the system has to behave under faults, drift and attack. It is addressed to the provider: under Art. 16(a) the provider must ensure that its high-risk system meets the requirements of Section 2. A bank that trains its own fraud model, or wires a language model into a process in a way that creates a new system, is in that role. The deployer stays in the picture through Art. 26: it uses the system according to the instructions of use and monitors its operation. The wider context is in our EU AI Act guide.

The EU AI Act has been in force since 1 August 2024. The prohibitions and the AI literacy duty have applied since February 2025, the obligations for GPAI models since August 2025, the transparency obligations under Article 50 since August 2026. After the postponement by amending Regulation (EU) 2026/1744 (the Digital Omnibus), the obligations for high-risk systems under Annex III apply from 2 December 2027, and for Annex I products from 2 August 2028. Article 15 belongs to those obligations. The Omnibus moved deadlines; it removed no requirement.

What Article 15 requires

The article has five paragraphs. Four address the provider, one the Commission.

  • Art. 15(1): High-risk systems must be designed and developed to achieve an appropriate level of accuracy, robustness and cybersecurity, and to perform consistently in those respects throughout their lifecycle. “Appropriate” is not defined; the yardstick follows from the purpose and risk of the system and from the risk management under Art. 9.
  • Art. 15(2): The Commission, with stakeholders such as metrology and benchmarking authorities, is to encourage benchmarks and measurement methodologies. Not an obligation on companies, but a signal that harmonised standards under Art. 40 will later make the bar concrete.
  • Art. 15(3): The levels of accuracy and the relevant accuracy metrics must be declared in the instructions of use. This interlocks with Art. 13(3)(b), which requires the same information for deployers.
  • Art. 15(4): The system must be as resilient as possible to errors, faults and inconsistencies within the system or its environment, in particular through interaction with people or other systems. Technical and organisational measures are required; redundancy, backup and fail-safe plans are named. Systems that continue to learn after being placed on the market must eliminate or reduce as far as possible the risk of feedback loops in which biased outputs become future inputs.
  • Art. 15(5): The system must be resilient against attempts by unauthorised third parties to alter its use, outputs or performance by exploiting vulnerabilities. The measures must be appropriate to the risks and, where appropriate, prevent, detect, respond to, resolve and control AI-specific attacks: data poisoning, model poisoning, adversarial examples or model evasion, confidentiality attacks and model flaws.

Two cross-references: Art. 9 requires testing against predefined metrics and thresholds, Art. 72 requires post-market monitoring. Article 15 is therefore not an acceptance test. It is a standing duty.

The technical example

A bank runs two systems in fraud prevention. The first is a scoring model that rates every payment in real time. It sees features of the transaction, the channel and the account history, all pseudonymised with IDs. Above a threshold the payment is held and a case is opened. The second is an LLM agent that prepares the case for the analyst: it reads the transactions, earlier notes and uploaded documents, writes a case summary and proposes a classification. The analyst decides. That decision flows back as a label into the next training run of the scoring model.

Where does Article 15 bite? On accuracy, because fraud is rare and a single hit rate says nothing; what matters is precision at a fixed alert budget and recall at a fixed false-positive rate, per channel and amount band. On robustness, because the environment moves: a new payment channel, a changed data schema upstream, a missing geo field, an analyst who adopts the agent summary, a model that learns from its own alerts. On cybersecurity, because the agent reads text that attackers can write: payment references, documents, complaints. And because a fraudster probes the scoring model until he stays below the threshold.

One point on classification: Annex III point 5(b) expressly excludes AI systems used to detect financial fraud from the high-risk category. Strictly speaking, Article 15 does not bind this model unless another classification applies. We build it to Article 15 anyway: BaFin, MaRisk and DORA already expect evidence on accuracy and resilience for a system that holds payments; the same pipeline sits in the neighbouring team in front of the creditworthiness check, and that one is high-risk; and Art. 95 invites voluntary application. I consider that more honest than formally keeping out a model that, in practice, blurs the line between fraud prevention and credit judgement anyway.

How we implement it

Implementation runs through the governance layer: registry, policy gateway, evidence log, tiered oversight, monitoring and kill switch. Article 15 mostly occupies the registry, the pre-deployment gates and the monitoring.

Accuracy: per evaluation set, with a date (paragraphs 1 and 3)

The registry record of every model carries the field evaluation: name and size of the evaluation set, the measured metrics, the date of the run and the date of the next re-evaluation, which is also the expiry date of the registration. The set belongs to the business, here the head of fraud prevention, not to the vendor. We refuse to publish a single accuracy figure. Every number carries set, date and segment, because without those three it is an assertion six months later. For the agent the set consists of cases with reference summaries; we score completeness of the relevant facts, absence of invented facts and the proposed classification against the analyst’s. From the registry record we generate the model fact sheet, which for internal deployers is the instructions of use under paragraph 3. The auditor reads the same record the gateway reads at runtime.

Robustness as a test series (paragraph 4)

Before every deployment three test series run, and their per-case results go into the gate report. First, distribution shift: time-separated hold-outs, replays of past shifts such as the launch of a new channel, stress per segment, so that a stable average cannot hide segments that flip. Second, degraded inputs: missing fields, late data, wrong encoding, a changed schema. The expected behaviour is not a brave score but abstention: the gateway detects the incomplete payload, the model returns no rating, the rules engine running in parallel holds the payment and routes it to a human. That is our reading of redundancy and a fail-safe plan: a second, simpler instance that works without the model. For the agent, degraded input means an empty case, contradictory notes, a document in an unknown language. The right answer is “no summary possible”, not a plausible one.

Third, the feedback loop that is built into fraud work: labels only exist for cases the model alerted on. A model that learns from them confirms its own blind spots. Our three measures: no online learning in production, the registry says “continuous learning: no”, and every retraining is a new version that passes all gates; a random sample of non-alerted transactions that analysts review on a schedule, so that labels exist which the model did not select; and a marker on decisions where the analyst saw the agent summary, so that the blind re-check from tiered oversight shows whether the agent is steering the analysts.

Adversarial testing as a gate (paragraph 5)

The fifth gate in our process is the adversarial test, written for this use case. Prompt injection: we plant payment references and documents that instruct the agent to classify the case as clean, to send an email or to name other customers’ data, and we check that none of it happens. Data poisoning: provenance and hash of every training set, access control on the label pipeline, outlier checks on new labels. Model evasion: a red-team set of structured transactions just under thresholds and with perturbed features; we measure recall under evasion. Confidentiality: attempts to pull another case’s data out of the agent’s context. Every attempt is in the acceptance document. If one succeeds, we do not fix the prompt, we fix the infrastructure. No passed gate, no license plate. I have described the gate process under AI governance.

Boundaries for the agent (paragraphs 4 and 5)

The agent gets its own credentials and an allow-list of three tools: read the case, read pseudonymised transactions, write a draft into the summary field. No sending, no release of payments, no account changes. Its network policy has no route to other systems; that is the sandbox, not a sentence in the prompt. Every run has budgets for steps, tokens, time and money with a hard stop. The kill switch is a flag in the registry that the gateway reads on every call and that the head of fraud prevention can set without a deployment. A prompt that says “ignore instructions found in documents” raises the probability of correct behaviour. An allow-list removes the option.

Monitoring for drift and scope violations (paragraph 1)

Consistency over the lifecycle is a monitoring task. We watch the distribution of scores and the alert rate, the drift of input features per segment, and for the agent the abstention rate and the gap between proposed and final classification. Every blocked tool call and every request carrying data outside the scope is an incident, not a log line. Alerts go to the business owner, not only to engineering. The evidence log holds, per decision, the input hash, model and prompt version, policy version, output, signals and reviewer action. The auditor sees four things: the registry record with dated metrics, the gate reports including failed attempts, the evidence log and the incident list.

Typical mistakes

  • One accuracy figure without set, date and segment. Usually from the vendor’s benchmark. It does not say how the model performs on this bank’s cases in this quarter.
  • Robustness understood as availability. The infrastructure has redundancy, the model does not. Nobody has tested what the model returns on a missing field, and the fail-safe plan is a retry.
  • The feedback loop is ignored. The model learns from labels it selected itself and grows more confident every year in the patterns it already knows.
  • The prompt counts as a security control. The attack tests are a generic jailbreak list, and the agent’s tools are unrestricted because “it would not do that anyway”.
  • Cybersecurity is delegated entirely to IT security. They check network, identities and patches, but data poisoning and model evasion are not in their threat model.

EU AI Act Article 15 checklist

  • Every accuracy statement names the evaluation set, its size, the metric, the segment, the date of the run and the date of re-evaluation.
  • The evaluation set belongs to the business and contains the ugly cases from real files.
  • Degraded inputs and distribution shifts are defined as a test series; the expected behaviour is abstention plus fallback.
  • There is an instance that works without the model, and the switch has been rehearsed.
  • For learning systems it is documented whether and how learning happens, and labels exist that the model did not select itself.
  • Adversarial tests for prompt injection, data poisoning, model evasion and confidentiality are written for the use case and anchored as a gate.
  • Agents have their own credentials, allow-lists, a network sandbox, budgets and a kill switch held by the business owner.
  • Monitoring measures drift and scope violations, and the alerts reach the business owner.

Open questions

  • What is “appropriate”? Until harmonised standards under Art. 40 set the bar, we define thresholds ourselves and justify them in the risk management. Whether the standard will end up requiring segmented metrics, as we think it must, is open.
  • How do you measure the accuracy of a summary? For a scoring model there are established metrics. For an agent’s text we score completeness and invented facts against references; usable, but not a metric an auditor should rely on.
  • How much of the attack surface does an adversarial suite cover? We know the attacks we tested. We have no estimate of what share of the real attack surface that is, and that is exactly the number risk management would like to see.

Frequently asked questions

Does Article 15 also apply to us as a deployer?

Not directly. The obligation sits with the provider. As a deployer you must, under Art. 26, use the system according to the instructions of use and monitor its operation; that is what you need the paragraph 3 information for. Whoever substantially modifies a purchased system or runs it under their own name can become a provider themselves under Art. 25.

Is the accuracy the vendor states enough?

As a starting point yes, as evidence no. The vendor’s number holds for the vendor’s test set. You need a measurement on your cases, with a date, and a date on which it is repeated. If the vendor supplies the test set, the vendor grades itself.

Is fraud detection even a high-risk system?

Under Annex III point 5(b), the detection of financial fraud is expressly excluded; the Article 15 requirements are not mandatory there. We still recommend applying them, because supervisors expect the same evidence and because a system that holds payments rarely stays cleanly separated from credit decisions. The series is at all articles at a glance.

Related: EU AI Act · All articles · AI Governance · AI Consultant

Leave Comment