Least Privilege for AI Agents: Boundaries, Budgets and a Kill Switch
An agent that can call tools is the first AI system that acts instead of answers. This is the control design we insist on before one touches production, and why a sentence in the prompt is not a boundary.
For two years the question with language models was whether the answer is correct. With AI agents the question changes. An agent is a model that can call APIs, read and write files, send messages and decide on its own what the next step is. It no longer only answers, it acts. And an action inside a bank or an insurer has consequences that no correction in the chat window can undo.
At regulated companies I keep seeing the same pattern. The demo is impressive, the business side is excited, and nobody has asked what the agent is physically able to do on the day it gets something wrong. So we made a rule for ourselves: before an agent touches production, its boundaries have to exist, be tested and be documented. Control before model, as I like to put it.
Why a prompt is not a boundary
The most common safeguard I see in agent projects is a sentence in the system prompt: do not delete records, do not contact customers, stay inside this folder. It is cheap and it looks like a rule. But a prompt is an instruction to a probabilistic system. It shifts the odds, it does not set a limit. The model can misread it, a document the agent reads can contradict it, and a long task can push it out of view.
A boundary is something the agent cannot cross even if it tries. If the model were actively hostile, would the control still hold? A prompt fails that test every time. A network policy with no route to production passes it. That is the difference between a wish and a control, and an auditor asks about controls.
The same goes for the agent promising to behave. An agent that writes “I will only read, never write” into its plan has produced text, not a guarantee. We log such statements. We never count them as a control.
Seven boundaries for AI agents, before the first tool call
This is the list we work through with a client before the first production call. Nothing on it is exotic. It is ordinary security engineering, applied to a new kind of actor.
An allow-list of tools and scopes per agent. Which tools the agent may call and, per tool, which systems, which record types, read or write. Anything not on the list does not exist for the agent. The list lives in the governance layer, not in the prompt, and is part of the agent’s registration. No registration, no deployment.
Separate credentials per agent, least rights. An agent never runs on a human’s account and never shares a service account with another agent. Its credential carries only the rights the allow-list requires and expires quickly. When the log shows an action, I want to know which agent did it, not which team.
A sandbox that cannot reach production by construction. Testing happens in an environment whose network policy has no route to production, with no production credentials in reach and only synthetic or pseudonymised data. The isolation is enforced by infrastructure, not by telling the agent where it is.
Budgets with hard stops. Every run gets a ceiling on steps, tokens, wall-clock time and money. When a ceiling is reached, the run stops and a human is notified. Not warned, stopped. Loops that burn money are the most common agent failure, and a budget turns them into a log entry.
Human approval for irreversible actions. Messaging a customer, moving money, deleting data, changing a permission: these go into an approval queue. The agent prepares, a named person releases, and the approval is recorded with that person’s identity and the exact payload they saw.
An append-only action log. Every tool call with timestamp, agent identity, inputs, outputs and decision context goes to a log the agent cannot alter. Not a summary written by the agent, the actual calls. This is what makes an agent explainable, verifiable and correctable after the fact.
A kill switch the business owner can pull. One action that revokes the agent’s credentials and halts its runs, available to the business owner and not only to engineering. If the person accountable for a process cannot stop it at two in the morning, the process is not under control.
Who gets paged, and when
Boundaries without an operating model are documentation. So for every agent we define what happens at each threshold. A budget stop pages the engineer on duty and notifies the business owner. An approval request expires after a defined time, and expiry means the action does not happen. A blocked tool call, meaning the agent tried something outside its allow-list, is an incident: the run stops, the attempt is logged, and somebody looks at it the same day. Three blocked calls in a week mean the task and the allow-list do not match, and that is a design question.
The governance layer we have built registers the agent with its allow-list and its owner, monitors every call against that list, and executes the response when something is off. Registration, monitoring, response. Agents did not change that pattern, they made it unavoidable. Regulation points the same way: the EU AI Act, in force since August 2024, phases in its obligations over the coming years, and for high-risk systems it asks for exactly this, human oversight, logging and risk management that an auditor can inspect.
Test the boundaries by breaking them on purpose
A boundary that has never been attacked is an assumption. So before go-live we spend time in staging trying to get out. We give the agent a task that can only be solved by reaching production and check that it fails. We plant a document that instructs the agent to email a customer and check that the message never leaves the sandbox. We hand the agent a credential that is deliberately too broad and confirm that the allow-list still blocks the call.
Every attempt goes into the acceptance document. If an attempt succeeds, we do not fix the prompt. We fix the infrastructure, because the prompt was never the control, and then we try again. This is uncomfortable for project timelines, and it is the cheapest incident response a company will ever buy.
Open questions
This design is not finished. A few things keep me up at night, and they are research questions as much as engineering ones.
Oversight at scale. Human approval works for ten agents and a hundred irreversible actions a day, not for a thousand agents and a million actions. What a meaningful approval looks like at that scale, and how we keep the human from rubber-stamping, is unsolved.
Robustness of the boundary itself. Network policies and allow-lists are only as good as their configuration. How do we verify continuously, not only at acceptance, that the sandbox has no path out while the agent inside it is actively searching for one?
Interpretability of intent. The action log tells us what the agent did, not why. For a pattern of individually permitted calls that together achieve something forbidden, we currently have no reliable signal.
Honesty of the agent about itself. Agents report on their own progress and plans. We treat those reports as untrusted. A method for knowing when an agent’s self-report can be relied on would change the economics of oversight entirely.
Frequently asked questions
What does least privilege mean for AI agents?
The agent gets only the tools, scopes and credentials its defined task needs, and nothing else is reachable. The limit is enforced by infrastructure, not by an instruction in the prompt.
Is a system prompt enough to keep an agent within bounds?
No. A prompt changes probabilities, a boundary removes options. Use the prompt for quality and infrastructure for limits.
Who should be able to stop an agent?
The business owner of the process, the engineers who run it and the governance function. Stopping has to be one action that revokes credentials and halts all runs, practised before go-live, not during the first incident.
Least Privilege for AI Agents: Boundaries, Budgets and a Kill Switch
An agent that can call tools is the first AI system that acts instead of answers. This is the control design we insist on before one touches production, and why a sentence in the prompt is not a boundary.
For two years the question with language models was whether the answer is correct. With AI agents the question changes. An agent is a model that can call APIs, read and write files, send messages and decide on its own what the next step is. It no longer only answers, it acts. And an action inside a bank or an insurer has consequences that no correction in the chat window can undo.
At regulated companies I keep seeing the same pattern. The demo is impressive, the business side is excited, and nobody has asked what the agent is physically able to do on the day it gets something wrong. So we made a rule for ourselves: before an agent touches production, its boundaries have to exist, be tested and be documented. Control before model, as I like to put it.
Why a prompt is not a boundary
The most common safeguard I see in agent projects is a sentence in the system prompt: do not delete records, do not contact customers, stay inside this folder. It is cheap and it looks like a rule. But a prompt is an instruction to a probabilistic system. It shifts the odds, it does not set a limit. The model can misread it, a document the agent reads can contradict it, and a long task can push it out of view.
A boundary is something the agent cannot cross even if it tries. If the model were actively hostile, would the control still hold? A prompt fails that test every time. A network policy with no route to production passes it. That is the difference between a wish and a control, and an auditor asks about controls.
The same goes for the agent promising to behave. An agent that writes “I will only read, never write” into its plan has produced text, not a guarantee. We log such statements. We never count them as a control.
Seven boundaries for AI agents, before the first tool call
This is the list we work through with a client before the first production call. Nothing on it is exotic. It is ordinary security engineering, applied to a new kind of actor.
Who gets paged, and when
Boundaries without an operating model are documentation. So for every agent we define what happens at each threshold. A budget stop pages the engineer on duty and notifies the business owner. An approval request expires after a defined time, and expiry means the action does not happen. A blocked tool call, meaning the agent tried something outside its allow-list, is an incident: the run stops, the attempt is logged, and somebody looks at it the same day. Three blocked calls in a week mean the task and the allow-list do not match, and that is a design question.
The governance layer we have built registers the agent with its allow-list and its owner, monitors every call against that list, and executes the response when something is off. Registration, monitoring, response. Agents did not change that pattern, they made it unavoidable. Regulation points the same way: the EU AI Act, in force since August 2024, phases in its obligations over the coming years, and for high-risk systems it asks for exactly this, human oversight, logging and risk management that an auditor can inspect.
Test the boundaries by breaking them on purpose
A boundary that has never been attacked is an assumption. So before go-live we spend time in staging trying to get out. We give the agent a task that can only be solved by reaching production and check that it fails. We plant a document that instructs the agent to email a customer and check that the message never leaves the sandbox. We hand the agent a credential that is deliberately too broad and confirm that the allow-list still blocks the call.
Every attempt goes into the acceptance document. If an attempt succeeds, we do not fix the prompt. We fix the infrastructure, because the prompt was never the control, and then we try again. This is uncomfortable for project timelines, and it is the cheapest incident response a company will ever buy.
Open questions
This design is not finished. A few things keep me up at night, and they are research questions as much as engineering ones.
Frequently asked questions
What does least privilege mean for AI agents?
The agent gets only the tools, scopes and credentials its defined task needs, and nothing else is reachable. The limit is enforced by infrastructure, not by an instruction in the prompt.
Is a system prompt enough to keep an agent within bounds?
No. A prompt changes probabilities, a boundary removes options. Use the prompt for quality and infrastructure for limits.
Who should be able to stop an agent?
The business owner of the process, the engineers who run it and the governance function. Stopping has to be one action that revokes credentials and halts all runs, practised before go-live, not during the first incident.
Related: AI Governance · EU AI Act · AI Consultant
Recent Posts
Recent Comments