The dominant metaphor for AI agents is employment. Companies introduce a “researcher,” “developer,” or “support representative,” then debate how much of a person’s workload it can absorb. The metaphor is appealing because it makes unfamiliar software legible. It is also beginning to distort system design.
An employee has judgment shaped by social context, professional norms, accountability, and a persistent understanding of the organization. An agent has a model, a prompt, tools, credentials, and whatever state its designers provide. Giving it a role name does not supply the institutional knowledge implied by that role.
The more useful design question is not “What job does this agent perform?” It is “What is this system’s jurisdiction?” That means specifying what it may observe, what it may change, which commitments it may make, how much it may spend, and when its authority expires.
A tool list is not a control model
Agent platforms often expose permissions as a list of connected tools: repository access, browser access, email, calendar, databases, deployment systems. This is a necessary inventory, but it is not enough. Authority depends on verbs, scope, timing, and consequence.
Reading a production log differs from deleting it. Drafting an email differs from sending it. Opening a pull request differs from merging one. A database query against an approved replica differs from an unbounded query against production. If both actions appear under one broad “database access” toggle, the product has delegated a governance decision to chance.
Anthropic’s discussion of trustworthy agents in practice highlights the tension between useful autonomy and meaningful human control. The practical response should go beyond inserting approval prompts everywhere. Constant prompts train users to approve mechanically; they convert oversight into notification fatigue.
Jurisdiction should instead be encoded as a set of durable boundaries. An agent may edit files on a temporary branch but not the protected branch. It may issue refunds below a defined threshold when the evidence meets a policy, but must escalate ambiguous cases. It may prepare a campaign and calculate its projected spend, while publication and budget commitment remain separate permissions.
Move approvals to consequential boundaries
A good agent can perform hundreds of reversible operations before reaching an action that matters. Requiring approval for each step destroys its value. Allowing the final action without review creates avoidable risk. The design challenge is to locate the commitment boundary.
In software delivery, that boundary might be deployment rather than code generation. In customer support, it may be a financial concession or a statement that creates a contractual expectation. In research, it may be the inclusion of an unverified claim in a client-facing document. The system should permit broad exploration inside a controlled workspace, then require stronger evidence and explicit authority when crossing into shared reality.
This produces a useful three-zone architecture:
- A sandbox zone allows experimentation with disposable copies, synthetic records, and non-production resources.
- A proposal zone lets the agent create patches, drafts, plans, or transactions that can be inspected and tested but have no external effect.
- A commitment zone contains actions that alter production state, communicate externally, transfer money, or create obligations.
The zones should not merely appear in a user interface. They should be enforced by separate credentials, service boundaries, and policy checks. A persuasive model output must never be able to talk its way around an infrastructure rule.
Responsibility cannot be delegated to the transcript
When an agent causes harm, teams often reach for the activity log. Logs are essential, but a detailed transcript does not answer the organizational question: who was responsible for defining the agent’s authority and monitoring its results?
Every deployed agent needs a human owner with the power to narrow or revoke its jurisdiction. It also needs operational indicators that describe more than task completion: escalation frequency, attempted boundary crossings, rollback rate, corrections after review, and the age of the policies it relies on. These measures reveal whether autonomy is healthy or merely quiet.
Evaluation must follow the same structure. The research behind model-written evaluations demonstrates one route for generating behavioral tests at scale. Applied teams can borrow the principle without pretending synthetic evaluation is sufficient. Test not only whether an agent completes nominal tasks, but whether it refuses unauthorized variants, notices conflicting instructions, protects sensitive context, and remains inside its limits after a long sequence of legitimate actions.
Prompt injection makes this especially urgent. An agent that reads external documents, websites, messages, or issue reports operates in a contested instruction environment. Content is data, but models can interpret it as direction. Security therefore cannot depend on asking the model to remember which instructions outrank others. Tool gateways must enforce the distinction.
The employment metaphor will persist because it is vivid. Teams can keep the friendly names if they help people understand a workflow. But underneath the interface, an agent should look less like a digital colleague and more like a carefully bounded public institution: explicit powers, limited territory, reviewable decisions, and a reliable way to appeal or reverse its acts.
The companies that deploy agents well will not be those that grant the most autonomy. They will be those that make autonomy precise. A capable system with vague authority is a liability; a capable system with well-designed jurisdiction can become dependable infrastructure.
Advertisement
