The standard safety control for an AI agent is a permission dialog. The agent proposes an action—send this email, modify this file, issue this refund—and a person clicks approve. It feels responsible because a human remains “in the loop.” It also fails surprisingly quickly.
When approvals are frequent, users stop evaluating them. When the underlying context is complex, users cannot evaluate them. When the agent bundles several consequences behind one harmless-looking action, the dialog describes less than it authorizes. A confirmation box transfers accountability without necessarily transferring understanding.
A better foundation for agent safety is reversibility. Before asking how often a human should approve an action, ask what happens when the action is wrong. Can it be previewed, isolated, rolled back, compensated, or reconstructed? If not, the system is spending authority it has no reliable way to recover.
Anthropic’s overview of effective agent architectures recommends simple designs, transparent behavior, and carefully constructed tool interfaces. Its work on trustworthy agents in practice similarly frames human control as more than a ceremonial checkpoint. The product lesson is that safety lives in architecture and interaction design, not in the number of modal windows.
Not every action deserves the same friction
An agent reading a public document, drafting a private response, sending that response, and deleting the source record are four different risk classes. Many products flatten them into “tool use” and then choose between constant approval or blanket autonomy. Both options are crude.
Actions should instead be classified along at least three axes: consequence, reversibility, and observability. Consequence asks how much harm a mistake can cause. Reversibility asks whether the prior state can be restored. Observability asks whether a person or automated monitor can tell what happened. A low-consequence, reversible, well-observed action can run quietly. An irreversible action affecting money, identity, access, or external communication deserves a much stronger boundary.
This classification produces practical design rules. Let an agent reorganize a copied workspace, not the only copy. Let it prepare a payment, not settle it. Let it draft access changes as a transaction plan, then validate invariants before committing. Let it send routine messages through a short recall window when the channel supports one. Where reversal is impossible, require narrower tools and richer evidence.
Design tools around transactions, not buttons
Many agent integrations simply expose an application’s existing API. That is convenient for developers and hazardous for users. Conventional APIs assume a caller that understands their semantics. A language-model agent may select tools through probabilistic inference, operate with incomplete context, and carry misleading instructions from untrusted content.
Agent-facing tools should therefore resemble transaction protocols. A strong pattern has separate stages: inspect, propose, validate, commit, and verify. The inspect stage gathers current state. Propose creates a concrete change set without applying it. Validate checks permissions, invariants, budgets, and policy. Commit performs the bounded mutation. Verify confirms that the observed result matches the intended one.
Each stage should produce durable, structured evidence. “Updated the account” is not evidence. A useful record identifies the object, prior state, requested change, authorization basis, resulting state, and recovery method. That record supports debugging, user trust, compliance, and automated anomaly detection at once.
Software engineering already contains mature versions of this idea. Version control, database transactions, migrations, feature flags, staged rollouts, and infrastructure plans all separate intent from irreversible effect. Agent builders do not need a new philosophy so much as the discipline to apply established operational patterns beyond code.
Undo is a system capability, not a UI feature
An undo button is meaningless when the underlying system cannot reverse the operation. Deleting a local draft may be recoverable from a snapshot. Sending private data to an external recipient is not undone by removing the message from an outbox. Revoking a credential does not erase what it already accessed. Reversibility must be defined in terms of consequences, not interface state.
Some operations require compensating actions rather than literal rollback. A refund cannot unsend a shipment, but it can restore financial position. A mistaken calendar invitation can be canceled, though the social consequence remains. A bad database migration may require a forward fix because restoring the old schema would discard new writes. Good agent design names these limits instead of painting every failure with the reassuring word “undo.”
This is also why audit logs must be outside the agent’s mutation boundary. If an agent can alter both the world and the record of what it did, review becomes theater. Logs should be append-only, tied to tool execution rather than generated narration, and accessible to the user in a form that highlights material changes.
Autonomy should expand through earned recovery
Teams often frame autonomy as a ladder: begin with approvals, gather confidence, then remove oversight. Reversibility suggests a better progression. Start the agent in a sandbox. Allow reversible production actions with monitoring. Expand its scope when recovery works under realistic failures. Reserve irreversible decisions for explicit authority or narrowly specified policies.
This changes the success metric. The question is not merely how often the agent completes a task without intervention. It is how safely the system absorbs misunderstanding, stale context, tool failure, and adversarial input. An agent that makes fewer mistakes is desirable. An operation that survives mistakes is dependable.
At XioX, we think the most useful agent products will feel less like eager interns asking permission for every click and more like well-run operational systems. They will show plans when plans matter, act quietly when risk is bounded, preserve evidence automatically, and stop before crossing boundaries that cannot be repaired.
Greater model intelligence will reduce some errors, but it will never eliminate ambiguous intent or changing conditions. The durable advantage will come from products that assume mistakes remain possible. Before an agent receives more authority, it should demonstrate something more valuable than confidence: a credible path back.
Advertisement
