Contents · 8 sections↓
AI agent governance is how a company keeps control of software that acts on its own: agents that open tickets, change records, send emails or spend money. Policies on paper matter less than the controls that run every day. This guide covers those operational controls: who owns each agent, who approves what, which limits apply, what gets logged, how often agents are reviewed and what happens in an incident. It includes an approval matrix, a register entry and a go-live checklist you can copy.
How do you govern AI agents?
Give every agent a named owner and a written scope. Limit what it can do with narrow permissions and budgets, and require a person's approval before actions that are risky or hard to undo. Log every action, review each agent on a fixed schedule, and keep a tested way to stop it within minutes.
Govern actions, not answers. The risk of an agent is in what it can do with its tools.
Controls live outside the model. A rule in the agent's instructions is a request. A permission, an approval step or a budget is a control.
Match controls to impact. An agent that drafts replies and an agent that issues refunds need different rules.
Practice the stop. A kill switch nobody has tested is a hope, not a control.
This guide covers operational controls only. It does not explain what any law or standard requires. Some uses of AI are regulated in some places. Ask your legal and compliance team what applies to you before you roll this out.
[02]
What does AI agent governance cover?
A chat assistant produces text that a person reads and then acts on. An agent acts itself, at machine speed, with real credentials. That shifts governance from "what may people paste into AI?" to "what may this software do, and how do we know what it did?" In June 2025, Gartner predicted that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value or inadequate risk controls. Cost and risk controls are exactly what operational governance provides.
Six controls cover it, plus a register that lists every agent:
Ownership: a business owner and a technical owner for each agent.
Approvals: who approves launching an agent, and who approves its risky actions.
Limits: permissions, budgets and caps that the agent can't exceed.
Logging: a record of every action, on whose behalf, with which result.
Review cadence: fixed dates to check that each agent still does the right thing.
Incident handling: how to stop an agent, undo its work and learn from it.
Every agent follows the same life cycle:
PROPOSE ─▶ REGISTER ─▶ LAUNCH REVIEW ─▶ RUN ─▶ SCHEDULED REVIEW
▲ │ │
│ │ limits, ├─▶ keep
│ │ approvals, ├─▶ retire
│ │ logs │
└────────────┼─────────────┴─▶ change
▼
INCIDENT: stop ─▶ contain ─▶ tell ─▶ fix ─▶ reviewA change to an agent's instructions, model or tools goes back through the launch review, because it changes what the agent does.
[03]
Which controls does each agent need?
Scale the controls to what the agent can do. Three tiers are enough for most companies.
Tier | What the agent does | Examples | Minimum controls |
|---|---|---|---|
1. Assist | Reads data and drafts; a person decides and acts | Summarizes tickets, drafts replies, searches policies | Owner, register entry, logging, approved data classes |
2. Act, reversible | Changes data within one team's systems, easy to undo | Tags tickets, updates a CRM field, books a meeting room | Tier 1, plus least-privilege permissions, limits per run and per day, monthly review |
3. Act, high impact | Moves money, contacts customers, changes production systems or access rights, or can't be undone | Issues refunds, sends customer emails, resets access | Tier 2, plus approval for risky actions, a tamper-evident audit log, an owner on call, an incident runbook, weekly review in the first month |
Decisions about people, such as hiring, credit, health, legal or financial decisions, are outside these tiers. AI should not make these decisions alone. HR or legal must be involved before any agent is designed to support them.
[04]
The six controls, step by step
1. Give every agent an owner and a register entry
Each agent needs a business owner, who is accountable for what it does, and a technical owner, who keeps it running and answers when it breaks. Both go into an agent register: one list of every agent, with its purpose, tier, tools, data classes, limits, approval rules and review dates. The template is further down.
Make the register the gate: no register entry, no credentials. To find agents that skipped it, review the service identities and API keys that AI tools use in your identity provider, and the callers at your model gateway, if you have one.
2. Decide who approves what
There are two kinds of approval. Launch approval decides whether an agent may run at all, and it repeats for every change. Action approval pauses a running agent before a specific risky action and asks a person.
An approval matrix makes the second kind concrete. A support team's matrix might look like this:
Action | Approver | Waits up to | If nobody answers |
|---|---|---|---|
Refund within the limit finance sets | No one; within limits | n/a | n/a |
Refund above that limit | Support team lead | 4 hours | Denied; the customer is told a person will follow up |
Email to a customer | The employee the agent works for | 15 minutes | Kept as a draft |
Change to a user's access rights | IT service desk technician | 1 hour | Denied |
Bulk change to more than 50 records | Business owner | 1 business day | Denied |
The safe default is always "no action". Approvals stop working when people approve hundreds of requests without reading them, so send only real decisions to people and let rules handle the obvious cases. Approval fatigue explains how. For your engineers, human-in-the-loop approval shows how an approval step pauses a tool call in code.
3. Set limits the agent can't exceed
Limits are what keep a confused or manipulated agent small. Use four kinds:
Permissions. The agent gets only the tools its task needs, and acts with the user's identity, so it can't see what the user can't see. AI agent identity covers the mechanics.
Budgets. Caps per run and per day on model spending, tool calls and paid services.
Action caps. A maximum number of high-impact actions, such as refunds, per run and per day.
Time and scope. For example, no changes to production systems during a change freeze.
Decide what happens at a limit. For most agents, the right answer is: pause the agent and alert the owner. AI agent cost control shows how per-run and per-day budgets work in practice.
4. Log every action
The test for logging is simple: could you reconstruct, the next morning, what the agent did and why? Each run should record:
the agent, its version, and the user or trigger it acted for;
each model call, with the model used, tokens and cost;
each tool call, with its arguments, result and duration;
each approval: who decided, when, and what;
every limit hit, guardrail block and error.
Protect the audit part of that record. Make it append-only or tamper-evident, which means any edit or deletion can be detected, and send it to your security monitoring system. Keep it as long as your existing records policy says, and restrict who may read logs that contain prompts or customer data. AI agent audit logging shows one way to build a tamper-evident log.
5. Review on a fixed schedule
Agents drift. Models get updated, data changes, and a scope that was right in spring is wrong by fall. Put reviews on the calendar:
Review | When | Who | Looks at |
|---|---|---|---|
Launch review | Before go-live and after every change to instructions, model or tools | Owners, platform team, security | Test results, permissions, limits, approval rules |
Operations review | Weekly for the first month, then monthly | Technical owner | Errors, limit hits, denied approvals, cost |
Owner review | Quarterly | Business owner | Value against cost, scope, incidents, whether the agent is still needed |
Access review | With your existing access recertification | IT or security | Credentials, permissions, unused tools |
Retirement | When unused for a set period, or replaced | Owners | Revoke credentials, archive logs under the records policy |
6. Prepare for incidents
An agent incident is any case where an agent did, or nearly did, something it shouldn't: a wrong action repeated many times, data shown to the wrong person, a cost spike, or instructions hidden in a document that the agent followed. Handle it through your existing incident process, with these agent-specific steps:
Stop. Pause the agent. If it might act through other routes, revoke its credentials in the identity provider.
Contain. List what it did from the audit log, and undo what can be undone.
Tell. Inform the owners, security and the affected teams.
Fix. Find the cause in the trace, change the control that failed, and test the change.
Resume. Restart through a launch review, with tighter limits at first.
Learn. Record the incident in the register and review similar agents.
Decide in advance who may pull the stop (the owners, the on-call engineer, security) and test it every quarter. An agent runtime gives you pause and stop controls, budgets and health checks in one place. What is an agentic runtime? explains the idea, and the autonomous AI agent runtime shows it in code.
[05]
Templates you can copy
Agent register entry
Fill one in for every agent before it gets credentials. Values in brackets come from your own policies.
Agent name: Refund assistant
Agent identity: agent-support-refunds (its own account in the identity provider)
Business owner: Head of Customer Support
Technical owner: Support platform team (on-call rotation: support-eng)
Purpose: Handles refund requests from the support inbox
Tier: 3 (moves money, contacts customers)
Read tools: lookup_order, lookup_customer, search_refund_policy
Write tools: issue_refund, send_customer_email
Data classes: [the classes this agent may use, from your classification]
Approved models: [models approved for those data classes]
Approvals: issue_refund above [finance limit]: support team lead, 4 h, deny
send_customer_email: employee sends the draft, 15 min, keep as draft
Limits: 20 tool calls per run, 300 runs per day, [N] refunds per day
At a limit: pause the agent and alert the technical owner
Logging: full trace; audit log to security monitoring; [retention policy]
Stop: pause in the runtime (owners, on-call, security); revoke identity
Status: production since [date]
Reviews: operations [date]; owner [date]; access [date]
Change history: [date] model changed, launch review passedGo-live checklist
[ ] Register entry complete, with both owners named
[ ] Tier agreed; for decisions about people, HR or legal involved
[ ] The agent has its own identity and acts with the user's permissions
[ ] Only the tools the task needs; write tools listed
[ ] Approval matrix set: approver, wait time, "no action" if no answer
[ ] Limits per run and per day set; action at a limit defined
[ ] Every model call, tool call and approval logged
[ ] Audit records reach security monitoring; retention set
[ ] Stop tested: who can pause it, and how long it took
[ ] Incident steps written and shared with the on-call team
[ ] Review dates in the calendar
[ ] Legal and compliance asked what applies to this use[06]
Common mistakes
Mistake | Why it fails | Fix |
|---|---|---|
Rules only in the agent's instructions | The model can be talked out of them, and hidden text in a document can override them | Enforce rules with permissions, approvals and budgets |
One service account for all agents | You can't tell which agent did what, or stop one without stopping all | Give each agent its own identity |
Sending every action for approval | Approvers stop reading and approve everything | Approve only high-impact actions; let rules handle the rest |
Logging only the final answer | You can't reconstruct or undo what happened | Log every tool call with its arguments and result |
No owner after the project team moves on | The agent keeps running with nobody watching | Owners in the register; reviews flag agents without one |
A stop button nobody has tested | In an incident it fails, or nobody knows who may use it | Test the stop every quarter and record how long it took |
Changing the model or instructions without review | The agent's behavior changes silently | Treat every change as a new launch review |
[07]
Frequently asked questions
What is AI agent governance?
AI agent governance is the set of operational controls a company uses to keep AI agents in check: named owners, approval rules, limits on what agents can do and spend, logs of every action, scheduled reviews and incident steps. It focuses on actions, because agents act with real credentials instead of only producing text.
Who should own an AI agent?
Two people. A business owner, usually the head of the team whose work the agent does, is accountable for what it does and whether it's worth keeping. A technical owner keeps it running and answers when it breaks. Both are named in the agent register, and an agent without both doesn't get credentials.
Do all AI agent actions need human approval?
No. Approving everything leads people to approve without reading. Require approval for actions that are high impact or hard to undo, such as payments above a limit, messages to customers or access changes. Let read-only actions and small, reversible ones run within limits, and log all of them.
What should an AI agent audit log record?
The agent and its version, the user or trigger it acted for, each tool call with its arguments and result, each approval with who decided, every limit hit and error, and model usage and cost. Make the log tamper-evident, send it to your security monitoring system, and keep it as long as your records policy says.
How often should AI agents be reviewed?
Before launch and after every change to instructions, model or tools. Then weekly operations reviews in the first month, monthly after that, a quarterly owner review of value and scope, and access reviews on the same schedule as the rest of your access recertification.
How do you stop an AI agent in an emergency?
Pause it in the runtime or platform it runs on, and if it might act through other routes, revoke its identity in your identity provider. Decide in advance who may do this, write the steps down, and test them every quarter. Then use the audit log to list and undo what it did.
[08]
Next steps
See where your company stands: the AI maturity model.
Put these controls into the stack: enterprise AI agent architecture.
Check that a platform supports them: AI agent platform evaluation checklist.
For your engineers, implementation detail: human-in-the-loop approval, AI agent cost control, AI agent audit logging, the autonomous AI agent runtime and what is an agentic runtime?