Enterprise AI Agent Architecture: Layers and Build Order

It is a layered design that separates what an agent decides from what it is allowed to do. The model reasons and tools act. Identity, guardrails, approvals and limits sit between the two, and observability records everything. Keeping the layers apart lets you change a model, add a tool or tighten a rule without rebuilding every agent.

• The model is one replaceable layer. Reach it through a gateway instead of wiring a provider into every agent.

• Tools are where the risk is. Connect them through one standard and give them the user's permissions, not an admin key.

• Controls sit outside the model. Guardrails, approvals and budgets are code the model can't talk its way past.

• Build observability first. Every other layer is tuned from what the logs show.

An AI agent is software that uses a language model to choose its next step and then takes it through a tool, as what is an AI agent? explains. In a company, that loop sits inside eight layers:

 Entry points: chat, email, tickets, schedules, webhooks, events
                               │
┌──────────────────────────────▼──────────────────────────────┐
│ AGENT RUNTIME: starts runs, enforces budgets, pauses agents │
│ and restarts them after a crash                             │
│  ┌───────────────────────────────────────────────────────┐  │
│  │ AGENT: instructions and the loop                      │  │
│  │ decide → call a tool → read the result → answer       │  │
│  └──────────┬──────────────────────────────┬─────────────┘  │
│             │ model calls                  │ tool calls     │
│  ┌──────────▼──────────┐      ┌────────────▼─────────────┐  │
│  │ GUARDRAILS          │      │ GUARDRAILS               │  │
│  │ check input, output │      │ APPROVALS for risky acts │  │
│  └──────────┬──────────┘      └────────────┬─────────────┘  │
└─────────────┼──────────────────────────────┼────────────────┘
              ▼                              ▼
   ┌─────────────────────┐      ┌──────────────────────────┐
   │ MODEL GATEWAY       │      │ TOOLS (MCP servers)      │
   │ → approved models   │      │ → business systems, DATA │
   └─────────────────────┘      └──────────────────────────┘

 IDENTITY       the agent's ID and the user's ID go with every call
 OBSERVABILITY  every box writes traces, audit records and cost

The product names below are examples of each kind, not recommendations. Several vendors sell products that cover more than one layer.

1. Models and a model gateway

The model is the language model that reads the request and decides what to do. Options are hosted model APIs (OpenAI, Anthropic, Google Gemini), many of the same models through your cloud provider (Microsoft Foundry, Amazon Bedrock, Google's Gemini Enterprise Agent Platform), and open-weight models such as Llama, Mistral or Qwen on your own servers. A model gateway sits in front of all of them and gives you one place for keys, routing, fallback to a second model, rate limits and spend reports. LiteLLM, Portkey, Kong AI Gateway and Cloudflare AI Gateway are examples. Decide which models are approved for which classes of data your company already defines.

2. Tools, connected through MCP

Agents act through tools: search the knowledge base, read an order, create a ticket, issue a refund. MCP (Model Context Protocol) is an open standard for connecting agents to tools, so a tool built once works with any agent that speaks it. What is MCP? explains it in depth. You'll use three kinds of tool servers: ones vendors host for their own products (GitHub, Atlassian and Salesforce offer them), ones your teams build with the official MCP SDKs or frameworks such as FastMCP or Promptise Foundry, and gateways that put many servers behind one endpoint with shared sign-in and logging, such as Amazon Bedrock AgentCore Gateway or Docker MCP Gateway. Decide which tools only read data and which change it, and give every tool server an owner.

3. Identity

Every tool call should carry two identities: the agent's own, and the user's on whose behalf it acts. The agent's identity says which software is calling. The user's identity limits what it may touch to what that person could touch. Your identity provider, such as Microsoft Entra ID, Okta, Ping Identity or Keycloak, issues user tokens. Workload identity issues the agent's: your cloud's built-in identities, or SPIFFE for workloads across clouds. Some vendors now sell agent identity products, such as Microsoft Entra Agent ID and Amazon Bedrock AgentCore Identity. Decide on short-lived tokens, one identity per agent and a way to revoke it in minutes. AI agent identity shows the details for engineers.

4. Guardrails

Guardrails are automatic checks on the text going into and out of the model. They look for prompt injection (text in a document, email or web page that tries to give the agent new instructions), sensitive data, harmful content and broken output formats. Managed services include Amazon Bedrock Guardrails, Azure AI Content Safety and Google Cloud Model Armor. Open-source options include NVIDIA NeMo Guardrails, Guardrails AI and classifier models such as Llama Guard. Check tool results as well as user input, because injected instructions usually arrive through data. Decide what each check does when it fires: block, redact or log. What is an LLM guardrail? covers the types and their limits.

5. Approvals

An approval step pauses the agent before an action that is risky or hard to undo, and asks a named person. It can live in the agent framework (for example, LangGraph's interrupts), on the tool server, or in a workflow tool. Send requests to where approvers already work: Slack, Microsoft Teams, ServiceNow or Jira. Decide which actions need approval, who approves, how long a request waits, and what happens if nobody answers. The safe default is no action. See what is human-in-the-loop AI?

6. Agent runtime

The runtime is where agents live. It starts runs from chat messages, schedules and events, applies limits per run and per day, pauses an agent on request, and recovers after a crash. Managed options include Amazon Bedrock AgentCore Runtime, Microsoft Foundry Agent Service, Google's Agent Runtime and LangSmith Deployment. Durable workflow engines such as Temporal or Restate can carry agent steps. Self-hosted runtimes, such as the Promptise Agent Runtime, run on your own containers or Kubernetes. What is an agentic runtime? compares them. Decide who can pause an agent and how fast.

7. Observability

Observability means recording what each agent did, so you can debug it, audit it and pay for it. You need three records: traces of every model call and tool call with tokens, cost, time and result; an audit log of who did what on whose behalf, protected from edits; and metrics with alerts. OpenTelemetry is the common format. LLM observability tools such as Langfuse, LangSmith, Arize Phoenix and Datadog LLM Observability read it, and your security monitoring system (Splunk or Microsoft Sentinel, for example) should receive the audit records. Decide retention under your existing records policy, and who may read prompts. AI agent observability shows what a trace contains.

8. Data

Agents read documents, remember conversations and reach systems of record. Retrieval runs on search indexes and vector databases, which find text by meaning rather than exact words, such as Elasticsearch, OpenSearch, pgvector or Pinecone. Memory and conversation state live in stores such as PostgreSQL or Redis. Systems of record, such as your ERP or CRM, should be reached through tools, not copied into the agent's memory. The rule for this layer: an agent sees only what the user could see, and each model receives only the classes of data approved for it.

An employee writes to the IT help desk agent in Microsoft Teams: "My laptop locked me out. Can you reset my MFA?" (MFA is multi-factor authentication, the second sign-in step.)

• Entry and runtime. The Teams message reaches the runtime, which starts a run for the help desk agent with the employee's identity attached and checks today's run limit.

• Guardrails on input. The message is scanned. It is a normal request, so it passes.

• Model gateway. The agent's model call goes to the approved model. The model decides to call lookup_user, then reset_mfa.

• Identity. Each tool call carries the agent's identity and the employee's token. The tool server checks that this agent may call reset_mfa, and only for the person asking.

• Approval. reset_mfa is marked high risk, so a request goes to the service desk queue. A technician confirms the employee's identity by phone and approves.

• Tool and guardrails on output. The reset runs. Its result is scanned before the model reads it, because tool results can carry injected text.

• Observability. Every step is in the trace. The reset and the approval go to the audit log and on to the security monitoring system.

• Answer. The agent tells the employee the reset is done and how to enroll a new device.

If the technician doesn't answer within the timeout, the reset doesn't happen and the agent tells the employee who to call.

Not on day one. The layers you need depend on what the agents may do.

If you buy a single assistant as a SaaS product, the vendor runs most of these layers. Your work is then sign-in, data rules, and getting logs out. The AI agent platform evaluation checklist helps you check what a vendor covers.

Build the layers in this order. Each step makes the next one safer.

• 1. Observability and audit logging. Turn it on before the first pilot. Without traces, you can't tune guardrails, size budgets or answer "what did the agent do?"

• 2. Identity. One identity per agent, the user's identity passed to tools, and no shared admin keys. Retrofitting identity means re-permissioning every tool.

• 3. A model gateway. Route all model calls through one place, so keys, approved models and spend sit in one spot from the start.

• 4. Read-only tools. Connect the first tools over MCP, read-only, with the user's permissions. Most first use cases need only reads.

• 5. Guardrails. Add checks on input, output and tool results, tuned against the traces from step 1.

• 6. Approvals, then write tools. Put the approval step in place before the first tool that changes data goes live, not after.

• 7. Runtime limits and triggers. Add budgets per run and per day, pause and restart, then schedules and event triggers.

• 8. The shared platform. Once two or three agents work, extract what they share: templates, the tool catalog, an agent register and review routines.

Steps 1 to 4 are often enough for a first pilot. Steps 5 and 6 come before any agent changes data. Step 8 comes from experience, not from a plan drawn before the first agent.

Use it before an agent goes to production, and again each time it gets a new tool.

MODELS
[ ] Model calls go through the gateway, not directly to a provider
[ ] Each model is approved for the data classes the agent sends it
[ ] A fallback model is set, or the agent fails safely
TOOLS
[ ] Every tool server has a named owner
[ ] Tools are marked read-only or write; write tools are listed
[ ] The agent has only the tools its task needs
IDENTITY
[ ] The agent has its own identity; no shared admin keys
[ ] Tool calls carry the user's identity, and tools check it
[ ] Credentials are short-lived and can be revoked in minutes
GUARDRAILS
[ ] User input, model output and tool results are checked
[ ] Each check has a defined action: block, redact or log
APPROVALS
[ ] Every write action is either approved by a person or listed as safe
[ ] Approvers, timeouts and the no-answer default (deny) are set
RUNTIME
[ ] Limits per run and per day are set, with an alert to the owner
[ ] A named person can pause the agent, and it has been tested
OBSERVABILITY
[ ] Every model and tool call is traced with identity, cost and result
[ ] Audit records reach the security monitoring system
[ ] Retention follows the company's existing records policy
DATA
[ ] The agent sees only what the user could see
[ ] Memory and retrieval stores are inside approved boundaries

What are the layers of an AI agent architecture?

Eight: models behind a gateway, tools connected through a standard such as MCP, identity, guardrails, approvals, an agent runtime, observability and data. Entry points such as chat, email, schedules and webhooks start the runs. Identity and observability cut across all the other layers.

Where does MCP fit in an enterprise AI agent architecture?

MCP is the tool layer's standard: agents use it to find and call tools, and tool servers use it to expose your systems. It lets one tool server serve many agents and frameworks. Put identity checks, approvals for write tools and audit logging on or in front of the MCP servers.

Do we need a model gateway?

Once more than one team or more than one model is involved, yes. A gateway keeps keys out of agent code, enforces which models are approved, routes to a fallback when a provider fails, and reports spend per team. A single pilot can call one model directly, if it logs every call.

Should an AI agent use its own identity or the user's?

Both. The agent's own identity says which software is calling and lets you revoke one agent without touching others. The user's identity limits the agent to what that person may see and do. An agent that acts for nobody in particular, such as a nightly job, needs its own narrowly scoped identity.

How is an AI agent architecture different from a RAG architecture?

RAG (retrieval-augmented generation) fetches documents so a model can answer from them. It is part of the data layer. An agent architecture adds what RAG doesn't need: tools that change things, identity, approvals, limits and a runtime. Many agents use RAG as one of their tools.

Should we build or buy these layers?

Usually both. Most companies buy models and often the gateway, observability and guardrail services, and build the tools that reach their own systems. Build vs buy AI agents covers the decision.

• Compare platforms against this architecture: AI agent platform evaluation checklist.

• Set the operating rules around it: AI agent governance in practice.

• Plan which layers to build this year: the AI maturity model.

• For your engineers, how the runtime layer works: what is an agentic runtime?

• How agents authenticate to tools: AI agent identity.