An AI agent platform evaluation checklist helps you compare products that let AI agents work inside your company: read data, call your systems and change records. Vendor demos look alike. The differences show up in identity, approvals, logging, cost limits and what it takes to leave. This guide gives IT, security and platform leads nine weighted criteria, a 1–5 scoring sheet to copy, the questions to put to vendors, and a pilot plan that tests their claims with your own workflow.
How do you evaluate an AI agent platform?
Start with pass/fail gates for what you can't live without, such as where data is stored and support for your single sign-on (staff sign in once with their company account). Score the platforms that pass from 1 to 5 on weighted criteria: fit, security, identity, approvals, observability, cost control, deployment, portability and support. Then run a two- to four-week pilot on one real workflow and score again from what you saw. A demo shows what a platform can do; a pilot shows what it does with your data and your systems.
Gates come before scores. A high total must never buy back a failed must-have.
Set the weights before you meet vendors, so they don't drift toward a favorite.
Score evidence, not answers. A 5 means you saw it work in your pilot, not that the vendor said yes.
Ask how to leave before you sign. Exports, open standards and model choice decide what switching costs later.
[02]
What is an AI agent platform?
An AI agent is software that uses a language model to decide which steps to take, and then takes them through tools: it searches a knowledge base, creates a ticket or issues a refund. What is an AI agent? explains the idea in more depth. An agent platform is everything around the model that makes this safe to run at work: connections to your systems, identity, approval steps, limits, logs and a place for agents to run.
Products come in four kinds. Many companies end up using more than one.
Kind | Examples | Usually strong at | Check carefully |
|---|---|---|---|
Cloud provider agent services | Amazon Bedrock AgentCore, Microsoft Foundry Agent Service, Google Gemini Enterprise Agent Platform | Managed hosting, the provider's identity and network controls | Ties to one cloud, usage-based pricing |
Agents inside business software | Salesforce Agentforce, ServiceNow AI Agents, Microsoft Copilot Studio | Fast start on data already in that system | Reach into other systems, pricing per conversation or action |
Frameworks you run yourself | LangGraph with LangSmith, CrewAI, Promptise Foundry | Control, portability, no license fee for the open-source parts | You operate it; support depends on the community or a paid plan |
Automation tools with agent steps | n8n, Zapier, Make | Business teams build quickly | Fine-grained identity, audit and approvals |
The checklist below works for all four kinds, because it asks about outcomes, not technology.
[03]
When do you need a formal evaluation (and when not)?
The effort should match what the agents will be allowed to do.
Situation | What to do |
|---|---|
Staff want a chat assistant for writing and summaries | Use the assistant your company already licensed. This checklist is more than you need. See is it safe to use ChatGPT at work? |
One team wants an agent that reads internal documents and drafts answers | A light evaluation: fit, security, identity and observability |
Agents will change records, send messages or spend money | The full checklist and a pilot |
Several business units will build agents on one platform | The full checklist, with more weight on deployment, portability and support |
The agent would help decide about people: hiring, credit, health, legal or financial matters | AI should not make these decisions alone. Involve HR and legal before you evaluate any platform for it |
Whether to build or buy is a separate decision that comes first. Build vs buy AI agents covers it.
[04]
The evaluation criteria and what to ask vendors
For each criterion, the table says what good looks like and what to ask the vendor to show you. Ask for a live demonstration on a test account, not a slide. In June 2025, Gartner predicted that over 40% of agentic AI projects will be canceled by the end of 2027 because of escalating costs, unclear business value or inadequate risk controls. It also warned about "agent washing": vendors relabeling chatbots and automation as agents. A demonstration on your own test data shows which kind you're looking at.
Criterion | What good looks like | Ask the vendor to show you |
|---|---|---|
Fit and quality | The agent completes your real workflow correctly, and you can test a change before it goes live | One of your workflows running on your test data; how they test a prompt or model change before release |
Security | Data stays where you need it, text from documents and tools is checked for hidden instructions, code runs in an isolated sandbox, and secrets never reach the model | Where prompts, logs and files are stored and for how long; whether your data trains their models; a blocked prompt-injection attempt |
Identity and access | Each agent has its own identity in your identity provider and acts with the user's permissions, not a shared admin key | An agent that can't see a record the user can't see; how one agent's credentials are revoked |
Approvals | Risky actions wait for a named person, and no answer means no action | A refund held for approval in your chat or ticket tool, and one that timed out |
Observability and audit | Every model call and tool call is recorded with who, what, cost and result, and the records reach your own tools | The full trace of one run; the export to your logging or security monitoring system; how logs are protected from edits |
Cost control | Hard limits per run and per day, checked before money is spent, with alerts | A run stopped by a budget, and what the agent told the user |
Deployment and operations | Runs where you need it (your cloud, your data center or theirs), with separate test and production and recovery after a crash | An agent restarted in the middle of a task; how a change moves from test to production |
Portability | Tools connect through open standards, you choose the model, and you can export agent definitions and logs | Switching the model of a working agent; a full export of one agent |
Support and viability | Clear support terms, a security contact, a public changelog and a roadmap you can check | Their support terms, and their last security fix and how long it took |
Two terms need a plain explanation. Prompt injection is text inside an email, document or web page that tries to give the agent new instructions. An open standard for tools, such as MCP (Model Context Protocol), lets any agent use a tool built once, so your tool work survives a platform change.
Your engineers can see what good approvals, cost limits and logs look like in working code: human-in-the-loop approval, AI agent cost control and AI agent observability.
[05]
How to run the evaluation in eight steps
1. Write down the workflow. The goal, the systems the agent touches and every action it may take. Mark each action that changes data, sends a message or spends money.
2. Set the gates. List the must-haves as pass/fail: the regions where data may be stored, your identity provider, log export, the contract terms procurement needs. A platform that fails a gate is out.
3. Set the weights. Share 100 points across the nine criteria before you talk to any vendor. The sheet below has defaults.
4. Shortlist two or three platforms, ideally of different kinds, so you also test the trade-off between them.
5. Send the questions, then ask for a live demonstration. Mark each answer as shown, described or on the roadmap. Only "shown" can score above 3.
6. Pilot one real workflow for two to four weeks. Use representative test data and the people who will use the agent. Plan at least three failures: a tool that times out, an approval nobody answers, a budget that runs out.
7. Score with two people. For example, one from security or IT and one from the business team. Discuss every criterion where your scores differ by two points or more.
8. Write the decision down, with the scores, the evidence and the risks you accepted. Revisit it at renewal.
[06]
AI agent platform scoring sheet (copy this)
Score each criterion from 1 to 5:
1, missing: no feature and no workaround.
2, workaround: possible with custom work or another product.
3, partial: available, with gaps that matter to you.
4, meets the need: described and demonstrated.
5, proven: meets the need and worked in your pilot.
For each criterion, the weighted score is weight × score ÷ 5. The weighted scores add up to a total out of 100. Paste the sheet into a spreadsheet and make each weighted cell a formula: for the first criterion, cell D6 is =B6*C6/5.
Gate or criterion,Weight,Platform A score (1-5),Platform A weighted,Platform B score (1-5),Platform B weighted,Evidence and notes
GATE: Data stored only in the regions we need,pass/fail,,,,,
GATE: Works with our identity provider (single sign-on),pass/fail,,,,,
GATE: Logs export to our logging or security monitoring system,pass/fail,,,,,
GATE: Contract terms our procurement team requires,pass/fail,,,,,
Fit and quality,20,,,,,
Security,15,,,,,
Identity and access,12,,,,,
Approvals,10,,,,,
Observability and audit,12,,,,,
Cost control,8,,,,,
Deployment and operations,10,,,,,
Portability,8,,,,,
Support and viability,5,,,,,
Total (weighted = weight x score / 5),100,,,,,Ask your compliance team which requirements apply and add them as criteria.
The default weights suit agents that act on business systems. Adjust them to your case before you score:
If your agents will... | Raise | Lower |
|---|---|---|
Run unattended on schedules or events | Cost control, deployment and operations | Fit (once proven in the pilot) |
Talk to customers or send external messages | Security, approvals | Portability |
Serve many teams on one platform | Portability, support and viability | Approvals (set per agent) |
Only read data and draft for a person | Fit and quality, security | Approvals, cost control |
Worked example: an IT service desk agent
A company wants an agent that resets passwords and closes routine tickets. Two platforms pass the gates. Platform A is a managed cloud agent service. Platform B is an open-source framework that the platform team would run on its own Kubernetes cluster. After the pilot, the scores look like this:
Criterion | Weight | A score | A weighted | B score | B weighted |
|---|---|---|---|---|---|
Fit and quality | 20 | 4 | 16.0 | 4 | 16.0 |
Security | 15 | 4 | 12.0 | 4 | 12.0 |
Identity and access | 12 | 5 | 12.0 | 4 | 9.6 |
Approvals | 10 | 3 | 6.0 | 5 | 10.0 |
Observability and audit | 12 | 4 | 9.6 | 4 | 9.6 |
Cost control | 8 | 3 | 4.8 | 5 | 8.0 |
Deployment and operations | 10 | 5 | 10.0 | 3 | 6.0 |
Portability | 8 | 2 | 3.2 | 5 | 8.0 |
Support and viability | 5 | 4 | 4.0 | 2 | 2.0 |
Total | 100 | | 77.6 | | 81.2 |
The totals are close, so treat them as a tie and decide on the risks you'd accept. With A, the vendor runs the platform, but approvals need a workaround and leaving would be expensive. With B, the company controls approvals, limits and exit, but its own team must run and support the platform. The answer depends on whether that team exists. Write the reasoning into the decision record.
[07]
How Promptise Foundry answers this checklist
Promptise Foundry is the open-source Python framework behind this site. Here is how it does against this checklist, gaps included, as of version 1.4.3 (October 2026).
Criterion | What Promptise Foundry provides | Gaps |
|---|---|---|
Fit and quality | Agents that call tools over MCP; prompt tests and a test client for tool servers | No general evaluation suite; Python only |
Security | Local scanners for prompt injection, personal data and credentials, including tool results; a Docker sandbox with no network by default | Docker-only sandbox; no connectors to external secret managers |
Identity and access | Agent identity from Microsoft Entra, AWS, Google Cloud, SPIFFE or other OpenID Connect providers; the user's token passed to tool servers; role and scope checks per tool | No identity store of its own (by design); no mutual TLS |
Approvals | Approval rules per tool, through your code, a signed webhook or a queue; deny on timeout by default; auto-approval rules for safe actions | No ready-made Slack or Teams app; server-side pending approvals live in one process |
Observability and audit | Traces of model and tool calls with tokens; OpenTelemetry and Prometheus export; redaction on by default; a tamper-evident audit log on tool servers | Cost only in developer tools, from prices you supply; no web dashboard |
Cost control | Limits per run and per day on tool calls, runs and cost, checked before each tool call; token caps per run | Budget cost is a weight you assign to tools, not money; model spend needs the token cap |
Deployment and operations | Runs in your process, a container or Kubernetes; schedule, webhook and event triggers; restarts and a journal for crash recovery | No managed hosting: you run and scale it; no published container image |
Portability | Apache-2.0 license; OpenAI, Anthropic, Azure and many OpenAI-compatible providers; tools over MCP; agents defined in YAML files | Built on LangChain components; the MCP library is pinned to 1.x, not 2.x yet |
Support and viability | Public changelog; security reports through GitHub advisories | Small community and core team; no commercial support plan or SLA; marked Beta on PyPI |
The fair reading: Promptise Foundry suits teams that want agents in their own environment and have engineers to run them. If you need a managed service with a support contract, a cloud provider's agent service fits better. The Promptise docs cover each feature in detail.
[08]
Common mistakes
Mistake | Why it fails | Fix |
|---|---|---|
Scoring the demo | Demos run the vendor's best path on the vendor's data | Score only what you saw in your pilot; mark the rest "described" |
Letting a high total override a gate | You pick the best platform you can't use | Check gates as pass/fail before any scoring |
Changing weights after seeing scores | The sheet confirms a decision already made | Agree on the weights first and have the sponsor sign them off |
Testing only the success path | Agents fail through timeouts, unanswered approvals and loops | Plan at least three failures into the pilot |
Ignoring the exit | Switching later costs more than the license | Score portability and test a full export during the pilot |
Counting features instead of limits | "Supports approvals" can mean one channel and no timeout | Ask what happens when nobody answers, a limit is hit or a log is deleted |
[09]
Frequently asked questions
What should an AI agent platform evaluation checklist include?
Pass/fail gates for your must-haves, then weighted criteria: fit and quality, security, identity and access, approvals, observability and audit, cost control, deployment and operations, portability, and support. Add vendor questions, a defined 1–5 scale and a pilot plan that includes failures.
How do you weight criteria when evaluating AI agent platforms?
Share 100 points across the criteria before you meet any vendor, based on what the agents will do. Agents that act on business systems need more weight on security, identity and approvals; a platform many teams share needs more on portability and support. Have the sponsor sign off the weights before scoring.
How long should an AI agent platform pilot take?
Two to four weeks on one real workflow is usually enough to see integration problems, failure handling and daily use. Shorter pilots tend to test only the success path. Run every finalist on the same workflow and test data, so the scores are comparable.
What questions should you ask AI agent platform vendors?
Ask them to show, not describe: where your data is stored and for how long, whether it trains their models, how an agent's credentials are revoked, what happens when an approval times out, what stops a run that keeps spending, how you export logs and agent definitions, and how fast they fixed their last security issue.
Should we choose an open-source framework or a managed agent platform?
A managed platform suits you if you want no infrastructure to run and a support contract. An open-source framework suits you if you need control over where agents run, how approvals and limits work and which models you use, and you have engineers to operate it. Score both on the same sheet.
[10]
Next steps
Design the stack around the platform you pick: enterprise AI agent architecture.
Set owners, approvals, limits and reviews: AI agent governance in practice.
Decide whether to build or buy first: build vs buy AI agents.
See where your company stands: the AI maturity model.
For your engineers, how agents run in production: what is an agentic runtime?
How agents authenticate to tools: AI agent identity.