PromptisePromptise
Docs
GitHub
Promptise - AI Framework LogoPromptise

The foundation layer for agentic intelligence. Build, secure and operate autonomous AI systems with Promptise Foundry.

pip install promptise

[01] Foundry

  • MCPcast
  • The Promptise Agent
  • Reasoning Engine
  • MCP
  • Agent Runtime
  • Prompt Engineering
  • Execution Engine
  • Agent Identity

[02] Resources

  • Documentation
  • GitHub
  • Guides
  • Learning Paths
  • Questions

[03] Company

  • About
  • Terms of Service
  • Privacy Policy
  • Cookie Policy
  • Subprocessors

© 2026 Promptise by Manser Ventures. All rights reserved.

Open source · Python

← Guides> Topic

Testing and evals

Model output changes from run to run, so a test that compares an answer to one exact string will fail on correct answers. Testing agents means checking properties instead: did it call the right tool with the right arguments, does the answer contain the facts it should, does a grader model or a rule accept it. A set of such checks run over many cases is called an eval.

These guides cover the three levels most teams need. The first builds an eval suite for a whole agent, with graders and a score you can track between versions. The second brings prompts and agents into pytest so changes are checked like any other code. The third tests MCP servers directly, so a tool bug is caught before an agent ever calls it.

[n.01/02]> 15 guides, in order
  1. 01How to Evaluate AI Agents: Evals, LLM Judges and pass^kHow to evaluate AI agents: write golden and refusal tasks, grade tool calls in code, calibrate an LLM judge, measure pass^k and gate CI on the results.Intermediate · 20 minOct 18, 2026
  2. 02How to Test LLM Prompts and AI Agents in PythonUnit test LLM prompts with pytest, catch prompt regressions in CI, grade outputs with an LLM judge and evaluate agent tool calls. Real runs, real costs.Intermediate · 24 minOct 11, 2026
  3. 03Testing MCP Servers in Python: pytest Without a Live ServerTest an MCP server in Python without running it: auth, guards, middleware, resources, prompts, error envelopes and job queues in 35 pytest tests, plus CI.Intermediate · 16 minOct 11, 2026
  4. 04AI Agent Evals in CI: Run Them on Every Pull RequestRun AI agent evals in CI on every pull request: test tool choice, arguments, answers and safety, keep cost and flakiness down, with GitHub Actions.Intermediate · 18 minOct 18, 2026
  5. 05Prompt Evaluation for Beginners: A Simple Eval LoopEvaluate prompts with a simple eval loop: build a golden set, write pass/fail checks, compare prompt versions and read the failures. Real Python output.Beginner · 7 minOct 18, 2026
  6. 06AI Agent Production Checklist: What to Check Before LaunchAn AI agent production checklist: security, approvals, guardrails, limits, cost, observability, evals, deployment and incident response, with code.Intermediate · 17 minOct 18, 2026
  7. 07AI Code Hallucinations: Why Plausible Output Fools YouWhy LLM code looks right but fails: plausibility vs correctness, hallucinated packages, a real regex that broke, and how to verify output by running it.Intermediate · 4 minOct 18, 2026
  8. 08Best LLM for AI Agents: Test and Switch Models in PythonFind the best LLM for your AI agent with a tool-calling eval, then switch between OpenAI, Anthropic, Azure, Bedrock, Gemini and Ollama in one line.Intermediate · 22 minOct 11, 2026
  9. 09Build a Production MCP Server in Python (Step by Step)Build an MCP server in Python ready for real agents: typed tools, clear errors, API keys with roles, rate limits, a tamper-evident audit log and tests.Intermediate · 18 minOct 11, 2026
  10. 10LLM as a Judge vs a Jury: Making AI Evaluations ReliableLLM-as-a-judge explained: position, length and self-preference bias, and how a jury of swapped orders, rubrics and mixed models fixes it. Real example.Intermediate · 5 minOct 18, 2026
  11. 11Prompts as Code: The LLM as a Probabilistic CompilerTreat prompts as code: write specs, enforce output shape with structured outputs, surface assumptions and test every change. Real date-extraction example.Intermediate · 5 minOct 18, 2026
  12. 12Prompts as Interfaces: Design Prompts Like API ContractsDesign prompts like API contracts: delimited inputs, a fixed output, a defined failure path, versions and tests. A real classifier prompt before and after.Intermediate · 4 minOct 18, 2026
  13. 13System Prompt Best Practices for AI Agents in PythonBuild agent system prompts from reusable blocks, switch them by conversation phase, and test them like code. Real Python runs, honest limits.Intermediate · 24 minOct 11, 2026
  14. 14Automatic Prompt Optimization: APE, ProTeGi and AMPOAutomatic prompt optimization explained: APE, ProTeGi, OPRO and multi-branch prompts, with one real optimization round on a ticket classifier in Python.Advanced · 9 minOct 18, 2026
  15. 15LLM Self-Correction: Self-Refine, Reflexion and CRITICLLM self-correction without fine-tuning: how Self-Refine, Reflexion and CRITIC work, why external feedback matters, and a real self-review vs test example.Advanced · 7 minOct 18, 2026
[n.02/02]> go further

In the documentation

  • Prompt testingdocs ↗
  • Testing MCP serversdocs ↗

Other topics

  • Building AI agents in Python→
  • Models and providers→
  • Tools and the Model Context Protocol (MCP)→
  • Memory, context and RAG→
  • Agent reasoning patterns→
  • Multi-agent systems→
  • Guardrails and security→
  • Human approvals (human-in-the-loop)→
  • Observability and debugging→
  • Cost control and rate limits→
  • Runtime, triggers and deployment→
  • Frameworks and comparisons→

Build it with Promptise Foundry.

The code in these guides runs on Promptise Foundry, an open-source Python framework for AI agents and MCP servers.

QuickstartAll guides