How do I design an evaluation set for prompts?

Assemble 20–100 real cases that represent your distribution—easy, typical, and tricky. Label expected outputs or acceptance criteria. Include edge cases and near-misses. Keep it small enough to run daily. Track metrics that matter (accuracy, adherence, time to fix) and review failures. Refresh the set as your data shifts.

In practice

Tag examples by failure mode (missing data, ambiguity, formatting) to spot patterns quickly.

Where to start

1Collect real examples and redact sensitive info.
2Write gold outputs or rubrics for each.
3Automate a daily run and dashboard.
4Rotate in new examples monthly.

prompt evaluationLLM benchmark setprompt testing dataset

Build it properly

Promptise Foundry treats prompts as software — typed blocks, versioning, guards and an inspector that shows you every assembly decision.

Prompt engineering in Foundry Read the guides

How do I design an evaluation set for prompts?

Where to start

Related questions

Build it properly

How do I design an evaluation set for prompts?

Where to start

Related questions

Build it properly