THE SHORT ANSWER
A useful AI evaluation set connects real tasks to verifiable expectations and meaningful consequences of error. Include normal work, difficult exceptions and observed failures rather than only successful examples. Separate development cases from held-out assessment and record provenance. The set should inform decisions about your particular workflow, not merely produce an impressive general model score whose relationship to everyday work is unclear.
To connect individual AI applications with reliable marketing operations, read the pillar guide AI transformation in B2B marketing: from individual prompts to reliable operations.
Define the unit of a test case
A case includes the actual request, necessary inputs, relevant system state and a verifiable expected outcome. For a research agent this may be an answer containing specific supported claims; for an update workflow it may be the correct destination state without additional effects. Avoid prescribing a single ideal wording unnecessarily. Different texts can serve the same purpose. Record mandatory boundaries, such as no unsupported price claim or no sending without appropriate authorisation. Strong performance on minor style features must not compensate for crossing one of those consequential boundaries. Explore this further in the guide Reviewing marketing claims with evidence and explicit limits.
Collect cases from different work paths
Use authorised historical work samples and observed failures. Cover languages, products, input formats and different source conditions. Also include tasks the process should reject or hand to a person. A set of complete briefs alone cannot test recognition of a decisive missing detail. Label each case’s work domain and reason for inclusion. When sensitive information is removed or altered, check whether the task retains its original difficulty. A sanitised example may accidentally lose precisely the context that triggered the failure, creating a reassuring test that no longer examines the original problem.
Add counterexamples and boundary conditions
Create focused variations of known failures: an expired source, similarly named companies, contradictory statements or missing approval. Label synthetic cases explicitly. They are useful but do not replace the distribution of real work. Each difficult case needs a specific testing purpose. Deliberately impossible tasks without a defined acceptable outcome merely create scoring disputes. Specify when clarification, a limited answer or a controlled stop should count as correct. Deliberately declining to execute an action can be a successful result, particularly when the task lacks a necessary source, identity check or authorisation. Explore this further in the guide Testing AI changes: catch regressions before rollout.
Separate development from independent assessment
Use part of the collection to improve prompts and tools while retaining other cases for later comparison. Optimising every change directly against the same examples can tailor the process to their peculiarities. Version cases, criteria and expected sources. Do not silently remove difficult examples because the current implementation performs poorly. If a case becomes professionally invalid, record why. Add production failures to the collection without quietly changing what historical scores mean. Preserve enough information to compare versions on a common subset and separately inspect newly covered scenarios that earlier evaluations did not contain.
Build an explicitly fictional source case
A fictional knowledge agent must answer a regional product availability question. Inputs include an old global overview and a current regional approval list. The expected result is an answer for the requested region with the appropriate source; if the region is absent, clarification is required. This hypothetical case tests more than polished wording. It examines timing, scope and missing context. A second variation removes the current list. The scoring method should then recognise a properly limited answer rather than mistakenly penalising it for failing to provide unsupported completeness about regional availability.
Review the evaluation set itself
Have specialists independently assess some cases and discuss disagreements. If they reach different expectations despite complete information, the case or business rule may be unclear. Report results by task type and error consequence, not only overall. Many easy cases can hide a small critical group. Check that reference sources remain accessible and valid. Passing the collection establishes behaviour only within the scope it covers. Document known gaps and the additional work samples required before assessing substantially broader use, rather than treating a high score as permission to apply the process to unrelated work.
INTERACTIVE / FROM READING TO ACTION
Three checkpoints for your implementation
Choose a checkpoint and track your reading progress. Checks are neither saved nor sent and do not score your business.
CHECKPOINT 01
Add counterexamples and boundary conditions
Create focused variations of known failures: an expired source, similarly named companies, contradictory statements or missing approval.
Explore in the article ↗CHECKPOINT 02
Separate development from independent assessment
Use part of the collection to improve prompts and tools while retaining other cases for later comparison.
Explore in the article ↗CHECKPOINT 03
Build an explicitly fictional source case
A fictional knowledge agent must answer a regional product availability question.
Explore in the article ↗Frequently asked questions
How many cases are enough?
There is no universal number. Relevant work paths, consequences and visible coverage gaps matter most.
Can we use synthetic cases?
Yes, when labelled and linked to a clear testing purpose. Supplement them with authorised real work samples.
Must each answer exactly match a reference?
No. Assess necessary content and boundaries; different factually correct phrasings may be acceptable.
How can I put “Define the unit of a test case” into practice?
A case includes the actual request, necessary inputs, relevant system state and a verifiable expected outcome. For a research agent this may be an answer containing specific supported claims; for an update workflow it may be the correct destination state without additional effects. Avoid prescribing a single ideal wording unnecessarily.
What matters when working on “Collect cases from different work paths”?
Use authorised historical work samples and observed failures. Cover languages, products, input formats and different source conditions. Also include tasks the process should reject or hand to a person.
