An AI workflow may appear better after a prompt or model change because the team tested only easy examples. To know whether improvement extends beyond the demo, use a stable evaluation set: a small, representative collection of inputs and expected criteria.
Define success for the actual task
Start with the business outcome rather than similarity to an ideal text. Classification may require each enquiry to reach the right team. Summarisation may require essential information without invented commitments. Marketing drafts may be judged on accuracy, brand-brief adherence and editing time. Record non-negotiable requirements that good performance elsewhere cannot offset.
Represent real variation
Collect cases reflecting the work and remove unnecessary information. Separate typical, difficult, incomplete, ambiguous and unacceptable requests. Keep some cases locked away from prompt tuning. Refresh the collection when the work changes while retaining versions for comparison.
Write criteria two reviewers can apply
For every case, define required content, prohibited content and error severity. An easily repaired omission differs from an invented price or altered intention. Have two reviewers assess part of the sample and examine disagreements. If the rubric is unclear, improve it first.
Keep test conditions stable
Record tool version, system instructions, inputs, settings and date. Run identical cases without case-specific manual adjustments. Where results vary between runs, repeat tests and retain the variation instead of choosing the most impressive output.
Translate scores into decisions
Report findings by case category and error type, not only an average. Show the proportion needing correction, editing time and failures to recognise uncertainty. Then decide whether the workflow remains draft-only, gains limited autonomy or pauses until a specific error is resolved.
Make evaluation continuous
Run the stable set whenever the model, prompt, data source or integration changes. Add real failures only after anonymisation and permission checks. Record who approves changes and how rollback works. The collection becomes the workflow's memory, warning when an improvement in one case breaks another.
This evaluation framework and rubric structure are DigitalNow's original methodology.