What you need to know
- Define acceptable outcomes before optimizing a prompt.
- Combine deterministic checks with validated judgment where needed.
- Use error consequences to inform release decisions.
Write the rubric before the prompt
An evaluation is useful when its result helps a team make a release decision. Start with a concrete task and describe the properties of an acceptable result. A support draft might need to cite the applicable policy, avoid promising an unavailable remedy, and identify missing account information. A summary might instead be judged on coverage, accuracy, and whether it separates a source's claims from established facts.
Use examples from the target workflow, with appropriate privacy controls. Include ordinary inputs alongside edge cases and previous failures. Keep a held-out set for checking whether an apparent improvement generalizes. If the team repeatedly tunes against the same visible examples, the score can become a measure of familiarity rather than capability.
Use a grader suited to the claim
Some properties can be checked deterministically: a required field exists, a total matches, or a forbidden action never ran. Other properties require expert review or a carefully validated model-based grader. Document what each grader measures and where it is unreliable. Review disagreements instead of treating a single aggregate score as a complete description of quality.
OpenAI's evaluation guidance emphasizes structured testing for variable model behavior. Our recommendation is to keep the quality measurement close to a product decision: which errors block a release, which require an approval step, and which can be monitored after a limited rollout? Different consequences justify different thresholds.
Carry the test into production
Version the model configuration, prompt, tools, and evaluation set together. Inspect regressions by task type before averaging them away. Production monitoring should then look for changes in input mix, failures, latency, and human correction. Add representative new problems to a maintained evaluation collection. The process becomes valuable when it informs an actual change, a narrower feature, or a decision to leave the current system in place.
Sources & further reading
This article draws on the source material and official product documentation below. Product capabilities can change; check each source for current details.
