LLM evaluation should answer a product question: “Can this route perform this job safely and consistently for our users?” Public benchmarks are useful context, but they rarely match your prompts, documents, tools, language mix, or definition of a correct result.
Build a representative evaluation set
Collect real examples only with the appropriate privacy controls, then remove sensitive data or create faithful synthetic variants. Include normal examples, difficult cases, short and long inputs, ambiguous requests, malformed data, and examples where the correct action is to refuse or ask a clarifying question. Label the expected answer or define a reviewer rubric before evaluating models.
Choose task-specific metrics
| Workflow | Useful measures |
|---|---|
| Extraction | Exact field accuracy, valid schema rate |
| Support drafting | Rubric score, factual correctness, escalation accuracy |
| Tool-using agent | Correct tool choice, argument validity, task completion |
| Code generation | Tests passed, review findings, runtime behavior |
Track latency and token cost beside quality. The best route is usually the one that reaches the acceptance threshold with the lowest cost per successful result—not the lowest price per token or the highest abstract score.
Use human review where it matters
Automated checks are excellent for structure, exact answers, and policy constraints. Human review is still valuable for tone, usefulness, nuanced correctness, and user trust. Blind reviewers to model identity when possible, randomize samples, and document the rubric so the comparison is repeatable.
Protect against regressions
Run the same frozen evaluation set when you change a model, prompt, tool, system instruction, or routing threshold. Keep a separate holdout set so teams do not overfit to their visible test cases. If a candidate improves average quality but fails a critical category, its rollout should be gated until that failure is understood.
Make evaluation part of operations
Production feedback reveals new edge cases. Add them to the set, version the dataset and scoring rules, and review route performance on a recurring schedule. This is how a model stack improves with the product rather than becoming a one-time vendor choice.