Choosing a model by a leaderboard alone creates expensive systems. Production work has constraints that rankings rarely capture: answer format, acceptable delay, request volume, tool use, and the cost of a wrong answer.
Start with the task, not the brand
Write a one-sentence job definition before comparing models. “Extract invoice fields as JSON with a 99% valid-schema rate” is testable. “Use the best model” is not. Build an evaluation set of representative inputs, including difficult and malformed cases.
| Workload | Optimize for | Model profile |
|---|---|---|
| Classification / extraction | Valid structure, throughput, cost | Fast, economical |
| Customer-facing writing | Voice, accuracy, safety | Balanced general |
| Coding / complex analysis | Reasoning, tools, context | High capability |
| Agents | Tool reliability, recovery | Measured fallback |
Measure five things
- Task quality: use pass/fail checks, a human rubric, or exact-match criteria.
- Latency: track time to first token and total completion at realistic concurrency.
- Cost per successful outcome: a cheap model that needs retries may not be cheap.
- Format reliability: test JSON, citations, schemas, and tool calls.
- Operational behavior: measure timeouts and transient failures.
Use a model ladder
Send routine, bounded tasks to a fast route; escalate ambiguous or high-impact work to a stronger one; retain a safe fallback when the preferred route is unavailable. Drive the ladder with a confidence or deterministic rule rather than a vague sense that one model is smarter.
Keep routing reversible
Place model IDs, prompt versions, and thresholds in configuration. Record them with each result. When behavior or pricing changes, rerun your test set and change one route without refactoring the product.